dynamoid 3.1.0 → 3.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: ec92a10397fa1be691eaebe98659952c7909ab97
-  data.tar.gz: 766016509f22d0ce3f4fe93ca68342ed6038d0b9
+SHA256:
+  metadata.gz: 193003bce5df92dae1a2ad3a4f94345bc610193b0da7692eb6f722231e1becae
+  data.tar.gz: 36fb0bdea8126c6fd9af32343ad5d0ebe87ca92d01caab17fb9fe87aa74d07a6
 SHA512:
-  metadata.gz: 6b0de822f2cda614a74a933c714072b97525a7b1a03444f2b12034327a8769d61e13220eb9d0b0f7ec15b2fbc2bf244574efe45a0660fac26cc118bb84fe848e
-  data.tar.gz: d6f3287d3741c21167b521d35771a4d794a9dff90047f9b46ebaa09373c8776389d4afa161c1430559b6fda4e0b9728f22a0a0007dbcdce507d8eedccc31018c
+  metadata.gz: 3c53982905e1e487db42f0497f682e4a62b33d8adf1b1b7079e98f5d1635e338396aa62bef467483406ab558d4ad66a2a285ac72c90d5f278530026fbcb96fa1
+  data.tar.gz: 2d8e83e5ea4ba495ac6ec37c7178b8f77db4d0b2e33210c698ef00f8f92f1762672df3ca64c70e053f173d040e2abafe01bdd0c850499f65523fb9a50c2f0a7b

data/.rubocop.yml

@@ -8,12 +8,28 @@ AllCops:
 # It's a matter of taste
 Layout/AlignParameters:
   EnforcedStyle: with_fixed_indentation
+Layout/AlignHash:
+  Enabled: false
 Style/GuardClause:
   Enabled: false
 Style/FormatStringToken:
   Enabled: false
 Style/DoubleNegation:
   Enabled: false
+Style/IfUnlessModifier:
+  Enabled: false
+Style/EachWithObject:
+  Enabled: false
+Style/SafeNavigation:
+  Enabled: false
+Style/BlockDelimiters:
+  Enabled: false
+Layout/MultilineMethodCallIndentation:
+  EnforcedStyle: indented
+Naming/VariableNumber:
+  Enabled: false
+Style/MultilineBlockChain:
+  Enabled: false
 
 # We aren't so brave to tackle all these issues right now
 Metrics/LineLength:
@@ -50,4 +66,6 @@ Style/MissingRespondToMissing:
   Enabled: false
 Naming/PredicateName:
   Enabled: false
+Security/YAMLLoad:
+  Enabled: false
 

data/.travis.yml

@@ -2,10 +2,12 @@ sudo: required
 
 language: ruby
 rvm:
-  - ruby-2.3.7
-  - ruby-2.4.4
-  - ruby-2.5.1
+  - ruby-2.3.8
+  - ruby-2.4.5
+  - ruby-2.5.3
+  - ruby-2.6.1
   - jruby-9.1.17.0
+  - jruby-9.2.6.0
 gemfile:
 gemfile:
   - gemfiles/rails_4_2.gemfile

data/CHANGELOG.md

@@ -10,6 +10,21 @@
 
 
 
+# 3.2.0
+
+## Features
+* Feature: [#341](https://github.com/Dynamoid/dynamoid/pull/341), [#342](https://github.com/Dynamoid/dynamoid/pull/342) Add `find_by_pages` method to provide access to DynamoDB query result pagination mechanism (@bmalinconico, @arjes)
+* Feature: [#354](https://github.com/Dynamoid/dynamoid/pull/354) Add `map` field type
+
+## Improvements
+* Improvement: [#340](https://github.com/Dynamoid/dynamoid/pull/340) Improve selecting more optimal GSI for Query operation - choose GSI with sort key if it's used in criteria (@ryz310)
+* Improvement: [#351](https://github.com/Dynamoid/dynamoid/pull/351) Add warnings about nonexistent fields in `where` conditions
+* Improvement: [#352](https://github.com/Dynamoid/dynamoid/pull/352) Add warning about skipped conditions
+* Improvement: [#356](https://github.com/Dynamoid/dynamoid/pull/356) Simplify requiring Rake tasks in non-Rails application
+* Improvement: Readme.md. Minor improvements and fixes (@cabello)
+
+
+
 # 3.1.0
 
 ## Improvements

data/README.md

@@ -1,8 +1,12 @@
 # Dynamoid
 
-You are viewing the README for version 3 of Dynamoid.  See the [CHANGELOG](https://github.com/Dynamoid/Dynamoid/blob/master/CHANGELOG.md#200) for details on breaking changes since 1.3.x.
-
-For version 1.3.x use the [1-3-stable branch](https://github.com/Dynamoid/Dynamoid/blob/1-3-stable/README.md).
+[![Build Status](https://travis-ci.org/Dynamoid/dynamoid.svg?branch=master)](https://travis-ci.org/Dynamoid/dynamoid)
+[![Code Climate](https://codeclimate.com/github/Dynamoid/dynamoid.svg)](https://codeclimate.com/github/Dynamoid/dynamoid)
+[![Coverage Status](https://coveralls.io/repos/github/Dynamoid/dynamoid/badge.svg?branch=master)](https://coveralls.io/github/Dynamoid/dynamoid?branch=master)
+[![CodeTriage Helpers](https://www.codetriage.com/dynamoid/dynamoid/badges/users.svg)](https://www.codetriage.com/dynamoid/dynamoid)
+[![Yard Docs](http://img.shields.io/badge/yard-docs-blue.svg)](https://www.rubydoc.info/github/Dynamoid/dynamoid/frames)
+[![Inline docs](http://inch-ci.org/github/Dynamoid/Dynamoid.svg?branch=master)](http://inch-ci.org/github/Dynamoid/Dynamoid)
+![GitHub](https://img.shields.io/github/license/Dynamoid/dynamoid.svg)
 
 Dynamoid is an ORM for Amazon's DynamoDB for Ruby applications. It
 provides similar functionality to ActiveRecord and improves on
@@ -14,27 +18,12 @@ DynamoDB is not like other document-based databases you might know, and is very
 
 But if you want a fast, scalable, simple, easy-to-use database (and a Gem that supports it) then look no further!
 
-
-| Project                 |  Dynamoid         |
-|------------------------ | ----------------- |
-| gem name                |  dynamoid         |
-| license                 |  MIT              |
-| download rank           |  [![Total Downloads](https://img.shields.io/gem/rt/Dynamoid.svg)](https://rubygems.org/gems/dynamoid) |
-| version                 |  [![Gem Version](https://badge.fury.io/rb/dynamoid.svg)](https://badge.fury.io/rb/dynamoid) |
-| dependencies            |  [![Depfu](https://badges.depfu.com/badges/6661c063c8e77a5008344fc7283a50aa/status.svg)](https://depfu.com) |
-| code quality            |  [![Code Climate](https://codeclimate.com/github/Dynamoid/dynamoid.svg)](https://codeclimate.com/github/Dynamoid/dynamoid) |
-| continuous integration  |  [![Build Status](https://travis-ci.org/Dynamoid/dynamoid.svg?branch=master)](https://travis-ci.org/Dynamoid/dynamoid) |
-| test coverage           |  [![Coverage Status](https://coveralls.io/repos/github/Dynamoid/Dynamoid/badge.svg?branch=master)](https://coveralls.io/github/Dynamoid/Dynamoid?branch=master) |
-| triage helpers          |  [![CodeTriage Helpers](https://www.codetriage.com/dynamoid/dynamoid/badges/users.svg)](https://www.codetriage.com/dynamoid/dynamoid) |
-| homepage                |  [https://github.com/Dynamoid/dynamoid](https://github.com/Dynamoid/dynamoid) |
-| documentation           |  [http://rdoc.info/github/Dynamoid/dynamoid/frames](http://rdoc.info/github/Dynamoid/dynamoid/frames) |
-
 ## Installation
 
 Installing Dynamoid is pretty simple. First include the Gem in your Gemfile:
 
 ```ruby
-gem 'dynamoid', '~> 2'
+gem 'dynamoid'
 ```
 ## Prerequisities
 
@@ -98,14 +87,9 @@ Then you need to initialize Dynamoid config to get it going. Put code similar to
 
 Dynamoid supports Ruby >= 2.3 and Rails >= 4.2.
 
-Its compatibility is tested in following way:
-
-| Ruby / Active Record  | 4.2.x | 5.0.x | 5.1.x | 5.2.x |
-|:---------------------:|:-----:|:-----:|:-----:|:-----:|
-| 2.3.7                 | ✓     | ✓     | ✓     | ✓     |
-| 2.4.4                 | ✓     | ✓     | ✓     | ✓     |
-| 2.5.1                 | ✓     | ✓     | ✓     | ✓     |
-| jruby-9.1.17.0        | ✓     | ✓     | ✓     | ✓     |
+Its compatibility is tested against following Ruby versions: 2.3.8,
+2.4.5, 2.5.3 and 2.6.1, JRuby versions 9.1.17.0 and 9.2.6.0 and
+ against Rails versions: 4.2.x, 5.0.x, 5.1.x and 5.2.x.
 
 ## Setup
 
@@ -137,7 +121,8 @@ These fields will not change an existing table: so specifying a new read_capacit
 You'll have to define all the fields on the model and the data type of each field. Every field on the object must be included here; if you miss any they'll be completely bypassed during DynamoDB's initialization and will not appear on the model objects.
 
 By default, fields are assumed to be of type `:string`. Other built-in types are
-`:integer`, `:number`, `:set`, `:array`, `:datetime`, `date`, `:boolean`, `:raw` and `:serialized`.
+`:integer`, `:number`, `:set`, `:array`, `:map`, `:datetime`, `date`, `:boolean`, `:raw` and `:serialized`.
+`array` and `map` match List and Map DynamoDB types respectively.
 `raw` type means you can store Ruby Array, Hash, String and numbers.
 If built-in types do not suit you, you can use a custom field type represented by an arbitrary class, provided that the class supports a compatible serialization interface.
 The primary use case for using a custom field type is to represent your business logic with high-level types, while ensuring portability or backward-compatibility of the serialized representation.
@@ -181,7 +166,7 @@ class Document
 end
 ```
 
-WARNING: Fields in numeric format are stored with nanoseconds as a fraction part and precision could be lost.
+**WARNING:** Fields in numeric format are stored with nanoseconds as a fraction part and precision could be lost.
 That's why `datetime` field in numeric format shouldn't be used as a range key.
 
 You have two options if you need to use a `datetime` field as a range key:
@@ -547,6 +532,18 @@ u.addresses.where(city: 'Chicago').all
 
 But keep in mind Dynamoid -- and document-based storage systems in general -- are not drop-in replacements for existing relational databases. The above query does not efficiently perform a conditional join, but instead finds all the user's addresses and naively filters them in Ruby. For large associations this is a performance hit compared to relational database engines.
 
+**WARNING:** There is a limitation of conditions passed to `where`
+method. Only one condition for some particular field could be specified.
+The last one only will be applyed and others will be ignored. E.g. in
+examples:
+
+```ruby
+User.where('age.gt': 10, 'age.lt': 20)
+User.where(name: 'Mike').where('name.begins_with': 'Ed')
+```
+
+the first one will be ignored and the last one will be used.
+
 #### Limits
 
 There are three types of limits that you can query with:
@@ -585,6 +582,42 @@ Address.record_limit(10_000).batch(100).each { … } # Batch specified as part o
 The implication of batches is that the underlying requests are done in the batch sizes to make the request and responses
 more manageable. Note that this batching is for `Query` and `Scans` and not `BatchGetItem` commands.
 
+#### DynamoDB pagination
+
+At times it can be useful to rely on DynamoDB [low-level pagination](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.Pagination)
+instead of fixed pages sizes. Each page results in a single Query or Scan call
+to DyanmoDB, but returns an unknown number of records.
+
+Access to the native DynamoDB pages can be obtained via the `find_by_pages`
+method, which yields arrays of records.
+
+```ruby
+Address.find_by_pages do |addresses, metadata|
+end
+```
+
+Each yielded pages returns page metadata as the second argument, which is a hash
+including a key `:last_evaluated_key`. The value of this key can be used for
+the `start` method to fetch the next page of records.
+
+This way it can be used for instance to implement efficiently
+pagination in web-application:
+
+```ruby
+class UserController < ApplicationController
+  def index
+    next_page = params[:next_page_token] ? JSON.parse(Base64.decode64(params[:next_page_token])) : nil
+
+    records, metadata = User.start(next_page).find_by_pages.first
+
+    render json: {
+      records: records,
+      next_page_token: Base64.encode64(metadata[:last_evaluated_key].to_json)
+    }
+  end
+end
+```
+
 #### Sort Conditions and Filters
 
 You are able to optimize query with condition for sort key. Following operators are available: `gt`, `lt`, `gte`, `lte`,
@@ -684,7 +717,7 @@ class User
   field :name
   field :age, :number
 
-  global_secondary_index hash_key: :age
+  global_secondary_index hash_key: :age # Must come after field definitions.
 end
 ```
 
@@ -698,44 +731,17 @@ There are following options:
 
 The only mandatory option is `name`.
 
-To use index in `Document.where` implicitly you need to project all the fields with option `projected_attributes: :all`.
-
-There are two ways to query Global Secondary Indexes (GSI).
-
-#### Explicit
-
-The first way explicitly uses your GSI and utilizes the `find_all_by_secondary_index` method which will lookup a valid
-GSI to use based on the inputs, you MUST provide the correct keys to match the GSI you want:
+**WARNING:** In order to use global secondary index in `Document.where` implicitly you need to have all the attributes of the original table in the index and declare it with option `projected_attributes: :all`:
 
 ```ruby
-find_all_by_secondary_index(
-    {
-        dynamo_primary_key_column_name => dynamo_primary_key_value
-    }, # The signature of find_all_by_secondary_index is ugly, so must be an explicit hash here
-    :range => {
-        "#{range_column}.#{range_modifier}" => range_value
-    },
-    # false is the same as DESC in SQL (newest timestamp first)
-    # true is the same as ASC in SQL (oldest timestamp first)
-    scan_index_forward: false # or true
-)
-```
-
-Where the range modifier is one of `Dynamoid::Finders::RANGE_MAP.keys`, where the `RANGE_MAP` is:
+class User
+  # ...
 
-```ruby
-RANGE_MAP = {
-  'gt'            => :range_greater_than,
-  'lt'            => :range_less_than,
-  'gte'           => :range_gte,
-  'lte'           => :range_lte,
-  'begins_with'   => :range_begins_with,
-  'between'       => :range_between,
-  'eq'            => :range_eq
-}
+  global_secondary_index hash_key: :age, projected_attributes: :all
+end
 ```
 
-Most range searches, like `eq`, need a single value, and searches like `between`, need an array with two values.
+There is only one implicit way to query Global and Local Secondary Indexes (GSI/LSI).
 
 #### Implicit
 
@@ -791,6 +797,10 @@ Listed below are all configuration options.
 `'t'` and `'f'`. Default is true
 * `backoff` - is a hash: key is a backoff strategy (symbol), value is parameters for the strategy. Is used in batch operations. Default id `nil`
 * `backoff_strategies`: is a hash and contains all available strategies. Default is { constant: ..., exponential: ...}
+* `http_continue_timeout`: The number of seconds to wait for a 100-continue HTTP response before sending the request body. Default option value is `nil`. If not specified effected value is `1`
+* `http_idle_timeout`: The number of seconds an HTTP connection is allowed to sit idble before it is considered stale. Default option value is `nil`. If not specified effected value is `5`
+* `http_open_timeout`: The number of seconds to wait when opening a HTTP session. Default option value is `nil`. If not specified effected value is `15`
+* `http_read_timeout`:The number of seconds to wait for HTTP response data. Default option value is `nil`. If not specified effected value is `60`
 
 
 ## Concurrency
@@ -859,9 +869,23 @@ end
 
 ## Rake Tasks
 
+There are a few Rake tasks available out of the box:
+
   * `rake dynamoid:create_tables`
   * `rake dynamoid:ping`
 
+In order to use them in non-Rails application they should be required explicitly:
+
+```ruby
+# Rakefile
+
+Rake::Task.define_task(:environment)
+require 'dynamoid/tasks'
+```
+
+The Rake tasks depend on `:environment` task so it should be declared
+as well.
+
 ## Test Environment
 
 In test environment you will most likely want to clean the database between test runs to keep tests completely isolated. This can be achieved like so
@@ -904,6 +928,32 @@ Dynamoid.configure do |config|
 end
 ```
 
+## Logging
+
+There is a config option `logger`. Dynamoid writes requests and
+responses to DynamoDB using this logger on the `debug` level. So in
+order to troubleshoot and debug issues just set it:
+
+```ruby
+class User
+  include Dynamoid::Document
+  field name
+end
+
+Dynamoid.config.logger.level = :debug
+Dynamoid.config.endpoint = 'localhost:8000'
+
+User.create(name: 'Alex')
+
+# => D, [2019-05-12T20:01:07.840051 #75059] DEBUG -- : put_item | Request "{\"TableName\":\"dynamoid_users\",\"Item\":{\"created_at\":{\"N\":\"1557680467.608749\"},\"updated_at\":{\"N\":\"1557680467.608809\"},\"id\":{\"S\":\"1227eea7-2c96-4b8a-90d9-77b38eb85cd0\"}},\"Expected\":{\"id\":{\"Exists\":false}}}" | Response "{}"
+
+# => D, [2019-05-12T20:01:07.842397 #75059] DEBUG -- : (231.28 ms) PUT ITEM - ["dynamoid_users", {:created_at=>0.1557680467608749e10, :updated_at=>0.1557680467608809e10, :id=>"1227eea7-2c96-4b8a-90d9-77b38eb85cd0", :User=>nil}, {}]
+```
+
+The first line is a body of HTTP request and response. The second line -
+Dynamoid internal logging of API call (`PUT ITEM` in our case) with
+timing (231.28 ms).
+
 ## Credits
 
 Dynamoid borrows code, structure, and even its name very liberally from the truly amazing [Mongoid](https://github.com/mongoid/mongoid). Without Mongoid to crib from none of this would have been possible, and I hope they don't mind me reusing their very awesome ideas to make DynamoDB just as accessible to the Ruby world as MongoDB.

data/Vagrantfile

@@ -2,7 +2,7 @@
 
 Vagrant.configure('2') do |config|
   # Choose base box
-  config.vm.box = 'bento/ubuntu-16.04'
+  config.vm.box = 'bento/ubuntu-18.04'
 
   config.vm.provider 'virtualbox' do |vb|
     # Prevent clock skew when host goes to sleep while VM is running
@@ -20,7 +20,7 @@ Vagrant.configure('2') do |config|
     # Pillars
     salt.pillar(
       'ruby' => {
-        'version' => '2.4.1'
+        'version' => '2.6.2'
       }
     )
 

data/docker-compose.yml

@@ -2,6 +2,6 @@ version: '2'
 
 services:
   dynamodb:
-    image: deangiberson/aws-dynamodb-local
+    image: amazon/dynamodb-local
     ports:
       - 8000:8000

data/gemfiles/rails_4_2.gemfile

@@ -4,8 +4,8 @@
 
 source 'https://rubygems.org'
 
+gem 'activemodel', '~> 4.2.0'
 gem 'nokogiri', '~> 1.6.8'
 gem 'pry-byebug', platforms: :ruby
-gem 'rails', '~> 4.2.0'
 
 gemspec path: '../'

data/gemfiles/rails_5_0.gemfile

@@ -4,7 +4,7 @@
 
 source 'https://rubygems.org'
 
+gem 'activemodel', '~> 5.0.0'
 gem 'pry-byebug', platforms: :ruby
-gem 'rails', '~> 5.0.0'
 
 gemspec path: '../'

data/gemfiles/rails_5_1.gemfile

@@ -4,7 +4,7 @@
 
 source 'https://rubygems.org'
 
+gem 'activemodel', '~> 5.1.0'
 gem 'pry-byebug', platforms: :ruby
-gem 'rails', '~> 5.1.0'
 
 gemspec path: '../'

data/gemfiles/rails_5_2.gemfile

@@ -4,7 +4,7 @@
 
 source 'https://rubygems.org'
 
+gem 'activemodel', '~> 5.2.0'
 gem 'pry-byebug', platforms: :ruby
-gem 'rails', '~> 5.2.0'
 
 gemspec path: '../'

data/lib/dynamoid/adapter.rb

@@ -155,6 +155,7 @@ module Dynamoid
     # @since 0.2.0
     def method_missing(method, *args, &block)
       return benchmark(method, *args) { adapter.send(method, *args, &block) } if adapter.respond_to?(method)
+
       super
     end
 

data/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb

@@ -1,7 +1,11 @@
 # frozen_string_literal: true
 
-require_relative 'query'
-require_relative 'scan'
+require_relative 'aws_sdk_v3/query'
+require_relative 'aws_sdk_v3/scan'
+require_relative 'aws_sdk_v3/create_table'
+require_relative 'aws_sdk_v3/item_updater'
+require_relative 'aws_sdk_v3/table'
+require_relative 'aws_sdk_v3/until_past_table_status'
 
 module Dynamoid
   module AdapterPlugin
@@ -187,6 +191,7 @@ module Dynamoid
 
         table_ids.each do |t, ids|
           next if ids.blank?
+
           ids = Array(ids).dup
           tbl = describe_table(t)
           hk  = tbl.hash_key.to_s
@@ -293,58 +298,7 @@ module Dynamoid
       # @since 1.0.0
       def create_table(table_name, key = :id, options = {})
         Dynamoid.logger.info "Creating #{table_name} table. This could take a while."
-        read_capacity = options[:read_capacity] || Dynamoid::Config.read_capacity
-        write_capacity = options[:write_capacity] || Dynamoid::Config.write_capacity
-
-        secondary_indexes = options.slice(
-          :local_secondary_indexes,
-          :global_secondary_indexes
-        )
-        ls_indexes = options[:local_secondary_indexes]
-        gs_indexes = options[:global_secondary_indexes]
-
-        key_schema = {
-          hash_key_schema: { key => (options[:hash_key_type] || :string) },
-          range_key_schema: options[:range_key]
-        }
-        attribute_definitions = build_all_attribute_definitions(
-          key_schema,
-          secondary_indexes
-        )
-        key_schema = aws_key_schema(
-          key_schema[:hash_key_schema],
-          key_schema[:range_key_schema]
-        )
-
-        client_opts = {
-          table_name: table_name,
-          provisioned_throughput: {
-            read_capacity_units: read_capacity,
-            write_capacity_units: write_capacity
-          },
-          key_schema: key_schema,
-          attribute_definitions: attribute_definitions
-        }
-
-        if ls_indexes.present?
-          client_opts[:local_secondary_indexes] = ls_indexes.map do |index|
-            index_to_aws_hash(index)
-          end
-        end
-
-        if gs_indexes.present?
-          client_opts[:global_secondary_indexes] = gs_indexes.map do |index|
-            index_to_aws_hash(index)
-          end
-        end
-        resp = client.create_table(client_opts)
-        options[:sync] = true if !options.key?(:sync) && ls_indexes.present? || gs_indexes.present?
-        until_past_table_status(table_name, :creating) if options[:sync] &&
-                                                          (status = PARSE_TABLE_STATUS.call(resp, :table_description)) &&
-                                                          status == TABLE_STATUSES[:creating]
-        # Response to original create_table, which, if options[:sync]
-        #   may have an outdated table_description.table_status of "CREATING"
-        resp
+        CreateTable.new(client, table_name, key, options).call
       rescue Aws::DynamoDB::Errors::ResourceInUseException => e
         Dynamoid.logger.error "Table #{table_name} cannot be created as it already exists"
       end
@@ -402,9 +356,9 @@ module Dynamoid
       # @since 1.0.0
       def delete_table(table_name, options = {})
         resp = client.delete_table(table_name: table_name)
-        until_past_table_status(table_name, :deleting) if options[:sync] &&
-                                                          (status = PARSE_TABLE_STATUS.call(resp, :table_description)) &&
-                                                          status == TABLE_STATUSES[:deleting]
+        UntilPastTableStatus.new(table_name, :deleting).call if options[:sync] &&
+                                                                (status = PARSE_TABLE_STATUS.call(resp, :table_description)) &&
+                                                                status == TABLE_STATUSES[:deleting]
         table_cache.delete(table_name)
       rescue Aws::DynamoDB::Errors::ResourceInUseException => e
         Dynamoid.logger.error "Table #{table_name} cannot be deleted as it is in use"
@@ -461,6 +415,7 @@ module Dynamoid
         yield(iu = ItemUpdater.new(table, key, range_key))
 
         raise "non-empty options: #{options}" unless options.empty?
+
         begin
           result = client.update_item(table_name: table_name,
                                       key: key_stanza(table, key, range_key),
@@ -532,11 +487,14 @@ module Dynamoid
       #
       # @todo Provide support for various other options http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#query-instance_method
       def query(table_name, options = {})
-        table = describe_table(table_name)
-
         Enumerator.new do |yielder|
+          table = describe_table(table_name)
+
           Query.new(client, table, options).call.each do |page|
-            page.items.each { |row| yielder << result_item_to_hash(row) }
+            yielder.yield(
+              page.items.map { |row| result_item_to_hash(row) },
+              last_evaluated_key: page.last_evaluated_key
+            )
           end
         end
       end
@@ -562,11 +520,14 @@ module Dynamoid
       #
       # @todo: Provide support for various options http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#scan-instance_method
       def scan(table_name, conditions = {}, options = {})
-        table = describe_table(table_name)
-
         Enumerator.new do |yielder|
+          table = describe_table(table_name)
+
           Scan.new(client, table, conditions, options).call.each do |page|
-            page.items.each { |row| yielder << result_item_to_hash(row) }
+            yielder.yield(
+              page.items.map { |row| result_item_to_hash(row) },
+              last_evaluated_key: page.last_evaluated_key
+            )
           end
         end
       end
@@ -591,7 +552,7 @@ module Dynamoid
         hk    = table.hash_key
         rk    = table.range_key
 
-        scan(table_name, {}, {}).each do |attributes|
+        scan(table_name, {}, {}).flat_map{ |i| i }.each do |attributes|
           opts = {}
           opts[:range_key] = attributes[rk.to_sym] if rk
           delete_item(table_name, attributes[hk], opts)
@@ -604,59 +565,6 @@ module Dynamoid
 
       protected
 
-      def check_table_status?(counter, resp, expect_status)
-        status = PARSE_TABLE_STATUS.call(resp)
-        again = counter < Dynamoid::Config.sync_retry_max_times &&
-                status == TABLE_STATUSES[expect_status]
-        { again: again, status: status, counter: counter }
-      end
-
-      def until_past_table_status(table_name, status = :creating)
-        counter = 0
-        resp = nil
-        begin
-          check = { again: true }
-          while check[:again]
-            sleep Dynamoid::Config.sync_retry_wait_seconds
-            resp = client.describe_table(table_name: table_name)
-            check = check_table_status?(counter, resp, status)
-            Dynamoid.logger.info "Checked table status for #{table_name} (check #{check.inspect})"
-            counter += 1
-          end
-        # If you issue a DescribeTable request immediately after a CreateTable
-        #   request, DynamoDB might return a ResourceNotFoundException.
-        # This is because DescribeTable uses an eventually consistent query,
-        #   and the metadata for your table might not be available at that moment.
-        # Wait for a few seconds, and then try the DescribeTable request again.
-        # See: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#describe_table-instance_method
-        rescue Aws::DynamoDB::Errors::ResourceNotFoundException => e
-          case status
-          when :creating then
-            if counter >= Dynamoid::Config.sync_retry_max_times
-              Dynamoid.logger.warn "Waiting on table metadata for #{table_name} (check #{counter})"
-              retry # start over at first line of begin, does not reset counter
-            else
-              Dynamoid.logger.error "Exhausted max retries (Dynamoid::Config.sync_retry_max_times) waiting on table metadata for #{table_name} (check #{counter})"
-              raise e
-            end
-          else
-            # When deleting a table, "not found" is the goal.
-            Dynamoid.logger.info "Checked table status for #{table_name}: Not Found (check #{check.inspect})"
-          end
-        end
-      end
-
-      # Converts from symbol to the API string for the given data type
-      # E.g. :number -> 'N'
-      def api_type(type)
-        case type
-        when :string then STRING_TYPE
-        when :number then NUM_TYPE
-        when :binary then BINARY_TYPE
-        else raise "Unknown type: #{type}"
-        end
-      end
-
       #
       # The key hash passed on get_item, put_item, delete_item, update_item, etc
       #
@@ -706,152 +614,6 @@ module Dynamoid
         end
       end
 
-      # Converts a Dynamoid::Indexes::Index to an AWS API-compatible hash.
-      # This resulting hash is of the form:
-      #
-      #   {
-      #     index_name: String
-      #     keys: {
-      #       hash_key: aws_key_schema (hash)
-      #       range_key: aws_key_schema (hash)
-      #     }
-      #     projection: {
-      #       projection_type: (ALL, KEYS_ONLY, INCLUDE) String
-      #       non_key_attributes: (optional) Array
-      #     }
-      #     provisioned_throughput: {
-      #       read_capacity_units: Integer
-      #       write_capacity_units: Integer
-      #     }
-      #   }
-      #
-      # @param [Dynamoid::Indexes::Index] index the index.
-      # @return [Hash] hash representing an AWS Index definition.
-      def index_to_aws_hash(index)
-        key_schema = aws_key_schema(index.hash_key_schema, index.range_key_schema)
-
-        hash = {
-          index_name: index.name,
-          key_schema: key_schema,
-          projection: {
-            projection_type: index.projection_type.to_s.upcase
-          }
-        }
-
-        # If the projection type is include, specify the non key attributes
-        if index.projection_type == :include
-          hash[:projection][:non_key_attributes] = index.projected_attributes
-        end
-
-        # Only global secondary indexes have a separate throughput.
-        if index.type == :global_secondary
-          hash[:provisioned_throughput] = {
-            read_capacity_units: index.read_capacity,
-            write_capacity_units: index.write_capacity
-          }
-        end
-        hash
-      end
-
-      # Converts hash_key_schema and range_key_schema to aws_key_schema
-      # @param [Hash] hash_key_schema eg: {:id => :string}
-      # @param [Hash] range_key_schema eg: {:created_at => :number}
-      # @return [Array]
-      def aws_key_schema(hash_key_schema, range_key_schema)
-        schema = [{
-          attribute_name: hash_key_schema.keys.first.to_s,
-          key_type: HASH_KEY
-        }]
-
-        if range_key_schema.present?
-          schema << {
-            attribute_name: range_key_schema.keys.first.to_s,
-            key_type: RANGE_KEY
-          }
-        end
-        schema
-      end
-
-      # Builds aws attributes definitions based off of primary hash/range and
-      # secondary indexes
-      #
-      # @param key_data
-      # @option key_data [Hash] hash_key_schema - eg: {:id => :string}
-      # @option key_data [Hash] range_key_schema - eg: {:created_at => :number}
-      # @param [Hash] secondary_indexes
-      # @option secondary_indexes [Array<Dynamoid::Indexes::Index>] :local_secondary_indexes
-      # @option secondary_indexes [Array<Dynamoid::Indexes::Index>] :global_secondary_indexes
-      def build_all_attribute_definitions(key_schema, secondary_indexes = {})
-        ls_indexes = secondary_indexes[:local_secondary_indexes]
-        gs_indexes = secondary_indexes[:global_secondary_indexes]
-
-        attribute_definitions = []
-
-        attribute_definitions << build_attribute_definitions(
-          key_schema[:hash_key_schema],
-          key_schema[:range_key_schema]
-        )
-
-        if ls_indexes.present?
-          ls_indexes.map do |index|
-            attribute_definitions << build_attribute_definitions(
-              index.hash_key_schema,
-              index.range_key_schema
-            )
-          end
-        end
-
-        if gs_indexes.present?
-          gs_indexes.map do |index|
-            attribute_definitions << build_attribute_definitions(
-              index.hash_key_schema,
-              index.range_key_schema
-            )
-          end
-        end
-
-        attribute_definitions.flatten!
-        # uniq these definitions because range keys might be common between
-        # primary and secondary indexes
-        attribute_definitions.uniq!
-        attribute_definitions
-      end
-
-      # Builds an attribute definitions based on hash key and range key
-      # @params [Hash] hash_key_schema - eg: {:id => :string}
-      # @params [Hash] range_key_schema - eg: {:created_at => :datetime}
-      # @return [Array]
-      def build_attribute_definitions(hash_key_schema, range_key_schema = nil)
-        attrs = []
-
-        attrs << attribute_definition_element(
-          hash_key_schema.keys.first,
-          hash_key_schema.values.first
-        )
-
-        if range_key_schema.present?
-          attrs << attribute_definition_element(
-            range_key_schema.keys.first,
-            range_key_schema.values.first
-          )
-        end
-
-        attrs
-      end
-
-      # Builds an aws attribute definition based on name and dynamoid type
-      # @params [Symbol] name - eg: :id
-      # @params [Symbol] dynamoid_type - eg: :string
-      # @return [Hash]
-      def attribute_definition_element(name, dynamoid_type)
-        aws_type = api_type(dynamoid_type)
-
-        {
-          attribute_name: name.to_s,
-          attribute_type: aws_type
-        }
-      end
-
       # Build an array of values for Condition
       # Is used in ScanFilter and QueryFilter
       # https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
@@ -871,139 +633,8 @@ module Dynamoid
         end
       end
 
-      #
-      # Represents a table. Exposes data from the "DescribeTable" API call, and also
-      # provides methods for coercing values to the proper types based on the table's schema data
-      #
-      class Table
-        attr_reader :schema
-
-        #
-        # @param [Hash] schema Data returns from a "DescribeTable" call
-        #
-        def initialize(schema)
-          @schema = schema[:table]
-        end
-
-        def range_key
-          @range_key ||= schema[:key_schema].find { |d| d[:key_type] == RANGE_KEY }.try(:attribute_name)
-        end
-
-        def range_type
-          range_type ||= schema[:attribute_definitions].find do |d|
-            d[:attribute_name] == range_key
-          end.try(:fetch, :attribute_type, nil)
-        end
-
-        def hash_key
-          @hash_key ||= schema[:key_schema].find { |d| d[:key_type] == HASH_KEY }.try(:attribute_name).to_sym
-        end
-
-        #
-        # Returns the API type (e.g. "N", "S") for the given column, if the schema defines it,
-        # nil otherwise
-        #
-        def col_type(col)
-          col = col.to_s
-          col_def = schema[:attribute_definitions].find { |d| d[:attribute_name] == col.to_s }
-          col_def && col_def[:attribute_type]
-        end
-
-        def item_count
-          schema[:item_count]
-        end
-
-        def name
-          schema[:table_name]
-        end
-      end
-
-      #
-      # Mimics behavior of the yielded object on DynamoDB's update_item API (high level).
-      #
-      class ItemUpdater
-        attr_reader :table, :key, :range_key
-
-        def initialize(table, key, range_key = nil)
-          @table = table
-          @key = key
-          @range_key = range_key
-          @additions = {}
-          @deletions = {}
-          @updates   = {}
-        end
-
-        #
-        # Adds the given values to the values already stored in the corresponding columns.
-        # The column must contain a Set or a number.
-        #
-        # @param [Hash] vals keys of the hash are the columns to update, vals are the values to
-        #               add. values must be a Set, Array, or Numeric
-        #
-        def add(values)
-          @additions.merge!(sanitize_attributes(values))
-        end
-
-        #
-        # Removes values from the sets of the given columns
-        #
-        # @param [Hash] values keys of the hash are the columns, values are Arrays/Sets of items
-        #               to remove
-        #
-        def delete(values)
-          @deletions.merge!(sanitize_attributes(values))
-        end
-
-        #
-        # Replaces the values of one or more attributes
-        #
-        def set(values)
-          @updates.merge!(sanitize_attributes(values))
-        end
-
-        #
-        # Returns an AttributeUpdates hash suitable for passing to the V2 Client API
-        #
-        def to_h
-          ret = {}
-
-          @additions.each do |k, v|
-            ret[k.to_s] = {
-              action: ADD,
-              value: v
-            }
-          end
-          @deletions.each do |k, v|
-            ret[k.to_s] = {
-              action: DELETE,
-              value: v
-            }
-          end
-          @updates.each do |k, v|
-            ret[k.to_s] = {
-              action: PUT,
-              value: v
-            }
-          end
-
-          ret
-        end
-
-        private
-
-        def sanitize_attributes(attributes)
-          attributes.transform_values do |v|
-            v.is_a?(Hash) ? v.stringify_keys : v
-          end
-        end
-
-        ADD    = 'ADD'
-        DELETE = 'DELETE'
-        PUT    = 'PUT'
-      end
-
       def sanitize_item(attributes)
-        attributes.reject do |_k, v|
+        attributes.reject do |_, v|
           v.nil? || ((v.is_a?(Set) || v.is_a?(String)) && v.empty?)
         end.transform_values do |v|
           v.is_a?(Hash) ? v.stringify_keys : v