activemodel-datastore 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 4d8d2d055a0934bc97f36c9c46a0ef375d30aa4d
-  data.tar.gz: 9640f7bb31c42d4a9c0f7e5d981cd75a20c847a7
+  metadata.gz: b11457bc388803b1f429027dd976ac9ed10250c9
+  data.tar.gz: 52247ef1a1e9b49ee738d756374c441b00bd9488
 SHA512:
-  metadata.gz: fcf58339c5a1e7afc910fc5c3db2b7334af33d194c38966f0b3b875c9b7cec36e6ce83f5a2514b4e0673b92bc50892a981658e912db6bff5a2ab0891cc0753c4
-  data.tar.gz: 13fb3b0621cd3b906156c0ef4e027c54edd20dc6b1cf98a9d34250961e45fa60ea4e89e34a5892f6892e4f5c8cdeb92138a3d1ec9caffb948a6f1afbc6179844
+  metadata.gz: 8677a69f2faafa802c634829f01a86cc438295f157b71da0d431fe7bc3175f22154ecd03696a6d93444df42db86bd9d459d62271f5722b0b091c10d83e2bffa8
+  data.tar.gz: 4caa657d528f99eb06963e028224a489ee3c9c3d44460b53336383422db86e8d3733f18c940854b75307cdf57ccb2a343a09d93d67084007332c02b5656fae27

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+### 0.2.0 / 2017-04-19
+
+* many documentation improvements
+* adding support for creating entity groups through either a parent or parent_key_id
+* example Rails 5 app
+
 ### 0.1.0 / 2017-03-27
 
 Initial release.

data/README.md

@@ -10,6 +10,8 @@ with Rails?
 When you want a Rails app backed by a managed, massively-scalable datastore solution. Cloud Datastore 
 automatically handles sharding and replication, providing you with a highly available and durable 
 database that scales automatically to handle your applications' load.
+
+[![Gem Version](https://badge.fury.io/rb/activemodel-datastore.svg)](https://badge.fury.io/rb/activemodel-datastore)
  
 ## Table of contents
  
@@ -17,10 +19,13 @@ database that scales automatically to handle your applications' load.
 - [Model Example](#model)
 - [Controller Example](#controller)
 - [Retrieving Entities](#queries)
+- [Datastore Consistency](#consistency)
+- [Datastore Indexes](#indexes)
+- [Datastore Emulator](#emulator)
 - [Example Rails App](#rails)
-- [Development and Test](#development)
+- [Track Changes](#track_changes)
 - [Nested Forms](#nested)
-- [Work In Progress](#wip)
+- [Datastore Gotchas](#gotchas)
  
 ## <a name="setup"></a>Setup
  
@@ -30,15 +35,22 @@ Generate your Rails app without ActiveRecord:
 rails new my_app -O
 ```
 
+You can remove the db/ directory as it won't be needed.
+
 To install, add this line to your `Gemfile` and run `bundle install`:
  
 ```ruby
 gem 'activemodel-datastore'
 ```
   
-Google Cloud requires a Project ID and Service Account Credentials to connect to the Datastore API.
+Create a Google Cloud account [here](https://cloud.google.com) and create a project.
+
+Google Cloud requires the Project ID and Service Account Credentials to connect to the Datastore API.
  
-*Follow the [activation instructions](https://cloud.google.com/datastore/docs/activate) to use the Google Cloud Datastore API.*
+*Follow the [activation instructions](https://cloud.google.com/datastore/docs/activate) to enable the 
+Google Cloud Datastore API. When running on Google Cloud Platform environments the Service Account 
+credentials will be discovered automatically. When running on other environments (such as AWS or Heroku)
+you need to create a service account with the role of editor and generate json credentials.*
 
 Set your project id in an `ENV` variable named `GCLOUD_PROJECT`.
 
@@ -48,9 +60,10 @@ To locate your project ID:
 2. From the projects list, select the name of your project.
 3. On the left, click Dashboard. The project name and ID are displayed in the Dashboard.
 
-When running on Google Cloud Platform environments the Service Account credentials will be discovered automatically. 
-When running on other environments (such as AWS or Heroku), the Service Account credentials need to be 
-specified in two additional `ENV` variables named `SERVICE_ACCOUNT_CLIENT_EMAIL` and `SERVICE_ACCOUNT_PRIVATE_KEY`.
+If you have an external application running on a platform outside of Google Cloud you also need to 
+provide the Service Account credentials. They are specified in two additional `ENV` variables named 
+`SERVICE_ACCOUNT_CLIENT_EMAIL` and `SERVICE_ACCOUNT_PRIVATE_KEY`. The values for these two `ENV` 
+variables will be in the downloaded service account json credentials file.
 
 ```bash
 SERVICE_ACCOUNT_PRIVATE_KEY = -----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\n
@@ -58,6 +71,11 @@ SERVICE_ACCOUNT_CLIENT_EMAIL = web-app@app-name.iam.gserviceaccount.com
 ```
 
 On Heroku the `ENV` variables can be set under 'Settings' -> 'Config Variables'.
+
+Active Model Datastore will then handle the authentication for you, and the datastore instance can 
+be accessed with `CloudDatastore.dataset`.
+
+There is an example Puma config file [here](https://github.com/Agrimatics/activemodel-datastore/blob/master/test/support/datastore_example_rails_app/config/puma.rb).
  
 ## <a name="model"></a>Model Example
  
@@ -67,21 +85,71 @@ Let's start by implementing the model:
 class User
   include ActiveModel::Datastore
 
-  attr_accessor :email, :name, :enabled, :state
+  attr_accessor :email, :enabled, :name, :role, :state
+
+  def entity_properties
+    %w[email enabled name role]
+  end
+end
+```
+
+Data objects in Cloud Datastore are known as entities. Entities are of a kind. An entity has one 
+or more named properties, each of which can have one or more values. Think of them like this:
+* 'Kind' (which is your table and the name of your Rails model)
+* 'Entity' (which is the record from the table)
+* 'Property' (which is the attribute of the record)
+
+The `entity_properties` method defines an Array of properties that belong to the entity in cloud 
+datastore. Define the attributes of your model using `attr_accessor`. With this approach, Rails 
+deals solely with ActiveModel objects. The objects are converted to/from entities automatically 
+during save/query operations. You can still use virtual attributes on the model (such as the 
+`:state` attribute above) by simply excluding it from `entity_properties`. In this example state 
+is available to the model but won't be persisted with the entity in datastore.
+
+Validations work as you would expect:
+
+```ruby
+class User
+  include ActiveModel::Datastore
+
+  attr_accessor :email, :enabled, :name, :role, :state
+
+  validates :email, format: { with: /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i }
+  validates :name, presence: true, length: { maximum: 30 }
+
+  def entity_properties
+    %w[email enabled name role]
+  end
+end
+```
+
+Callbacks work as you would expect. We have also added the ability to set default values through 
+[`default_property_value`](http://www.rubydoc.info/gems/activemodel-datastore/ActiveModel%2FDatastore:default_property_value) 
+and type cast the format of values through [`format_property_value`](http://www.rubydoc.info/gems/activemodel-datastore/ActiveModel%2FDatastore:format_property_value):
+
+```ruby
+class User
+  include ActiveModel::Datastore
+
+  attr_accessor :email, :enabled, :name, :role, :state
 
   before_validation :set_default_values
+  after_validation :format_values
+  
   before_save { puts '** something can happen before save **'}
   after_save { puts '** something can happen after save **'}
 
   validates :email, format: { with: /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i }
   validates :name, presence: true, length: { maximum: 30 }
+  validates :role, presence: true
 
   def entity_properties
-    %w[email name enabled]
+    %w[email enabled name role]
   end
 
   def set_default_values
     default_property_value :enabled, true
+    default_property_value :role, 1
   end
 
   def format_values
@@ -90,21 +158,6 @@ class User
 end
 ```
 
-Using `attr_accessor` the attributes of the model are defined. Validations and Callbacks all work 
-as you would expect. However, `entity_properties` is new. Data objects in Cloud Datastore
-are known as entities. Entities are of a kind. An entity has one or more named properties, each
-of which can have one or more values. Think of them like this:
-* 'Kind' (which is your table)
-* 'Entity' (which is the record from the table)
-* 'Property' (which is the attribute of the record)
-
-The `entity_properties` method defines an Array of the properties that belong to the entity in
-cloud datastore. With this approach, Rails deals solely with ActiveModel objects. The objects are
-converted to/from entities as needed during save/query operations.
-
-We have also added the ability to set default property values and type cast the format of values
-for entities.
-
 ## <a name="controller"></a>Controller Example
 
 Now on to the controller! A scaffold generated controller works out of the box:
@@ -169,6 +222,14 @@ end
 
 ## <a name="queries"></a>Retrieving Entities
 
+Each entity in Cloud Datastore has a key that uniquely identifies it. The key consists of the 
+following components:
+
+* the kind of the entity, which is User in these examples
+* an identifier for the individual entity, which can be either a a key name string or an integer numeric ID
+* an optional ancestor path locating the entity within the Cloud Datastore hierarchy
+
+#### [all(options = {})](http://www.rubydoc.info/gems/activemodel-datastore/ActiveModel%2FDatastore%2FClassMethods:all)
 Queries entities using the provided options. When a limit option is provided queries up to the limit 
 and returns results with a cursor.
 ```ruby
@@ -194,8 +255,9 @@ users, cursor = User.all(limit: 7)
 # @option options [Array] :where Adds a property filter of arrays in the format[name, operator, value].
 ```
 
+#### [find(*ids, parent: nil)](http://www.rubydoc.info/gems/activemodel-datastore/ActiveModel%2FDatastore%2FClassMethods:find)
 Find entity by id - this can either be a specific id (1), a list of ids (1, 5, 6), or an array of ids ([5, 6, 10]). 
-The parent key is optional.
+The parent key is optional. This method is a lookup by key and results will be strongly consistent.
 ```ruby
 user = User.find(1)
 
@@ -205,18 +267,161 @@ user = User.find(1, parent: parent)
 users = User.find(1, 2, 3)
 ```
 
-Finds the first entity matching the specified condition.
+#### [find_by(args)](http://www.rubydoc.info/gems/activemodel-datastore/ActiveModel%2FDatastore%2FClassMethods:find_by)
+Queries for the first entity matching the specified condition.
 ```ruby
 user = User.find_by(name: 'Joe')
 
 user = User.find_by(name: 'Bryce', ancestor: parent)
 ```
 
-## <a name="rails"></a>Example Rails App
+Cloud Datastore has documentation on how [Datastore Queries](https://cloud.google.com/datastore/docs/concepts/queries#datastore-basic-query-ruby) 
+work, and pay special attention to the the [restrictions](https://cloud.google.com/datastore/docs/concepts/queries#restrictions_on_queries).
+
+## <a name="consistency"></a>Datastore Consistency
+
+Cloud Datastore is a non-relational databases, or NoSQL database. It distributes data over many 
+machines and uses synchronous replication over a wide geographic area. Because of this architecture 
+it offers a balance of strong and eventual consistency.
+
+What is eventual consistency?
+
+It means that an updated entity value may not be immediately visible when executing a query. 
+Eventual consistency is a theoretical guarantee that, provided no new updates to an entity are made, 
+all reads of the entity will eventually return the last updated value.
+
+In the context of a Rails app, there are times that eventual consistency is not ideal. For example,
+let's say you create a user entity with a key that looks like this:
+
+`@key=#<Google::Cloud::Datastore::Key @kind="User", @id=1>`
+
+and then immediately redirect to the index view of users. There is a good chance that your new user 
+is not yet visible in the list. If you perform a refresh on the index view a second or two later 
+the user will appear.
+
+"Wait a minute!" you say. "This is crap!" you say. Fear not! We can make the query of users strongly
+consistent. We just need to use entity groups and ancestor queries. An entity group is a hierarchy 
+formed by a root entity and its children. To create an entity group, you specify an ancestor path 
+for the entity which is a parent key as part of the child key.
 
-There is an example Rails 5 app in the test directory [here](https://github.com/Agrimatics/activemodel-datastore/tree/master/test/support/datastore_example_rails_app) 
+Before using the `save` method, assign the `parent_key_id` attribute an ID. Let's say that 12345 
+represents the ID of the company that the users belong to. The key of the user entity will now 
+look like this:
 
-## <a name="development"></a>Development and Test
+`@key=#<Google::Cloud::Datastore::Key @kind="User", @id=1, @parent=#<Google::Cloud::Datastore::Key @kind="ParentUser", @id=12345>>`
+
+All of the User entities will now belong to an entity group named ParentUser and can be queried by the 
+Company ID. When we query for the users we will provide User.parent_key(12345) as the ancestor option.
+ 
+*Ancestor queries are always strongly consistent.*
+
+However, there is a small downside. Entities with the same ancestor are limited to 1 write per second.
+Also, the entity group relationship cannot be changed after creating the entity (as you can't modify 
+an entity's key after it has been saved).
+
+The Users controller would now look like this:
+
+```ruby
+class UsersController < ApplicationController
+  before_action :set_user, only: [:show, :edit, :update, :destroy]
+
+  def index
+    @users = User.all(ancestor: User.parent_key(12345))
+  end
+
+  def show
+  end
+
+  def new
+    @user = User.new
+  end
+
+  def edit
+  end
+
+  def create
+    @user = User.new(user_params)
+    @user.parent_key_id = 12345
+    respond_to do |format|
+      if @user.save
+        format.html { redirect_to @user, notice: 'User was successfully created.' }
+      else
+        format.html { render :new }
+      end
+    end
+  end
+
+  def update
+    respond_to do |format|
+      if @user.update(user_params)
+        format.html { redirect_to @user, notice: 'User was successfully updated.' }
+      else
+        format.html { render :edit }
+      end
+    end
+  end
+
+  def destroy
+    @user.destroy
+    respond_to do |format|
+      format.html { redirect_to users_url, notice: 'User was successfully destroyed.' }
+    end
+  end
+
+  private
+
+  def set_user
+    @user = User.find(params[:id], parent: User.parent_key(12345))
+  end
+
+  def user_params
+    params.require(:user).permit(:email, :name)
+  end
+end
+```
+
+See here for the Cloud Datastore documentation on [Data Consistency](https://cloud.google.com/datastore/docs/concepts/structuring_for_strong_consistency).
+
+## <a name="indexes"></a>Datastore Indexes
+
+Every cloud datastore query requires an index. Yes, you read that correctly. Every single one. The 
+indexes contain entity keys in a sequence specified by the index's properties and, optionally, 
+the entity's ancestors.
+
+There are two types of indexes, *built-in* and *composite*.
+
+#### Built-in
+By default, Cloud Datastore automatically predefines an index for each property of each entity kind. 
+These single property indexes are suitable for simple types of queries. These indexes are free and
+do not count against your index limit.
+
+#### Composite
+Composite index multiple property values per indexed entity. Composite indexes support complex 
+queries and are defined in an index.yaml file.
+
+Composite indexes are required for queries of the following form:
+
+* queries with ancestor and inequality filters
+* queries with one or more inequality filters on a property and one or more equality filters on other properties
+* queries with a sort order on keys in descending order
+* queries with multiple sort orders
+* queries with one or more filters and one or more sort orders
+
+*NOTE*: Inequality filters are LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREATER_THAN_OR_EQUAL.
+
+Google has excellent doc regarding datastore indexes [here](https://cloud.google.com/datastore/docs/concepts/indexes).
+
+The datastore emulator generates composite indexes in an index.yaml file automatically. The file
+can be found in /tmp/local_datastore/WEB-INF/index.yaml. If your localhost Rails app exercises every 
+possible query the application will issue, using every combination of filter and sort order, the 
+generated entries will represent your complete set of indexes.
+
+One thing to note is that the datastore emulator caches indexes. As you add and modify application 
+code you might find that the local datastore index.yaml contains indexes that are no longer needed. 
+In this scenario try deleting the index.yaml and restarting the emulator. Navigate through your Rails
+app and the index.yaml will be built from scratch.
+
+## <a name="emulator"></a>Datastore Emulator
 
 Install the Google Cloud SDK.
 
@@ -250,8 +455,26 @@ To create the local test datastore execute the following from the root of the pr
 
 To start the local Cloud Datastore emulator:
 
-    $ ./start-local-datastore.sh
+    $ cloud_datastore_emulator start --port=8180 tmp/local_datastore
     
+## <a name="rails"></a>Example Rails App
+
+There is an example Rails 5 app in the test directory [here](https://github.com/Agrimatics/activemodel-datastore/tree/master/test/support/datastore_example_rails_app).
+
+ ```bash
+ $ bundle
+ $ cloud_datastore_emulator create tmp/local_datastore
+ $ cloud_datastore_emulator create tmp/test_datastore
+ $ ./start-local-datastore.sh
+ $ rails s
+ ```
+ 
+ Navigate to http://localhost:3000.
+
+## <a name="track_changes"></a>Track Changes
+
+TODO: document the change tracking implementation.
+
 ## <a name="nested"></a>Nested Forms
 
 Adds support for nested attributes to ActiveModel. Heavily inspired by 
@@ -260,7 +483,7 @@ Rails ActiveRecord::NestedAttributes.
 Nested attributes allow you to save attributes on associated records along with the parent.
 It's used in conjunction with fields_for to build the nested form elements.
 
-See Rails ActionView::Helpers::FormHelper::fields_for for more info.
+See Rails [ActionView::Helpers::FormHelper::fields_for](http://api.rubyonrails.org/classes/ActionView/Helpers/FormHelper.html#method-i-fields_for) for more info.
 
 *NOTE*: Unlike ActiveRecord, the way that the relationship is modeled between the parent and
 child is not enforced. With NoSQL the relationship could be defined by any attribute, or with
@@ -329,12 +552,9 @@ Within the parent model `valid?` will validate the parent and associated childre
 a truthy `_destroy` key, the appropriate nested_models will have `marked_for_destruction` set
 to True.
 
-## <a name="wip"></a>Work In Progress
-
-TODO: document datastore eventual consistency and mitigation using ancestor queries and entity groups.
-
-TODO: document indexes.
-
-TODO: document using the datastore emulator to generate the index.yaml.
-
-TODO: document the change tracking implementation.
+## <a name="gotchas"></a>Datastore Gotchas
+#### Ordering of query results is undefined when no sort order is specified.
+When a query does not specify a sort order, the results are returned in the order they are retrieved. 
+As Cloud Datastore implementation evolves (or if a project's indexes change), this order may change. 
+Therefore, if your application requires its query results in a particular order, be sure to specify 
+that sort order explicitly in the query.

data/lib/active_model/datastore.rb

@@ -9,21 +9,25 @@
 #   class User
 #     include ActiveModel::Datastore
 #
-#     attr_accessor :email, :name, :enabled, :state
+#     attr_accessor :email, :enabled, :name, :role, :state
 #
 #     before_validation :set_default_values
-#     before_save { puts '** something can happen before save **' }
-#     after_save { puts '** something can happen after save **' }
+#     after_validation :format_values
+#
+#     before_save { puts '** something can happen before save **'}
+#     after_save { puts '** something can happen after save **'}
 #
 #     validates :email, format: { with: /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i }
 #     validates :name, presence: true, length: { maximum: 30 }
+#     validates :role, presence: true
 #
 #     def entity_properties
-#       %w[email name enabled]
+#       %w[email enabled name role]
 #     end
 #
 #     def set_default_values
 #       default_property_value :enabled, true
+#       default_property_value :role, 1
 #     end
 #
 #     def format_values
@@ -116,13 +120,20 @@ module ActiveModel::Datastore
   included do
     private_class_method :query_options, :query_sort, :query_property_filter, :find_all_entities
     define_model_callbacks :save, :update, :destroy
-    attr_accessor :id
+    attr_accessor :id, :parent_key_id
   end
 
   def entity_properties
     []
   end
 
+  ##
+  # Used to determine if the ActiveModel object belongs to an entity group.
+  #
+  def parent?
+    parent_key_id.present?
+  end
+
   ##
   # Used by ActiveModel for determining polymorphic routing.
   #
@@ -177,11 +188,18 @@ module ActiveModel::Datastore
   ##
   # Builds the Cloud Datastore entity with attributes from the Model object.
   #
+  # @param [Google::Cloud::Datastore::Key] parent An optional parent Key of the entity.
+  #
   # @return [Entity] The updated Google::Cloud::Datastore::Entity.
   #
   def build_entity(parent = nil)
     entity = CloudDatastore.dataset.entity self.class.name, id
-    entity.key.parent = parent if parent.present?
+    if parent.present?
+      raise ArgumentError, 'Must be a Key' unless parent.is_a? Google::Cloud::Datastore::Key
+      entity.key.parent = parent
+    elsif parent?
+      entity.key.parent = self.class.parent_key(parent_key_id)
+    end
     entity_properties.each do |attr|
       entity[attr] = instance_variable_get("@#{attr}")
     end
@@ -194,17 +212,6 @@ module ActiveModel::Datastore
 
   ##
   # For compatibility with libraries that require the bang method version (example, factory_girl).
-  # If you require a save! method that supports parents (ancestor queries), override this method
-  # in your own code with something like this:
-  #
-  #   def save!
-  #     parent = nil
-  #     if account_id.present?
-  #       parent = CloudDatastore.dataset.key 'Parent' + self.class.name, account_id.to_i
-  #     end
-  #     msg = 'Failed to save the entity'
-  #     save_entity(parent) || raise(ActiveModel::Datastore::EntityNotSavedError, msg)
-  #   end
   #
   def save!
     save_entity || raise(EntityNotSavedError, 'Failed to save the entity')
@@ -222,6 +229,7 @@ module ActiveModel::Datastore
   def destroy
     run_callbacks :destroy do
       key = CloudDatastore.dataset.key self.class.name, id
+      key.parent = self.class.parent_key(parent_key_id) if parent?
       self.class.retry_on_exception? { CloudDatastore.dataset.delete key }
     end
   end
@@ -234,12 +242,20 @@ module ActiveModel::Datastore
       entity = build_entity(parent)
       success = self.class.retry_on_exception? { CloudDatastore.dataset.save entity }
       self.id = entity.key.id if success
+      self.parent_key_id = entity.key.parent.id if entity.key.parent.present?
       success
     end
   end
 
   # Methods defined here will be class methods when 'include ActiveModel::Datastore'.
   module ClassMethods
+    ##
+    # A default parent key for specifying an ancestor path and creating an entity group.
+    #
+    def parent_key(parent_id)
+      CloudDatastore.dataset.key('Parent' + name, parent_id.to_i)
+    end
+
     ##
     # Retrieves an entity by id or name and by an optional parent.
     #
@@ -356,7 +372,7 @@ module ActiveModel::Datastore
     #
     # @example
     #   User.find_by(name: 'Joe')
-    #   User.find_by(name: 'Bryce', ancestor: parent)
+    #   User.find_by(name: 'Bryce', ancestor: parent_key)
     #
     def find_by(args)
       query = CloudDatastore.dataset.query name
@@ -391,6 +407,7 @@ module ActiveModel::Datastore
       model_entity = new
       model_entity.id = entity.key.id unless entity.key.id.nil?
       model_entity.id = entity.key.name unless entity.key.name.nil?
+      model_entity.parent_key_id = entity.key.parent.id if entity.key.parent.present?
       entity.properties.to_hash.each do |name, value|
         model_entity.send "#{name}=", value if model_entity.respond_to? "#{name}="
       end

data/lib/active_model/datastore/version.rb

@@ -1,5 +1,5 @@
 module ActiveModel
   module Datastore
-    VERSION = '0.1.0'
+    VERSION = '0.2.0'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activemodel-datastore
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Bryce McLean
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-04-10 00:00:00.000000000 Z
+date: 2017-04-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel