shameless 0.5.2 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ada388329571a9d4e7401b2258bd315cd92f9219
-  data.tar.gz: d12bbf07ca0ede9db4f4ea79b21048b0eeeab4b7
+  metadata.gz: 5bc5dbfda6dcf36c8ca5ac5d91394cf1c01b08ec
+  data.tar.gz: f6ce2ddeedc0506c1ca7ec6613a42971eafa86ef
 SHA512:
-  metadata.gz: df9d74db8abe98a8b5437511a35af4e159ae224d6db17e8fe3870acf12311a9bd407221248ca2a081c866b49da903283b37f8f423c6dc62ad11643cb924c006f
-  data.tar.gz: '0638d93b2c824cf7c4fa39b8454ec13ef8ae667f159a6ed8fe691b441334c6731c34f57022ce7a1d9297834ebae02b985897914a7e6ea57e4293212b668c2997'
+  metadata.gz: 42d218c63e8ef57dd1aa5b6a6f2a69db89f39720c8935f26e0d8e47e3872c4a21a5449915500520cbe109750028659349444258a795c3ed44fef2178aa7ae3d3
+  data.tar.gz: 762b0ce2284870d8518aa17a458746b6d0e80641af8f5c5f94ca4f25f386c7ea3481d8ce82598379c9848087ec4e4387b060fc775e3f8e43ba462555f93bacf9

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 ### Unreleased
 
+### 0.6.0 (2017-07-26)
+
+* Require `securerandom` explicitly
+* Add support for queries using Sequel's virtual rows
+
 ### 0.5.2 (2016-12-02)
 
 * Eagerly initialize `Model#cells`

data/README.md

@@ -8,29 +8,29 @@ Shameless is an implementation of a schemaless, distributed, append-only store b
 
 Shameless was born out of our need to have highly scalable, distributed storage for hotel rates. Rates are a way hotels package their rooms, they typically include check-in and check-out date, room type, rate plan, net price, discount, extra services, etc. Our original solution of storing rates in a typical relational SQL table was reaching its limits due to write congestion, migration anxiety, and high maintenance.
 
-Hotel rates change very frequently, so our solution needed to have consistent write latency. There are also mutliple agents mutating various aspects of those rates, so we wanted something that would enable versioning. We also wanted to avoid having to create migrations whenever we were adding more data to rates.
+Hotel rates change very frequently, so our solution needed to have consistent write latency. There are also multiple agents mutating various aspects of those rates, so we wanted something that would enable versioning. We also wanted to avoid having to create migrations whenever we were adding more data to rates.
 
 ## Concept
 
 The whole idea of Shameless is to split a regular SQL table into index tables and content tables. Index tables map the fields you want to query by to UUIDs, content tables map UUIDs to model contents (bodies). In addition, both index and content tables are sharded.
 
-The body of the model is schema-less, you can store an arbitrary data structures. Under the hood, the body is serialized using MessagePack and stored as a blob in a single database column (hence the need for index tables).
+The body of the model is schema-less, you can store arbitrary data structures in it. Under the hood, the body is serialized using MessagePack and stored as a blob in a single database column (hence the need for index tables).
 
 The process of querying for records can be described as:
 
 1. Query the index tables by index fields (e.g. hotel ID, check-in date, and length of stay), sharded by hotel ID, getting get back a list of UUIDs
-2. Query the content tables, sharded by UUID, for most recent version of model
+1. Query the content tables, sharded by UUID, for most recent version of model
 
 Inserting a record is similar:
 
 1. Generate a UUID
-2. Serialize and write model content into appropriate shard of the content tables
-2. Insert a row (index fields + model UUID) to the appropriate shard of the index table
+1. Serialize and write model content into appropriate shard of the content tables
+1. Insert a row (index fields + model UUID) to the appropriate shard of the index table
 
 Inserting a new version of an existing record is even simpler:
 
 1. Increment version
-2. Serialize and write model content into appropriate shard of the content tables
+1. Serialize and write model content into appropriate shard of the content tables
 
 Naturally, shameless hides all that complexity behind a straight-forward API.
 
@@ -46,12 +46,13 @@ The core object of shameless is a `Store`. Here's how you can set one up:
 RateStore = Shameless::Store.new(:rate_store) do |c|
   c.partition_urls = [ENV['RATE_STORE_DATABASE_URL_0'], ENV['RATE_STORE_DATABASE_URL_1']
   c.shards_count = 512 # total number of shards across all partitions
-  c.connection_options = {max_connections: 10} # connection options passed to Sequel.connect
+  c.connection_options = {max_connections: 10} # connection options passed to `Sequel.connect`
   c.database_extensions = [:newrelic_instrumentation]
+  c.create_table_options = {engine: "InnoDB"} # passed to Sequel's `create_table`
 end
 ```
 
-The initializer argument (`:rate_store`) defines the namespace by which all tables will be prefixed, in this case `rate_store_`.
+The initializer argument (`:rate_store`) defines the namespace by which all tables will be prefixed, in this case `rate_store_`. If you pass `nil`, there will be no prefix.
 
 Once you've got the Store configured, you can declare models.
 
@@ -135,7 +136,9 @@ end
 
 ### Reading/writing
 
-To insert and query the model, use `Model.put` and `Model.where`:
+To write data to the model, use `Model.put` `Model#save`, `Cell#save`, `Model#update`, or `Cell#update`. `Model.put` will perform an "upsert", i.e. it will try to find an existing record with the given index fields, and insert a new version for that record's base cell if it finds one, or pick a new UUID, write the first version of the base cell, and write to all indices otherwise.
+
+Here are some examples of how you can read and write data from/to a shameless store:
 
 ```ruby
 # Writing - all index fields are required, the rest is the schemaless content
@@ -146,16 +149,85 @@ rate[:net_price] # => 120.0 # access in the "base" cell
 rate[:net_price] = 130.0
 rate.save
 
+# You can also access the "base" cell explicitly
+rate.base[:net_price] = 140.0
+rate.base.save
+
 # Reading from/writing to a different cell is simple, too:
 rate.meta[:hotel_enabled] = true
 rate.meta.save
 
+# You can also do that in one go using `Model#update` or `Cell#update`. This writes a new
+# version of the cell, merging the hash passed in as parameter with existing values.
+rate.update(tax_rate: 11.0, gateway: 'pegasus')
+rate.body # => {net_price: 140.0, tax_rate: 11.0, gateway: 'pegasus'}
+
+rate.meta.update(hotel_enabled: false)
+```
+
+To query, use `Model.where` (also using Sequel's [virtual row blocks](http://sequel.jeremyevans.net/rdoc/files/doc/virtual_rows_rdoc.html)):
+
+```ruby
 # Querying by primary index
 rates = Rate.where(hotel_id: 1, room_type: '1 bed', check_in_date: Date.today)
 
 # Querying by a named index
 rates = Rate.secondary_index.where(hotel_id: 1, gateway: 'pegasus', discount_type: 'geo')
 rates.first[:net_price] # => 130.0
+
+# Query using Sequel's virtual row block (handy for inequality operators)
+rates = Rate.where(hotel_id: 1, room_type: '1 bed') { check_in_date > Date.today }
+```
+
+To access a cell field that you're not sure has a value, you can use and `Cell#fetch` (`Model#fetch` delegates to the base cell) to get a value from a cell, or a default, e.g.:
+
+```ruby
+rate[:net_price] = 130.0
+rate.fetch(:net_price, 100) # => 130.0
+rate.meta.fetch(:enabled, true) # => true
+```
+
+Cells are versioned, the current version is stored in a column called `ref_key`. The first version of a cell has a `ref_key` of zero. To access a previous version of a cell, use `Cell#previous` (`Model#previous` delegates to the base cell). Example:
+
+```ruby
+# ...
+rate[:net_price] # => 120.0
+rate.ref_key # => 1
+rate.update(net_price: 130.0)
+rate.ref_key # => 2
+rate.previous[:net_price] # => 120.0
+rate.previous.ref_key # => 1
+rate.previous.previous.ref_key # => 0
+rate.previous.previous.previous # => nil
+```
+
+It could happen that another process may have updated a model/cell. To fetch the latest state, use `Cell#reload` (`Model#reload` reloads *all* cells), e.g.:
+
+```ruby
+rate[:net_price] # => 120.0
+
+# Another process updates the cell
+Rate.where(hotel_id: rate[:hotel_id]).first.update(net_price: 130.0)
+
+rate[:net_price] # => 120.0
+rate.reload
+rate[:net_price] # => 130.0
+```
+
+To check if a given cell exists, use `Cell#present?` (as you can suspect, `Model#present?` delegates to the base cell). You can also use `Model#cells` to iterate over all cells, e.g.:
+
+```ruby
+rate.present? # => true
+rate.meta.present? # => false
+
+rate.cells.any?(&:present?) # => true
+```
+
+To see the cell's full state (body + metadata), use `Cell#as_json` (`Model#as_json` delegates to the base cell), e.g.:
+
+```ruby
+rate.as_json # => {id: 123, uuid: "...", created_at: "...", column_name: "base", ref_key: 3,
+  # body: {hotel_id: 1, check_in_date: "2017-01-03", room_type: "ROH", net_price: 130.0}}
 ```
 
 ### Creating tables
@@ -168,6 +240,18 @@ RateStore.create_tables!
 
 This will create the underlying index tables, content tables, together with database indices for fast access.
 
+### Concurrent writes
+
+Since writes to shameless aren't atomic, concurrency control needs to be moved to application code. We're using a setup where almost all our writes go through queues, one queue per shard. We're using `Store#each_shard` to match queue names to shards and to aggregate queue stats across all shards.
+
+### Using shameless as a data stream store (similar to Kafka)
+
+Thanks to storing each write to shameless as a new record in the underlying database table, we're able to use our shameless store as a log for stream processing. For each shard, we have a worker that goes through all new entries in that shard and triggers various event processors to handle all kinds of asynchronous work. For that purpose, we're using `Model.fetch_latest_cells(shard:, cursor:, limit:)`, and incrementing a cursor (stored in Redis) after each record has been processed successfully. We're using the cells' IDs as cursors, using `Cell#id`, which returns the underlying table's primary key value.
+
+### Utilities
+
+Sometimes it may be useful to know where a given model will end up, based on its shardable value. For this, you can use `Store#find_shard(shardable_value)`, e.g.: `RateStore.find_shard(hotel.id) # => 196`.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -194,7 +278,6 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/hoteltonight/shameless.
 
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/lib/shameless/configuration.rb

@@ -3,6 +3,8 @@ module Shameless
     attr_accessor :partition_urls, :shards_count, :connection_options, :database_extensions,
       :create_table_options
 
+    # Needed to deal with our legacy schema that stores created_at as an integer timestamp
+    # and does date conversions in Ruby-land, don't set to `true` for new projects
     attr_accessor :legacy_created_at_is_bigint
 
     def shards_per_partition_count

data/lib/shameless/index.rb

@@ -36,10 +36,10 @@ module Shameless
       @model.store.put(table_name, shardable_value, index_values)
     end
 
-    def where(query)
+    def where(query, &block)
       shardable_value = query.fetch(@shard_on).to_i
       query = index_values(query, false)
-      @model.store.where(table_name, shardable_value, query).map {|r| @model.new(r[:uuid]) }
+      @model.store.where(table_name, shardable_value, query, &block).map {|r| @model.new(r[:uuid]) }
     end
 
     def table_name

data/lib/shameless/model.rb

@@ -1,3 +1,4 @@
+require 'securerandom'
 require 'shameless/index'
 require 'shameless/cell'
 
@@ -98,8 +99,8 @@ module Shameless
       @indices.each(&:create_tables!)
     end
 
-    def where(query)
-      primary_index.where(query)
+    def where(query, &block)
+      primary_index.where(query, &block)
     end
 
     def reject_index_values(values)

data/lib/shameless/store.rb

@@ -22,8 +22,8 @@ module Shameless
       find_table(table_name, shardable_value).insert(values)
     end
 
-    def where(table_name, shardable_value, query)
-      find_table(table_name, shardable_value).where(query)
+    def where(table_name, shardable_value, query, &block)
+      find_table(table_name, shardable_value).where(query, &block)
     end
 
     def disconnect

data/lib/shameless/version.rb

@@ -1,3 +1,3 @@
 module Shameless
-  VERSION = "0.5.2"
+  VERSION = "0.6.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: shameless
 version: !ruby/object:Gem::Version
-  version: 0.5.2
+  version: 0.6.0
 platform: ruby
 authors:
 - Olek Janiszewski
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-12-02 00:00:00.000000000 Z
+date: 2017-07-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -140,7 +140,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 2.6.11
 signing_key: 
 specification_version: 4
 summary: Scalable distributed append-only data store