shameless 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f6279a53b06a5c2ceeb839ecc50eca06951c12f0
+  data.tar.gz: d8dd4210688a3f72e1ea7574ceeb9f2cb690796d
+SHA512:
+  metadata.gz: 3f69323c0db8d7fbf99018da32f743c6155f98766bd021d92edd9d1932d30294a0c145ca1be5514f2d6a9883ebde836f0e58d5f9fb3aca8fa278447d0e971621
+  data.tar.gz: d0843089e84f7cc1125166338d1fa38bd8dca4cd5cc3b4fede32cdf79b11854bc86a287e811c9111416b8f9c8111b74cfcfae70c284a6659ae4756b82525901a

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.12.5

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 HotelTonight
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,204 @@
+# Shameless
+
+Shameless is an implementation of a schemaless, distributed, append-only store built on top of MySQL and the Sequel gem. It was extracted from a battle-tested codebase of our main application at HotelTonight. Since it's using Sequel for database access, it could work on any database, e.g. postgres, although we've only used it with MySQL.
+
+## Background
+
+Shameless was born out of our need to have highly scalable, distributed storage for hotel rates. Rates are a way hotels package their rooms, they typically include check-in and check-out date, room type, rate plan, net price, discount, extra services, etc. Our original solution of storing rates in a typical relational SQL table was reaching its limits due to write congestion, migration anxiety, and high maintenance.
+
+Hotel rates change very frequently, so our solution needed to have consistent write latency. There are also mutliple agents mutating various aspects of those rates, so we wanted something that would enable versioning. We also wanted to avoid having to create migrations whenever we were adding more data to rates.
+
+## Concept
+
+The whole idea of Shameless is to split a regular SQL table into index tables and content tables. Index tables map the fields you want to query by to UUIDs, content tables map UUIDs to model contents (bodies). In addition, both index and content tables are sharded.
+
+The body of the model is schema-less, you can store an arbitrary data structures. Under the hood, the body is serialized using MessagePack and stored as a blob in a single database column (hence the need for index tables).
+
+The process of querying for records can be described as:
+
+1. Query the index tables by index fields (e.g. hotel ID, check-in date, and length of stay), sharded by hotel ID, getting get back a list of UUIDs
+2. Query the content tables, sharded by UUID, for most recent version of model
+
+Inserting a record is similar:
+
+1. Generate a UUID
+2. Serialize and write model content into appropriate shard of the content tables
+2. Insert a row (index fields + model UUID) to the appropriate shard of the index table
+
+Inserting a new version of an existing record is even simpler:
+
+1. Increment version
+2. Serialize and write model content into appropriate shard of the content tables
+
+Naturally, shameless hides all that complexity behind a straight-forward API.
+
+## Usage
+
+### Creating a store
+
+The core object of shameless is a `Store`. Here's how you can set one up:
+
+```ruby
+# config/initializers/rate_store.rb
+
+RateStore = Shameless::Store.new(:rate_store) do |c|
+  c.partition_urls = [ENV['RATE_STORE_DATABASE_URL_0'], ENV['RATE_STORE_DATABASE_URL_1']
+  c.shards_count = 512 # total number of shards across all partitions
+end
+```
+
+The initializer argument (`:rate_store`) defines the namespace by which all tables will be prefixed, in this case `rate_store_`.
+
+Once you've got the Store configured, you can declare models.
+
+### Declaring models
+
+Models specify the kinds of entities you want to persist in your store. Models are simple Ruby classes (even anonymous) that you attach to a `Store` using `Store#attach_to(model)`, e.g.:
+
+```ruby
+# app/models/rate.rb
+
+class Rate
+  RateStore.attach(self)
+
+  # ...
+end
+```
+
+By default, this will map to tables called `rate_store_rate_[000000-000511]` by lowercasing the class name. You can also provide the table namespace using a second argument, e.g.:
+
+```ruby
+my_model = Class.new do
+  RateStore.attach(self, :rates)
+end
+```
+
+A model is useless without indices. Let's see how to define them.
+
+### Defining indices
+
+Indices are a crucial component of shameless. They allow us to perform fast lookups for model UUIDs. Here's how you define an index:
+
+```ruby
+class Rate
+  RateStore.attach(self)
+
+  index do
+    integer :hotel_id
+    string :room_type
+    string :check_in_date # at the moment, only integer and string types are supported
+
+    shard_on :hotel_id # required, values need to be numeric
+  end
+end
+```
+
+The default index is called a primary index, the corresponding tables would be called `rate_store_rate_primary_[000000-000511]`. You can add additional indices you'd like to query by:
+
+```ruby
+class Rate
+  RateStore.attach(self)
+
+  index do
+    # ..
+  end
+
+  index :secondary do
+    integer :hotel_id
+    string :gateway
+    string :discount_type
+
+    shard_on :hotel_id
+  end
+end
+```
+
+### Defining cells
+
+Model content is stored in blobs called "cells". You can think of cells as separate model columns that can store rich data structures and can change independently over time. The default cell is called "base" (that's what all model-level accessors delegate to), but you can declare additional cells using `Model.cell`:
+
+```
+class Rate
+  RateStore.attach(self)
+
+  index do
+    # ..
+  end
+
+  cell :meta
+end
+```
+
+### Reading/writing
+
+To insert and query the model, use `Model.put` and `Model.where`:
+
+```ruby
+# Writing - all index fields are required, the rest is the schemaless content
+rate = Rate.put(hotel_id: 1, room_type: '1 bed', check_in_date: Date.today, gateway: 'pegasus', discount_type: 'geo', net_price: 120.0)
+rate[:net_price] # => 120.0 # access in the "base" cell
+
+# Create a new version of the "base" cell
+rate[:net_price] = 130.0
+rate.save
+
+# Reading from/writing to a different cell is simple, too:
+rate.meta[:hotel_enabled] = true
+rate.meta.save
+
+# Querying by primary index
+rates = Rate.where(hotel_id: 1, room_type: '1 bed', check_in_date: Date.today)
+
+# Querying by a named index
+rates = Rate.secondary.where(hotel_id: 1, gateway: 'pegasus', discount_type: 'geo')
+rates.first[:net_price] # => 130.0
+```
+
+### Creating tables
+
+To create all shards for all tables, across all partitions, run:
+
+```ruby
+RateStore.create_tables!
+```
+
+This will create the underlying index tables, content tables, together with database indices for fast access.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'shameless'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install shameless
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/hoteltonight/shameless.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+## Credits
+
+Shameless was inspired by the following resources:
+
+- Uber: https://eng.uber.com/schemaless-part-one
+- FriendFeed: https://backchannel.org/blog/friendfeed-schemaless-mysql
+- Pinterest: https://engineering.pinterest.com/blog/sharding-pinterest-how-we-scaled-our-mysql-fleet

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "shameless"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/shameless.rb

@@ -0,0 +1,2 @@
+require 'shameless/store'
+require 'shameless/version'

data/lib/shameless/cell.rb

@@ -0,0 +1,79 @@
+require 'msgpack'
+
+module Shameless
+  class Cell
+    BASE = 'base'
+
+    def self.base(model, body)
+      new(model, BASE, body)
+    end
+
+    def initialize(model, name, body = nil)
+      @model = model
+      @name = name
+      @body = body
+    end
+
+    def [](key)
+      body[key.to_s]
+    end
+
+    def []=(key, value)
+      @model.prevent_readonly_attribute_mutation!(key)
+      body[key.to_s] = value
+    end
+
+    def save
+      @created_at = Time.now
+      @ref_key ||= 0
+      @ref_key += 1
+      @model.put_cell(cell_values)
+    end
+
+    def ref_key
+      fetch
+      @ref_key
+    end
+
+    def created_at
+      fetch
+      @created_at
+    end
+
+    def body
+      fetch
+      @body
+    end
+
+    private
+
+    def cell_values
+      {
+        uuid: @model.uuid,
+        column_name: @name,
+        ref_key: ref_key,
+        created_at: created_at,
+        body: serialized_body
+      }
+    end
+
+    def serialized_body
+      MessagePack.pack(body)
+    end
+
+    def deserialize_body(body)
+      MessagePack.unpack(body)
+    end
+
+    private
+
+    def fetch
+      if @body.nil?
+        values = @model.fetch_cell(@name)
+        @ref_key = values[:ref_key] if values
+        @created_at = values[:created_at] if values
+        @body = values ? deserialize_body(values[:body]) : {}
+      end
+    end
+  end
+end

data/lib/shameless/configuration.rb

@@ -0,0 +1,13 @@
+module Shameless
+  class Configuration
+    attr_accessor :partition_urls, :shards_count
+
+    def shards_per_partition_count
+      shards_count / partitions_count
+    end
+
+    def partitions_count
+      partition_urls.count
+    end
+  end
+end

data/lib/shameless/errors.rb

@@ -0,0 +1,4 @@
+module Shameless
+  Error = Class.new(StandardError)
+  ReadonlyAttributeMutation = Class.new(Error)
+end

data/lib/shameless/index.rb

@@ -0,0 +1,66 @@
+require 'shameless/errors'
+
+module Shameless
+  class Index
+    PRIMARY = :primary
+
+    attr_reader :name
+
+    def initialize(name, model, &block)
+      @name = name || PRIMARY
+      @model = model
+      instance_eval(&block)
+    end
+
+    DataTypes = %i[integer string]
+
+    DataTypes.each do |type|
+      define_method(type) do |column|
+        self.column(column, type)
+      end
+    end
+
+    def column(name, type)
+      @columns ||= {}
+      @columns[name] = type
+    end
+
+    def shard_on(shard_on)
+      @shard_on = shard_on
+    end
+
+    def put(values)
+      shardable_value = values.fetch(@shard_on)
+      index_values = (@columns.keys + [:uuid]).each_with_object({}) {|column, o| o[column] = values.fetch(column) }
+
+      @model.store.put(table_name, shardable_value, index_values)
+    end
+
+    def where(query)
+      shardable_value = query.fetch(@shard_on)
+      @model.store.where(table_name, shardable_value, query).map {|r| @model.new(r[:uuid]) }
+    end
+
+    def table_name
+      "#{@model.table_name}_#{@name}"
+    end
+
+    def create_tables!
+      @model.store.create_table!(table_name) do |t|
+        @columns.each do |name, type|
+          t.column name, type, null: false
+        end
+
+        t.varchar :uuid, size: 36
+
+        t.index @columns.keys, unique: true
+      end
+    end
+
+    def prevent_readonly_attribute_mutation!(key)
+      if @columns.keys.any? {|c| c.to_s == key.to_s }
+        raise ReadonlyAttributeMutation, "The attribute #{key} cannot be modified because it's part of the #{@name} index"
+      end
+    end
+  end
+end

data/lib/shameless/model.rb

@@ -0,0 +1,129 @@
+require 'shameless/index'
+require 'shameless/cell'
+
+module Shameless
+  module Model
+    attr_reader :store
+
+    def attach_to(store, name)
+      @store = store
+      @name = name || self.name.downcase # TODO use activesupport?
+
+      include(InstanceMethods)
+    end
+
+    def index(name = nil, &block)
+      @indices ||= []
+      index = Index.new(name, self, &block)
+      @indices << index
+
+      define_singleton_method("#{index.name}_index") { index }
+    end
+
+    def cell(name)
+      name = name.to_s
+
+      define_method(name) do
+        @cells ||= {}
+        @cells[name] ||= Cell.new(self, name)
+      end
+    end
+
+    def put(values)
+      uuid = SecureRandom.uuid
+
+      new(uuid, values).tap do |model|
+        model.save
+
+        index_values = values.merge(uuid: uuid)
+        @indices.each {|i| i.put(index_values) }
+      end
+    end
+
+    def put_cell(shardable_value, cell_values)
+      @store.put(table_name, shardable_value, cell_values)
+    end
+
+    def fetch_cell(shardable_value, uuid, cell_name)
+      query = {uuid: uuid, column_name: cell_name}
+
+      @store.where(table_name, shardable_value, query).order(:ref_key).last
+    end
+
+    def table_name
+      "#{@store.name}_#{@name}"
+    end
+
+    def create_tables!
+      @store.create_table!(table_name) do |t|
+        t.primary_key :id
+        t.varchar :uuid, size: 36
+        t.varchar :column_name, null: false
+        t.integer :ref_key, null: false
+        t.mediumblob :body
+        t.datetime :created_at, null: false
+
+        t.index %i[uuid column_name ref_key], unique: true
+      end
+
+      @indices.each(&:create_tables!)
+    end
+
+    def where(query)
+      primary_index.where(query)
+    end
+
+    def prevent_readonly_attribute_mutation!(key)
+      @indices.each {|i| i.prevent_readonly_attribute_mutation!(key) }
+    end
+
+    private
+
+    module InstanceMethods
+      attr_reader :uuid
+
+      def initialize(uuid, base_body = nil)
+        @uuid = uuid
+        @base = Cell.base(self, base_body)
+      end
+
+      def [](field)
+        @base[field]
+      end
+
+      def []=(field, value)
+        @base[field] = value
+      end
+
+      def save
+        @base.save
+      end
+
+      def ref_key
+        @base.ref_key
+      end
+
+      def created_at
+        @base.created_at
+      end
+
+      def put_cell(cell_values)
+        self.class.put_cell(shardable_value, cell_values)
+      end
+
+      def fetch_cell(cell_name)
+       self.class.fetch_cell(shardable_value, uuid, cell_name)
+      end
+
+      def prevent_readonly_attribute_mutation!(key)
+        self.class.prevent_readonly_attribute_mutation!(key)
+      end
+
+      private
+
+      def shardable_value
+        uuid[0, 4].to_i(16)
+      end
+    end
+  end
+end

data/lib/shameless/store.rb

@@ -0,0 +1,69 @@
+require 'sequel'
+require 'shameless/configuration'
+require 'shameless/model'
+
+module Shameless
+  class Store
+    attr_reader :name
+
+    def initialize(name, &block)
+      @name = name
+      @configuration = Configuration.new
+      block.call(@configuration)
+    end
+
+    def attach(model_class, name = nil)
+      model_class.extend(Model)
+      model_class.attach_to(self, name)
+      @models ||= []
+      @models << model_class
+    end
+
+    def put(table_name, shardable_value, values)
+      find_table(table_name, shardable_value).insert(values)
+    end
+
+    def where(table_name, shardable_value, query)
+      find_table(table_name, shardable_value).where(query)
+    end
+
+    def create_tables!
+      @models.each(&:create_tables!)
+    end
+
+    def create_table!(table_name, &block)
+      each_shard do |shard|
+        partition = find_partition_for_shard(shard)
+        sharded_table_name = table_name_with_shard(table_name, shard)
+        partition.create_table(sharded_table_name) { block.call(self) }
+      end
+    end
+
+    private
+
+    def partitions
+      @partitions ||= @configuration.partition_urls.map {|url| Sequel.connect(url) }
+    end
+
+    def each_shard(&block)
+      0.upto(@configuration.shards_count - 1, &block)
+    end
+
+    def table_name_with_shard(table_name, shard)
+      padded_shard = shard.to_s.rjust(6, '0')
+      "#{table_name}_#{padded_shard}"
+    end
+
+    def find_table(table_name, shardable_value)
+      shard = shardable_value % @configuration.shards_count
+      partition = find_partition_for_shard(shard)
+      table_name = table_name_with_shard(table_name, shard)
+      partition.from(table_name)
+    end
+
+    def find_partition_for_shard(shard)
+      partition_index = shard / @configuration.shards_per_partition_count
+      partitions[partition_index]
+    end
+  end
+end

data/lib/shameless/version.rb

@@ -0,0 +1,3 @@
+module Shameless
+  VERSION = "0.1.0"
+end

data/shameless.gemspec

@@ -0,0 +1,27 @@
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'shameless/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "shameless"
+  spec.version       = Shameless::VERSION
+  spec.authors       = ["Olek Janiszewski"]
+  spec.email         = ["olek@hoteltonight.com"]
+
+  spec.summary       = %q{Scalable distributed append-only data store built}
+  spec.homepage      = "https://github.com/hoteltonight/shameless"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "msgpack"
+  spec.add_dependency "sequel", "~> 4.0"
+
+  spec.add_development_dependency "bundler"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec"
+  spec.add_development_dependency "sqlite3"
+end

metadata

@@ -0,0 +1,146 @@
+--- !ruby/object:Gem::Specification
+name: shameless
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Olek Janiszewski
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-10-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: msgpack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sequel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- olek@hoteltonight.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/shameless.rb
+- lib/shameless/cell.rb
+- lib/shameless/configuration.rb
+- lib/shameless/errors.rb
+- lib/shameless/index.rb
+- lib/shameless/model.rb
+- lib/shameless/store.rb
+- lib/shameless/version.rb
+- shameless.gemspec
+homepage: https://github.com/hoteltonight/shameless
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Scalable distributed append-only data store built
+test_files: []