lets_shard 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8e1242830aacecf1dfdb28c0253b37728055c0cb8b7c622b298e96c5ee743cc4
+  data.tar.gz: a02dfb193dd088631e62ba56b9e896a254097189630ce0a0f86f7fac0f70a17b
+SHA512:
+  metadata.gz: 8c768ca8d7cddec5be799fff150780fb945b16139344788ce70b933d7573b69fa36d77f1d1354135625ed8db72ec7f55e13d6ed29c97ebe6dc489506e8fce599
+  data.tar.gz: 1f380c18b3d65cbfd8315272e81fa11a29d96b57ba6ba76f116ebbdc0b5ce9a21d029a65d06e36d1b1f3ed4bfd877811645e13c64013033df8bbd69954078b46

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+Gemfile.lock
+.ruby-version

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at omar.abdula@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [https://contributor-covenant.org/version/1/4][version]
+
+[homepage]: https://contributor-covenant.org
+[version]: https://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in lets_shard.gemspec
+gemspec
+
+gem "rake", "~> 12.0"

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Omar Alshboul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+# LetsShard
+
+LetsShard is an easy way to shard and distribute load/use on any set of objects!
+One of the best uses is to shard [Redis](https://redis.io/)
+
+So simply run your instances and let **LetsShard** shard your usage on them!
+
+**LetsShard** distributes and shards things in a way somehow similar to [Redis Cluster](https://redis.io/topics/cluster-tutorial)
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'lets_shard'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install lets_shard
+
+## Usage
+
+```ruby
+require 'lets_shard'
+
+objects = (0..10).map { |_| AnyObject.new } # AnyObject is an arbitrary class and not included in the gem.
+
+my_shards = LetsShard.new(objects)
+
+# To see all objects including the intervals they cover
+my_shards.shards
+
+# To get the object that covers "blah"
+my_object = my_shards.get_object("blah")
+my_object.foo
+
+# To get the shard (including the intervals and the object it self) that covers "blah"
+my_shard = my_shards.get_shard("blah")
+
+# To see slots/intervals that "my_shard" covers
+my_shard.start_slot
+my_shard.end_slot
+
+# To use the object of this interval
+my_shard.object.foo
+```
+
+### An example with redis
+
+```ruby
+require 'lets_shard'
+require 'redis'
+
+redis1 = Redis.new(host: "10.0.0.1", port: 6379)
+redis2 = Redis.new(host: "10.0.0.2", port: 6379)
+
+redis_shards = LetsShard.new([redis1, redis2])
+
+redis_shards.get_object('key').set('key', 'value')
+redis_shards.get_object('key').get('key')
+
+# redis_shards.get_object('key')... any redis command!
+```
+
+### LetsShard let you distribute objects usage better!
+
+If you are use 3 redis instances in 3 servers, but one of these servers has more
+powerful specifications than the others and you want to utilize its power.
+
+**LetsShard** can solve it for you, easily!
+
+```ruby
+require 'lets_shard'
+require 'redis'
+
+redis1 = Redis.new(host: "10.0.0.1", port: 6379) # The most powerful server
+redis2 = Redis.new(host: "10.0.0.2", port: 6379)
+redis3 = Redis.new(host: "10.0.0.3", port: 6379)
+
+redises = [redis1, redis2, redis3]
+weights = [2, 1, 1] # By default, the weight of each object is 1
+
+redis_shards = LetsShard.new(redises, weights)
+
+# So now LetsShard will give redis1 more slots to utilize it more!
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/oshboul/lets_shard. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/lets_shard/blob/master/CODE_OF_CONDUCT.md).
+
+
+## Code of Conduct
+
+Everyone interacting in the LetsShard project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/lets_shard/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "lets_shard"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lets_shard.gemspec

@@ -0,0 +1,23 @@
+require_relative 'lib/lets_shard/version'
+
+Gem::Specification.new do |spec|
+  spec.name = 'lets_shard'
+  spec.version = LetsShard::VERSION
+  spec.authors = ['oshboul']
+  spec.email = ['omar.abdula@gmail.com']
+
+  spec.summary = spec.description = 'An easy way to shard anything!'
+  spec.homepage = 'https://github.com/oshboul/lets_shard'
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      f.match(%r{^(test|spec|features)/})
+    end
+  end
+
+  spec.bindir = 'exe'
+  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+end

data/lib/lets_shard/digest.rb

@@ -0,0 +1,51 @@
+class LetsShard
+  module Digest
+    extend self
+
+    X_MODEM_CRC16_LOOKUP = [
+      0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7,
+      0x8108,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef,
+      0x1231,0x0210,0x3273,0x2252,0x52b5,0x4294,0x72f7,0x62d6,
+      0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de,
+      0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485,
+      0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d,
+      0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6,0x5695,0x46b4,
+      0xb75b,0xa77a,0x9719,0x8738,0xf7df,0xe7fe,0xd79d,0xc7bc,
+      0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802,0x3823,
+      0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b,
+      0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50,0x3a33,0x2a12,
+      0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a,
+      0x6ca6,0x7c87,0x4ce4,0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41,
+      0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49,
+      0x7e97,0x6eb6,0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70,
+      0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,
+      0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f,
+      0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046,0x6067,
+      0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e,
+      0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214,0x6277,0x7256,
+      0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d,
+      0x34e2,0x24c3,0x14a0,0x0481,0x7466,0x6447,0x5424,0x4405,
+      0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c,
+      0x26d3,0x36f2,0x0691,0x16b0,0x6657,0x7676,0x4615,0x5634,
+      0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab,
+      0x5844,0x4865,0x7806,0x6827,0x18c0,0x08e1,0x3882,0x28a3,
+      0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a,
+      0x4a75,0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92,
+      0xfd2e,0xed0f,0xdd6c,0xcd4d,0xbdaa,0xad8b,0x9de8,0x8dc9,
+      0x7c26,0x6c07,0x5c64,0x4c45,0x3ca2,0x2c83,0x1ce0,0x0cc1,
+      0xef1f,0xff3e,0xcf5d,0xdf7c,0xaf9b,0xbfba,0x8fd9,0x9ff8,
+      0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0
+    ]
+    
+    def crc16(key)
+      result = 0
+
+      key.each_byte do |b|
+        result = ((result << 8) & 0xffff) ^
+                X_MODEM_CRC16_LOOKUP[((result >> 8) ^ b) & 0xff]
+      end
+
+      result
+    end
+  end
+end

data/lib/lets_shard/shard.rb

@@ -0,0 +1,11 @@
+class LetsShard
+  class Shard
+    attr_reader :object, :start_slot, :end_slot
+    
+    def initialize(object, start_slot, end_slot)
+      @object = object
+      @start_slot = start_slot
+      @end_slot = end_slot
+    end
+  end
+end

data/lib/lets_shard/version.rb

@@ -0,0 +1,3 @@
+class LetsShard
+  VERSION = "0.1.0"
+end

data/lib/lets_shard.rb

@@ -0,0 +1,78 @@
+require 'lets_shard/version'
+require 'lets_shard/digest'
+require 'lets_shard/shard'
+
+class LetsShard
+  attr_reader :shards
+
+  SLOTS = 16384
+
+  def initialize(objects, weights = [])
+    validate_parameters!(objects, weights)
+
+    @objects = objects
+    @weights = weights.any? ? weights : Array.new(@objects.size, 1)
+
+    set_unit_weight!
+    generate_shards!
+  end
+
+  def get_shard(key)
+    hkey = get_hkey(key)
+
+    @shards.find do |shard|
+      (shard.start_slot..shard.end_slot).include?(hkey)
+    end
+  end
+
+  def get_object(key)
+    get_shard(key).object
+  end
+
+  private
+
+  def generate_shards!
+    @shards = []
+    start_slot = 0
+
+    @objects.each_with_index do |object, index|
+      end_slot = [start_slot + (@weights[index] * @unit_weight), SLOTS - 1].min
+
+      @shards << Shard.new(object, start_slot, end_slot)
+
+      start_slot = end_slot + 1
+    end
+  end
+
+  def set_unit_weight!
+    @unit_weight = (SLOTS.to_f / @weights.inject(0, :+)).ceil
+  end
+
+  def get_hkey(key)
+    Digest.crc16(key.to_s) % SLOTS
+  end
+
+  def validate_parameters!(objects, weights)
+    unless objects.is_a?(Array)
+      raise LetsShardError, 'Objects should be an array!'
+    end
+
+    unless weights.is_a?(Array)
+      raise LetsShardError, 'Weights should be an array!'
+    end
+
+    unless objects.any?
+      raise LetsShardError, 'Objects array should not be empty!'
+    end
+
+    if weights.any? && (objects.size != weights.size)
+      raise LetsShardError, 'Weights should be as size as objects!'
+    end
+
+    if weights.any? && !weights.all? { |w| w.is_a?(Integer) }
+      raise LetsShardError, 'All weights should be integers!'
+    end
+  end
+
+  class LetsShardError < StandardError; end
+end

metadata

@@ -0,0 +1,55 @@
+--- !ruby/object:Gem::Specification
+name: lets_shard
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- oshboul
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2020-04-01 00:00:00.000000000 Z
+dependencies: []
+description: An easy way to shard anything!
+email:
+- omar.abdula@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lets_shard.gemspec
+- lib/lets_shard.rb
+- lib/lets_shard/digest.rb
+- lib/lets_shard/shard.rb
+- lib/lets_shard/version.rb
+homepage: https://github.com/oshboul/lets_shard
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.1.2
+signing_key: 
+specification_version: 4
+summary: An easy way to shard anything!
+test_files: []