dynamodb_geo 0.0.2 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34aa9b9ef408a4556afbc74c173c40ccb5932fc0a47d29a3e519c0cb803fe043
-  data.tar.gz: f7dd0b6c92a94e7da2c14417ee67296af8f04f86b768bfb2c8acd32572bdd8e9
+  metadata.gz: f54f79a1153702913ca165972a8936865949156e28970e62efc06e23076b0773
+  data.tar.gz: 4171efed028b0cc80b39b35fe951c065da01ac6b7978c11e4264e9da5e0d5be1
 SHA512:
-  metadata.gz: 9d3d2f46f82c2e9ece62dc006dbda654753d01fb7d82d06d803448171a9852cef5933ce814ce20057017ed9fcb318b9f6ff37dbe79cb7f5167c82aea65f3258c
-  data.tar.gz: 8df26e8041dfbb59020496faf2ae5014fae971b9b7d0f84274ff4eeb58c0c70487623a7a44343a4443647d08cfe72acc48c8115004c287853c2896dc46e91b9a
+  metadata.gz: 04fec0da4cf2f1f62df873fd364dda594028ad566b283bd6ad56573f5d8c156237c5eb691546697001195944a483038931ee4728a8f317d9600b5114bb22af57
+  data.tar.gz: d3a7da451c4e43850c0a3b44db65f967bd28df66ea35848be57d28eb5755a47a1cb1df7a2d87a29bb0af94eec583a889ba0231f35112f4bafa0003a86a539e68

data/README.md

@@ -1,7 +1,129 @@
 # DynamoDB_Geo
+## DynamoDB
+This is an attempt at storing and querying geohash data in DynamoDB.
 
+We store objects in DynamoDB, when we query, we look for a customizable number of stored items by zooming out exactly one level.  
+We return the objects in an attempt at the closest items first.
 
-# Geohashing
+Items are searched as the following.
+
+Given the lat, long we calculate a hash - assume 9x0qz.  We go up one level to 9x0q and calculate all neighbouring hashes.
+
+    9x0p 9x0r 9x0x
+    9x0n 9x0q 9x0w     => ["9x0q", "9x0r", "9x0x", "9x0w", "9x0t", "9x0m", "9x0j", "9x0n", "9x0p"]
+    9x0j 9x0m 9x0t
+
+From this, we ask DynamoDB starting from our own cell, going north, and wrapping clockwise for all objects in these cells up to a configurable number of objects.  
+We then calculate the neighbours of our more local hash 9x0qz, and sort the results of the larger hashes using similar rules.
+
+    9x0rn 9x0rp 9x0x0
+    9x0qy 9x0qz 9x0wb  => ["9x0qz", "9x0rp", "9x0x0", "9x0wb", "9x0w8", "9x0qx", "9x0qw", "9x0qy", "9x0rn"]
+    9x0qw 9x0qx 9x0w8
+
+Using the same pattern as before (Middle -> North -> Clockwise), we sort the returned stores giving priority to the localization.
+
+### Real talk
+I know this isn't the most wonderful way to do this, I am still trying to think of something better.  Currently it uses up to 8 queries to DynamoDB, I'd like to cut that down to a single query.
+
+There is a similar library written in Java and JS, but it uses Google S2 for the Geohashing, which has properties that allow them to do the zoom-out technique with a single query, however Google S2 does not exist as a C library (or Ruby library for that matter).  The other alternative is Uber H3, however it has the same issues of not being C, or Ruby.  
+
+If we were able to use a unique but deterministic way to calcuate the range key based off of the hash key that would allow us to query every single larger cell in a single batch query.  I am currently thinking of more clever ways to do this - obviously I am open to suggestions.
+
+## Objects
+### DynamodbManager  
+
+**Attributes**  
+table_name:  
+Sets the DynamoDB Table Name
+
+hash_key:  
+Sets the name of the primary hash attribute
+
+range_key:  
+Sets the name of the sort/range attribute
+
+geohash_key:  
+Sets the name of the localized hash attribute
+
+geojson_key:  
+Sets the name of the metadata attribute
+
+hash_key_length:  
+Sets size of the outer hash length
+
+local_area_size:  
+Sets the size of the inner hash length
+
+max_item_return:  
+Sets the max number of items to return
+
+**Methods**  
+Initialization
+
+    Description: Creates a new DynamodbManager object.  Inputs are variables to your AWS account.
+                 If access_key_id and secret_access_key are provided they are used.
+                 If not provided, it falls back to ENV variables, then secret credential storage (profile name).
+                 All arguments are keyword arguments
+    Input:  region            => String
+            table_name        => String
+            access_key_id     => String
+            secret_access_key => String
+            profile_name      => 'default'
+    Output: Aws::DynamoDB::Client
+
+    #new => Object
+
+Building and describing a DynamoDB Table
+
+    Description: Shows the current configured table, or creates a table to configured as requested
+    Input:
+    Output: Aws::DynamoDB::Types::DescribeTableOutput
+
+    #table => Object
+
+Creating a new item
+
+    Description: Inserts a new item
+    Input:  Store
+    Output: Aws::DynamoDB::Types::PutItemOutput
+
+    #put_store => Object
+
+Querying stores
+
+    Description: Look for stores dependent on the input Lat, Long (as described above)
+    Input:  Store
+    Output: Array[Store]
+
+    #get_stores => Array[Store]
+
+### Store  
+
+**Methods**  
+Initialization
+
+    Description: Initialization
+    Input: Hash => {
+      latitude  # Required
+      longitude # Required
+      address
+      city
+      state
+      zip
+      area_code
+      phone
+      name
+      geohash # Calculated based on lat,long if not provided
+    }
+    Output: Store
+
+    #new => Store
+
+### DynamodbGeo  
+This only exists as a quick and easy way to create a DynamodbManager.  The only method is `.new` and it passes all arguments along to DynamodbManager and returns an instance of DynamodbManager
+
+
+## Geohashing
 Includes a Geohash implimentation written in C  
 <https://github.com/simplegeo/libgeohash>
 
@@ -43,6 +165,19 @@ Viewing all my neighbours
 
     Geohash.neighbours(hash) => Array[Geohash]
 
+Viewing one of my neighbours
+
+    Description: Takes in a Geohash and shows the neighbour in the cardinal direction
+                 NORTH = 0
+                 EAST  = 1
+                 SOUTH = 2
+                 WEST  = 3
+    Input:  Geohash   => String
+            Direction => Integer
+    Output: Geohash
+
+    Geohash.neighbour(hash) => Geohash
+
 Get width and height about a specific precision
 
     Description: Takes in a precision value, returns the height and width of the box
@@ -51,5 +186,6 @@ Get width and height about a specific precision
 
     Geohash.dimensions(precision) => Dimension
 
+### Objects
 #### Geocoord(Object) => attr_accessor: :latitude, :longitude, :north, :south, :east, :west, :dimension
 #### Dimension(Object) => attr_accessor: :length, :width

data/ext/geohash_wrapper/geohash.h

@@ -71,6 +71,8 @@ extern GeoCoord geohash_decode(char* hash);
  */ 
 extern char** geohash_neighbors(char* hash);
 
+// sorry I added this, this was not in the original header
+extern char* get_neighbor(char *hash, int direction);
 /*
  * Returns the width and height of a precision value.
  */

data/ext/geohash_wrapper/geohash_wrapper.c

@@ -6,7 +6,8 @@
 void Init_geohash_wrapper();
 VALUE wrap_geohash_encode(VALUE self, VALUE lat, VALUE lng, VALUE precision);
 VALUE wrap_geohash_decode(VALUE self, VALUE hash);
-VALUE wrap_geohash_neighbors(VALUE self, VALUE hash);
+VALUE wrap_geohash_neighbours(VALUE self, VALUE hash);
+VALUE wrap_geohash_neighbour(VALUE self, VALUE hash, VALUE direction);
 VALUE wrap_geohash_dimensions_for_precision(VALUE self, VALUE precision);
 
 
@@ -17,7 +18,8 @@ void Init_geohash_wrapper()
 
   rb_define_singleton_method(Geohash, "encode",                   wrap_geohash_encode, 3);
   rb_define_singleton_method(Geohash, "wrap_decode",              wrap_geohash_decode, 1);
-  rb_define_singleton_method(Geohash, "neighbours",               wrap_geohash_neighbors, 1);
+  rb_define_singleton_method(Geohash, "neighbours",               wrap_geohash_neighbours, 1);
+  rb_define_singleton_method(Geohash, "neighbour",                wrap_geohash_neighbour, 2);
   rb_define_singleton_method(Geohash, "dimensions_for_precision", wrap_geohash_dimensions_for_precision, 1);
 }
 
@@ -46,7 +48,7 @@ VALUE wrap_geohash_decode(VALUE self, VALUE hash) {
   return r_hash;
 }
 
-VALUE wrap_geohash_neighbors(VALUE self, VALUE hash) {
+VALUE wrap_geohash_neighbours(VALUE self, VALUE hash) {
   char** hashed_neighbours = geohash_neighbors(StringValueCStr(hash));
   VALUE neighbours = rb_ary_new2(8);
   int i;
@@ -57,6 +59,10 @@ VALUE wrap_geohash_neighbors(VALUE self, VALUE hash) {
   return neighbours;
 }
 
+VALUE wrap_geohash_neighbour(VALUE self, VALUE hash, VALUE direction) {
+  return rb_str_new_cstr(get_neighbor(StringValueCStr(hash), NUM2INT(direction)));
+}
+
 VALUE wrap_geohash_dimensions_for_precision(VALUE self, VALUE precision) {
   GeoBoxDimension dimension;
   VALUE r_hash = rb_hash_new();

data/lib/dynamodb_geo.rb

@@ -1 +1,16 @@
-require 'geohash'
+require 'dynamodb_manager'
+
+module DynamodbGeo
+  class << self
+    def new(region:, table_name:, access_key_id: nil, secret_access_key: nil, profile_name: 'default', endpoint: nil)
+      DynamodbManager.new(
+        region:            region,
+        access_key_id:     access_key_id,
+        secret_access_key: secret_access_key,
+        profile_name:      profile_name,
+        table_name:        table_name,
+        endpoint:          endpoint
+      )
+    end
+  end
+end

data/lib/dynamodb_geo/version.rb

@@ -1,5 +1,5 @@
 module DynamodbGeo
-  module Version
-    STRING = '0.0.1'
+  class Version
+    STRING = '0.1.5'
   end
 end

data/lib/dynamodb_manager.rb

@@ -0,0 +1,165 @@
+require 'aws-sdk-dynamodb'
+require 'geohash'
+require 'store'
+
+class DynamodbManager
+  attr_accessor :client, :table_name, :hash_key, :range_key, :geohash_key, :geojson, :geohash_index, :hash_key_length, :local_area_size, :max_item_return
+  def initialize(region:, table_name:, access_key_id: nil, secret_access_key: nil, profile_name: 'default', endpoint: nil)
+    if access_key_id.nil? && secret_access_key.nil?
+      access_key_id = ENV['AWS_ACCESS_KEY_ID']
+      secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+
+      credentials = Aws::SharedCredentials.new(profile_name: profile_name).credentials if access_key_id.nil? && secret_access_key.nil?
+    end
+    credentials = Aws::Credentials.new(access_key_id, secret_access_key)
+
+    @table_name      = table_name
+    @hash_key        = 'hashkey'
+    @range_key       = 'rangekey'
+    @geohash_key     = 'geohash'
+    @geojson         = 'geoJson'
+    @geohash_index   = 'geohash-index'
+    @hash_key_length = 4
+    @local_area_size = 5
+    @max_item_return = 10
+
+    if endpoint
+      @client = Aws::DynamoDB::Client.new(
+        region:      region,
+        credentials: credentials,
+        endpoint:    endpoint
+      )
+    else
+      @client = Aws::DynamoDB::Client.new(
+        region:      region,
+        credentials: credentials,
+      )
+    end
+  end
+
+  def table
+    create_table unless @client.list_tables.table_names.include?(@table_name)
+
+    @client.describe_table(table_name: @table_name)
+  end
+
+  def put_store(store)
+    hash = store.geohash[0..(@local_area_size - 1)]
+    json = {
+      latitude:  store.lat,
+      longitude: store.long,
+      address:   store.address,
+      city:      store.city,
+      state:     store.state,
+      zip:       store.zip,
+      area_code: store.area_code,
+      phone:     store.phone,
+      name:      store.name,
+    }
+    put_point(hash, json)
+  end
+
+  def get_stores(lat, long)
+    geohash    = Geohash.encode(lat, long, @local_area_size)
+    hash       = geohash[0..(@hash_key_length - 1)]
+    all_stores = []
+    neighbours = Geohash.neighbours(hash)
+    neighbours.unshift(hash)
+
+    neighbours.each do |neighbour|
+      resp = query(neighbour)
+      resp.items.each do |item|
+        latitude  = item[@geojson]['latitude']
+        longitude = item[@geojson]['longitude']
+        address   = item[@geojson]['address']
+        city      = item[@geojson]['city']
+        state     = item[@geojson]['state']
+        zip       = item[@geojson]['zip']
+        area_code = item[@geojson]['area_code']
+        phone     = item[@geojson]['phone']
+        name      = item[@geojson]['name']
+        geohash   = item[@geohash_key]
+        all_stores << Store.new(
+          latitude: latitude,
+          longitude: longitude,
+          address: address,
+          city: city,
+          state: state,
+          zip: zip,
+          area_code: area_code,
+          phone: phone,
+          name: name,
+          geohash: geohash
+        )
+      end
+      break if all_stores.length >= max_item_return
+    end
+
+    # We got all the stores in the biggest possible area, we increase the hash by one and search around now
+    neighbours = Geohash.neighbours(geohash)
+
+    closest_stores     = all_stores.select { |store| store.geohash == geohash }
+    surrounding_stores = (all_stores - closest_stores).select { |store| neighbours.include?(store.geohash) }
+    remaining_stores   = all_stores - (closest_stores + surrounding_stores)
+
+    return closest_stores + surrounding_stores + remaining_stores
+  end
+
+  private
+
+  def query(hash)
+    client.query({
+      table_name: @table_name,
+      index_name: @geohash_index,
+      expression_attribute_values: {
+        ':hash' => hash,
+      },
+      key_condition_expression: "#{@hash_key} = :hash",
+    })
+  end
+
+  def put_point(hash, json)
+    uuid = SecureRandom.uuid
+
+    @client.put_item({
+      table_name: @table_name,
+      item: {
+        @hash_key    => hash[0..(@hash_key_length - 1)],
+        @range_key   => uuid,
+        @geohash_key => hash,
+        @geojson     => json
+      }
+    })
+  end
+
+  def create_table
+    @client.create_table({
+      attribute_definitions: [
+        { attribute_name: @hash_key,    attribute_type: 'S' },
+        { attribute_name: @range_key,   attribute_type: 'S' },
+        { attribute_name: @geohash_key, attribute_type: 'S' }
+      ],
+      key_schema: [
+        { attribute_name: @hash_key,  key_type: 'HASH' },
+        { attribute_name: @range_key, key_type: 'RANGE' }
+      ],
+      local_secondary_indexes: [
+        {
+          index_name: @geohash_index,
+          key_schema: [
+            { attribute_name: @hash_key,    key_type: 'HASH' },
+            { attribute_name: @geohash_key, key_type: 'RANGE' }
+          ],
+          projection: {
+            projection_type: "ALL"
+          }
+        }
+      ],
+      provisioned_throughput: {
+        read_capacity_units: 10,
+        write_capacity_units: 5,
+      },
+      table_name: @table_name
+    })
+  end
+end

data/lib/geohash.rb

@@ -26,10 +26,10 @@ class Geocoord
 end
 
 class Dimension
-  attr_accessor :length, :width
+  attr_accessor :height, :width
 
   def initialize(geobox)
-    length = geobox['length']
-    width  = geobox['width']
+    @height = geobox['height']
+    @width  = geobox['width']
   end
 end

data/lib/store.rb

@@ -0,0 +1,15 @@
+class Store
+  attr_accessor :lat, :long, :address, :city, :state, :zip, :area_code, :phone, :name, :geohash
+  def initialize(store_data)
+    @lat       = store_data[:latitude]
+    @long      = store_data[:longitude]
+    @address   = store_data[:address]
+    @city      = store_data[:city]
+    @state     = store_data[:state]
+    @zip       = store_data[:zip]
+    @area_code = store_data[:area_code]
+    @phone     = store_data[:phone]
+    @name      = store_data[:name]
+    @geohash   = store_data[:geohash] || Geohash.encode(lat, long, 10)
+  end
+end

data/spec/dynamodb_geo_spec.rb

@@ -0,0 +1,18 @@
+require 'spec_helper'
+require 'dynamodb_geo'
+
+describe DynamodbGeo do
+  describe '.new' do
+    it 'imported properly' do
+      dynamo = DynamodbGeo.new(
+        region:       'us-east-2',
+        table_name:   'geo-test',
+        profile_name: 'test'
+      )
+      expect(dynamo).to be_a DynamodbManager
+
+      dynamo.get_stores(37.746825, -122.413637)
+      puts 'foo'
+    end
+  end
+end

data/spec/geohash_spec.rb

@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'geohash'
 
 # Quick note, I'm not actually testing if the Geohash is accurate or not.
 # I'm assuming the C Lib I imported is doing that for me properly.
@@ -31,4 +32,11 @@ describe Geohash do
       expect(dimensions).to be_a Dimension
     end
   end
+
+  describe '.neighbour' do
+    it 'calculates the neighbour in a direction' do
+      north_neighbour = Geohash.neighbour('s', 0)
+      expect(north_neighbour).to be_a String
+    end
+  end
 end

data/spec/spec_helper.rb

@@ -1,7 +1,6 @@
 require 'bundler/setup'
 Bundler.setup
 
-require 'geohash'
 require 'byebug'
 
 RSpec.configure do |config|

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dynamodb_geo
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.1.5
 platform: ruby
 authors:
 - Justin Ahn
@@ -97,9 +97,11 @@ files:
 - ext/geohash_wrapper/geohash_wrapper.c
 - lib/dynamodb_geo.rb
 - lib/dynamodb_geo/version.rb
+- lib/dynamodb_manager.rb
 - lib/geohash.rb
 - lib/geohash/geohash_wrapper.so
-- spec/dynamodb_geo.rb
+- lib/store.rb
+- spec/dynamodb_geo_spec.rb
 - spec/geohash_spec.rb
 - spec/spec_helper.rb
 homepage: 
@@ -125,8 +127,8 @@ requirements: []
 rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
-summary: dynamodb_geo-0.0.1
+summary: dynamodb_geo-0.1.5
 test_files:
-- spec/dynamodb_geo.rb
+- spec/dynamodb_geo_spec.rb
 - spec/geohash_spec.rb
 - spec/spec_helper.rb

data/spec/dynamodb_geo.rb

@@ -1 +0,0 @@
-require 'spec_helper'