soulheart 0.0.5 → 0.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: cd989a4107aacd141e1ecbcdcab17c4565237768
-  data.tar.gz: eadaae2aad79d613be888a6cd39fe130071c8b00
+  metadata.gz: 0ee2b8a0da3cca62bb1236f16a0537e3b6d153e9
+  data.tar.gz: bc09f384eb18f2162d3eb7b852e137ecfe3a48d0
 SHA512:
-  metadata.gz: 8f476069f2c8ff7491142570abc525ca25d0ad9870ac22398b6ee1fa6c55653b828ae6f3cc2bc427ea32103fbd8cd4a3649483aefebdf31cdf3a0e4fad4714b1
-  data.tar.gz: a4a3682db05d5227469b5291399be897cb006ebed4654aae8a305f2c84eb388afce580974fbe4d6578cac17ce0521300921e1f6d76001df0b8c84e645a66fb12
+  metadata.gz: b1cc93669b128ddd68f594124ec0e9bee2b7ee7fa441a8c135c099f00513e288f37ccc0cd02efb67918b101b84220c8c9cc098d10f435678cabb6a4fbb17da42
+  data.tar.gz: 2de3adf63591024964833931ddfd2bff99573711509e633618b7e6a842ca4b2907ade4e91e836ee9e41a8ca5b902cb29ba20db90a2824b2f9b2e969fe4acfa34

data/README.md

@@ -1,5 +1,7 @@
 # <img src="https://raw.githubusercontent.com/sethherr/soulheart/master/logo.png" alt="Soulheart" width="200"> Soulheart [![Build Status](https://travis-ci.org/sethherr/soulheart.svg)](https://travis-ci.org/sethherr/soulheart) [![Code Climate](https://codeclimate.com/github/sethherr/soulheart/badges/gpa.svg)](https://codeclimate.com/github/sethherr/soulheart) [![Test Coverage](https://codeclimate.com/github/sethherr/soulheart/badges/coverage.svg)](https://codeclimate.com/github/sethherr/soulheart/coverage)
 
+To get started, check out examples and documentation at [sethherr.github.io/soulheart/](https://sethherr.github.io/soulheart/).
+
 **Soulheart is a ready-to-use remote data source for autocomplete**. It supports:
 
 - pagination
@@ -11,199 +13,22 @@
 
 ... and is [instantly deployable to heroku](https://heroku.com/deploy) (for free).
 
-To get started, check out examples and documentation at [sethherr.github.io/soulheart/](https://sethherr.github.io/soulheart/).
-
-----
-
-Soulheart is in Beta. It's probably appropriate to use in production... but maybe wait? There are a few more changes coming, and some documentation improvements to be made.
-
-### Adding data
-
-You can add data from json, CSV and TSV files. 
-
-Adding data is very simple - all you need is a `text` value.
-
-Soulheart uses [line delineated JSON streams](https://en.wikipedia.org/wiki/JSON_Streaming#Line_delimited_JSON), so it doesn't have to load the whole file into memory. Which just means - put each object onto a seperate line.
-
-For the simplest case, with just text values in JSON:
-
-    { "text": "Jamis" }
-    { "text": "Specialized" }
-    { "text": "Trek" }
-
-It accepts local files:
-
-    soulheart load my_json_file.json
-
-or remote files:
-  
-    soulheart load https://gist.githubusercontent.com/sethherr/96dbc011e508330ceec4/raw/95122b1fc9de85f241cd048f32b94568f54134e0/manufacturers.tsv
-
 
-In addition to term, there are a few optional values - 
+This project is in Beta. It's probably appropriate to use in production... but maybe wait? There are a few more changes coming, and some documentation improvements to be made.
 
-| Key          | Default     | What it does |
-| ------------ | ----------- | ------------ |
-| `priority`   | `100`       | Higher numbers come first |
-| `category`   | `'default'` | Sets the category |
-| `data`       | `{}`        | Returned object from search - the text and category will be added to this if you don't specify them. |
+=======
 
-Here is an example of what a possible hash you could pass is
+I'm testing with: `ruby 2.1` and `redis 3.0`. 
 
-    { "text": "Jamis", "category": "Bike Manufacturer" }
-    { "text": "Specialized" }
-    { "text": "Trek" }
+Run `bundle exec guard` to run the specs when they change as you work.
 
-*If you set `text` in `data`, it will respond with that rather than the term it searches by. I haven't figured out a use case for this yet, but I'm sure one exists.*
+This repo includes a `config.ru` and a `Gemfile.lock` so it and any forks of it can be deployed to Heroku. They shouldn't be in the Gem. if you build it separately.
 
 ======
 
-I'm testing with: `ruby >= 2.1` and `redis >= 3`. 
-
-Run `bundle exec guard` to run the specs while you work, it will just test the files you change.
-
-This repo includes a `config.ru` and a `Gemfile.lock` so it (and any forks of it) can be deployed to heroku. They shouldn't be in the Gem itself.
-
-
-======
-
-This is an updated fork of [Seatgeek/Soulmate](https://github.com/seatgeek/soulmate) to address a few issues - namely [CORS support](../../issues/2), [minimum entry length](../../issues/3) and [playing better with Selectize & Select2](../../issues/4) - also the future.
-
-Since [Seatgeek no longer uses Soulmate](https://news.ycombinator.com/item?id=9317891), and this isn't backward compatible it's a new project and gem.
-
-:x::o::x::o::x::o: *Soulmate's README follows ([issue for making new documentation](../../issues/1))*
-
-Soulmate is a tool to help solve the common problem of developing a fast autocomplete feature. It uses Redis's sorted sets to build an index of partially completed words and the corresponding top matching items, and provides a simple sinatra app to query them. Soulmate finishes your sentences.
-
-Soulmate was designed to be simple and fast, and offers the following:
-
- * Provide suggestions for multiple types of items in a single query (at SeatGeek we're autocompleting for performers, events, and venues)
- * Results are ordered by a user-specified score
- * Arbitrary metadata for each item (at SeatGeek we're storing both a url and a subtitle)
-
-An item is a simple JSON object that looks like:
-
-    {
-      "id": 3,
-      "term": "Citi Field",
-      "score": 81,
-      "data": {
-        "url": "/citi-field-tickets/",
-        "subtitle": "Flushing, NY"
-      }
-    }
-
-Where `id` is a unique identifier (within the specific type), `term` is the phrase you wish to provide completions for, `score` is a user-specified ranking metric (redis will order things lexicographically for items with the same score), and `data` is an optional container for metadata you'd like to return when this item is matched (at SeatGeek we're including a url for the item as well as a subtitle for when we present it in an autocomplete dropdown).
-
-Getting Started
----------------
-
-As always, kick things off with a `gem install`:
-
-    gem install soulmate
-
-### Loading Items
-
-You can load data into Soulmate by piping items in the JSON lines format into `soulmate load TYPE`.
-
-Here's a sample `venues.json` (one JSON item per line):
-
-    {"id":1,"term":"Dodger Stadium","score":85,"data":{"url":"\/dodger-stadium-tickets\/","subtitle":"Los Angeles, CA"}}
-    {"id":28,"term":"Angel Stadium","score":85,"data":{"url":"\/angel-stadium-tickets\/","subtitle":"Anaheim, CA"}}
-    {"id":30,"term":"Chase Field ","score":85,"data":{"url":"\/chase-field-tickets\/","subtitle":"Phoenix, AZ"}}
-    {"id":29,"term":"Sun Life Stadium","score":84,"data":{"url":"\/sun-life-stadium-tickets\/","subtitle":"Miami, FL"}}
-    {"id":2,"term":"Turner Field","score":83,"data":{"url":"\/turner-field-tickets\/","subtitle":"Atlanta, GA"}}
-
-And here's the load command (Soulmate assumes redis is running locally on the default port, or you can specify a redis connection string with the `--redis` argument):
-
-    $ soulmate load venue --redis=redis://localhost:6379/0 < venues.json
-
-You can also provide an array of strings under the `aliases` key that will also be added to the index for this item.
-
-### Querying for Data
-
-Once it's loaded, we can query this data by starting `soulmate-web`:
-
-    $ soulmate-web --foreground --no-launch --redis=redis://localhost:6379/0
-
-And viewing the service in your browser: http://localhost:5678/search?types[]=venue&term=stad. You should see something like:
-
-    {
-      "term": "stad",
-      "results": {
-        "venue": [
-          {
-            "id": 28,
-            "term": "Angel Stadium",
-            "score": 85,
-            "data": {
-              "url": "/angel-stadium-tickets/",
-              "subtitle": "Anaheim, CA"
-            }
-          },
-          {
-            "id": 1,
-            "term": "Dodger Stadium",
-            "score": 85,
-            "data": {
-              "url": "/dodger-stadium-tickets/",
-              "subtitle": "Los Angeles, CA"
-            }
-          },
-          {
-            "id": 29,
-            "term": "Sun Life Stadium",
-            "score": 84,
-            "data": {
-              "url": "/sun-life-stadium-tickets/",
-              "subtitle": "Miami, FL"
-            }
-          }
-        ]
-      }
-    }
-
-The `/search` method supports multiple `types` as well as an optional `limit`. For example: `http://localhost:5678/search?types[]=event&types[]=venue&types[]=performer&limit=3&term=yank`. You can also add the `callback` parameter to enable JSONP output.
-
-### Mounting soulmate into a rails app
-
-If you are integrating Soulmate into a rails app, an alternative to launching a separate 'soulmate-web' server is to mount the sinatra app inside of rails.
-
-Add this to routes.rb:
-
-    mount Soulmate::Server, :at => "/sm"
-
-Add this to gemfile:
-
-    gem 'rack-contrib'
-    gem 'soulmate', :require => 'soulmate/server'
-
-Then you can query soulmate at the /sm url, for example: http://localhost:3000/sm/search?types[]=venues&limit=6&term=kitten
-
-You can also config your redis instance:
-
-    # config/initializers/soulmate.rb
-    
-    Soulmate.redis = 'redis://127.0.0.1:6379/0'
-    # or you can asign an existing instance of Redis, Redis::Namespace, etc.
-    # Soulmate.redis = $redis
-
-### Rendering an autocompleter
-
-Soulmate doesn't include any client-side code necessary to render an autocompleter, but Mitch Crowe put together a pretty cool looking jquery plugin designed for exactly that: <a href="https://github.com/mcrowe/soulmate.js">soulmate.js</a>.
-
-Contributing to soulmate
-------------------------
- 
-* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
-* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
-* Fork the project
-* Start a feature/bugfix branch
-* Commit and push until you are happy with your contribution
-* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+[There is issue for improving documentation](../../issues/1) because the documentation needs to be improved. Also, for serious, check out [sethherr.github.io/soulheart/](https://sethherr.github.io/soulheart/).
 
-Copyright
----------
+=====
 
-Copyright (c) 2011 Eric Waller. See LICENSE.txt for further details.
 
+This is a fork of [Soulmate](https://github.com/seatgeek/soulmate), it's goal isn't backward compatibility.

data/bin/soulheart

@@ -38,10 +38,8 @@ parser = OptionParser.new do |opts|
 
   opts.separator ''
   opts.separator 'Commands:'
-  opts.separator '  load   TYPE FILE  Replaces collection specified by TYPE with items read from FILE in the JSON lines format.'
-  opts.separator '  add    TYPE       Adds items to collection specified by TYPE read from stdin in the JSON lines format.'
-  opts.separator "  remove TYPE       Removes items from collection specified by TYPE read from stdin in the JSON lines format. Items only require an 'id', all other fields are ignored."
-  opts.separator '  query  TYPE QUERY Queries for items from collection specified by TYPE.'
+  opts.separator '  load   FILE  Loads data from a FILE - can be a local file or a url. Accepted formats are .json, .tsv and .csv'
+  opts.separator "  reset  FILE  Removes all existing data, then runs load on the FILE."
 end
 
 
@@ -91,59 +89,19 @@ def load(file)
   puts "Loaded a total of #{count} items in #{Time.now.to_i - start_time} second(s)"
 end
 
-def add(type)
-  puts "Adding items of type #{type}..."
-  loader = Soulheart::Loader.new(type)
-  items = $stdin.read.split("\n").map { |l| MultiJson.decode(l) }
-  items.each do |item|
-    loader.add(item)
-  end
-  puts "Loaded a total of #{items.size} items"
-end
-
-def remove(type)
-  puts "Removing items of type #{type}..."
-  loader = Soulheart::Loader.new(type)
-  items = $stdin.read.split("\n").map { |l| MultiJson.decode(l) }
-  items.each do |item|
-    loader.remove(item)
-  end
-  puts "Removed a total of #{items.size} items"
-end
-
-def query(type, query)
-  puts "> Querying '#{type}' for '#{query}'"
-  matcher = Soulheart::Matcher.new(type)
-  results = matcher.matches_for_term(query, limit: 0)
-  results.each do |item|
-    puts MultiJson.encode(item)
-  end
-  puts "> Found #{results.size} matches"
-end
-
-def gen_redis_proto(*cmd)
-  proto = '*' + cmd.length.to_s + "\r\n"
-  cmd.each{|arg|
-    proto << '$' + arg.bytesize.to_s + "\r\n"
-    proto << arg + "\r\n"
-  }
-  proto
+def clear
+  Soulheart::Loader.new.clear
 end
 
 parser.parse!
 BATCH_SIZE ||= 1000
 
 case ARGV[0]
-when 'generate'
-  generate ARGV[1], ARGV[2]
 when 'load'
   load ARGV[1]
-when 'add'
-  add ARGV[1]
-when 'remove'
-  remove ARGV[1]
-when 'query'
-  query ARGV[1], ARGV[2]
+when 'reset'
+  clear
+  load ARGV[1]
 else
   puts parser.help
 end

data/lib/soulheart/loader.rb

@@ -15,6 +15,7 @@ module Soulheart
     end
 
     def delete_categories
+      redis.expire category_combos_id, 0
       redis.expire categories_id, 0
     end
 
@@ -23,27 +24,39 @@ module Soulheart
       redis.sadd categories_id, categories
     end
 
-    def delete_data
+    def delete_data(id="#{base_id}:")
       # delete the sorted sets for this type
       phrases = redis.smembers(base_id)
       redis.pipelined do
         phrases.each do |p|
-          redis.del("#{base_id}:#{p}")
+          redis.del("#{id}#{p}")
         end
-        redis.del(base_id)
+        redis.del(id)
       end
 
-      # Redis can continue serving cached requests for this type while the reload is
-      # occuring. Some requests may be cached incorrectly as empty set (for requests
+      # Redis can continue serving cached requests while the reload is
+      # occurring. Some requests may be cached incorrectly as empty set (for requests
       # which come in after the above delete, but before the loading completes). But
       # everything will work itself out as soon as the cache expires again.
+    end
 
-      # delete the data stored for this type
+    def remove_results_hash
+      # delete the data store
+      # We don't do this every time we clear because because it breaks the caching feature. 
+      # The option to clear this is only called in testing right now. 
+      # There should be an option to clear it other times though.
+      redis.expire results_hashes_id, 0
       redis.del(results_hashes_id)
     end
 
-    def load(items)
+    def clear(remove_results: false)
+      category_combos.each {|cat| delete_data(category_id(cat)) }
+      delete_categories
       delete_data
+      remove_results_hash if remove_results
+    end
+
+    def load(items)
       # Replace with item return so we know we have category_id
       items.each { |item| item.replace(add_item(item)) }
       set_category_combos_array.each do |category_combo|
@@ -87,22 +100,5 @@ module Soulheart
       end
       item
     end
-
-    # remove only cares about an item's id, but for consistency takes an object
-    def remove(item)
-      prev_item = Soulheart.redis.hget(base_id, item['term'])
-      if prev_item
-        prev_item = MultiJson.decode(prev_item)
-        # undo the operations done in add
-        Soulheart.redis.pipelined do
-          Soulheart.redis.hdel(base_id, prev_item['term'])
-          phrase = ([prev_item['term']] + (prev_item['aliases'] || [])).join(' ')
-          prefixes_for_phrase(phrase).each do |p|
-            Soulheart.redis.srem(base_id, p)
-            Soulheart.redis.zrem("#{base_id}:#{p}", prev_item['term'])
-          end
-        end
-      end
-    end
   end
 end

data/lib/soulheart/version.rb

@@ -1,3 +1,3 @@
 module Soulheart
-  VERSION = '0.0.5'
+  VERSION = '0.0.6'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: soulheart
 version: !ruby/object:Gem::Version
-  version: 0.0.5
+  version: 0.0.6
 platform: ruby
 authors:
 - Seth Herr