busca 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b4bb8a3f394f361469f5737cff36bbf104fadfdc
+  data.tar.gz: 9427a9b9fbe01dae878c272de043b810e560f63d
+SHA512:
+  metadata.gz: 19b15e5d14ba75446d2b5edf0880e16857b5a7b98a1df26e8839061e4b4bd7bc0f0b8835a710da534af88978832c1775c72ac8ad00a1e7b231a5ac56c4282d2d
+  data.tar.gz: b05df704a480f6e9a6e7be0549dea7bf264f67b5a6708620d6c01bb9a531a24dffa8af69e029bc6112b0362c49785b739e2549104f57c7dd6e55ab9f997b0feb

data/.gitignore

@@ -0,0 +1,3 @@
+busca.sublime-project
+busca.sublime-workspace
+dump.rdb

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Busca Changelog
+
+## [0.0.2] - 2015-07-26
+
+### Changed
+- Updated dependencies (Separa and Filtra).
+
+### Added
+- Changelog

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2012 Julián Porta
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,105 @@
+Busca
+====
+
+Busca is a simple redis search. Nothing more, nothing less.
+
+Description
+-----------
+
+Busca is a simple redis search that uses bitmaps to index words and performs searches. It is NOT recommended for your *big data*, unless you have *big <sup>big</sup> memory*. Also, there are no tests (yet) of how it performs with a that *big data* (if you are feeling adventurous, let me know).
+
+## Installation
+
+As usual, you can install it using rubygems.
+
+```
+$ gem install busca
+```
+
+## What kind of sorcery is this?
+
+Well, that's a fair question. Busca is a ruby gem that indexes text and perform searches. It does that using redis as a backend, which is nice, because redis is nice.
+
+To be honest, I have no idea how a fulltext search works, but I'm pretty sure it's complex. Busca is simple and works (at least for me).
+
+It started as a proof of concept, and *it's still* a proof of concept, but running in production in a couple of tiny sites.
+
+## Before usage
+
+Busca works (more or less) like this:
+
+* You create a new instance of Busca, something like `busca = Busca.new`
+* You pass some *identifier* (more on this later) and a string to the `index` function. Something like `busca.index(101, "Hey, please index this fancy text for me, will you?")`
+* Then, you search, like `result = busca.search("fancy")`
+
+That's it. `result` now contains an array of the *identifiers* that contain the word "fancy".
+
+This is pretty much it. Internally, however, a few things happen.
+
+`Busca` uses two gems:
+
+* [Separa](https://github.com/Porta/separa) splits the input (a string, an object, you name it) into an array of terms.
+* [Filtra](https://github.com/Porta/filtra) filters that array (removing duplicates, stemming, changing case, removing stopwords) to only index the terms you really want to index and make a effective use of space.
+
+While those gems are included in `Busca` and used by default, you're not tied to them. You can implement your own separator and filter based on your particular scenario. Feel free to check the docs of both in order to get an idea of what they do and how.
+
+Overall, this is the flow:
+
+#### Index:
+
+* You create a new instance of `Busca`. This is where you pass any configuration.
+
+* Then, you `index` something. A unique identifier is required first (in most cases, the id of whatever you're indexing, so you can retrieve it from it's store later). Then, whatever you want to index. It can be a string or a ruby object.
+
+* `Busca` receives whatever you passed to index and passes that to the separator.
+
+* Then, the resulting array is passed to the filter.
+
+* The resulting array is then indexed (saved into redis).
+
+### Search
+
+* With your instance of `Busca`, you call the `search` method with whatever you want to search for (string or object)
+
+* Busca passes that through the same separator and filter used in the indexing process.
+
+* The result is an array with the identifiers you passed on the indexing process.
+
+* Done!
+
+## Usage
+
+### Index/search a string
+
+```ruby
+    document = <<-eos
+Ay, marry, is't:
+But to my mind, though I am native here
+And to the manner born, it is a custom
+More honour'd in the breach than the observance.
+This heavy-headed revel east and west
+Makes us traduced and tax'd of other nations:
+They clepe us drunkards, and with swinish phrase
+Soil our addition; and indeed it takes
+From our achievements, though perform'd at height,
+The pith and marrow of our attribute.
+eos
+
+busca = Busca.new
+indexed_id = busca.index(123, document) #indexed_id return id assigned in the index.
+                                        #not especially relevant, but in case you need it for something
+#... later on
+result = busca.search("breach")
+result == [123] #the identifier of the document.
+```
+
+
+
+
+
+
+
+
+
+
+

data/busca.gemspec

@@ -0,0 +1,18 @@
+# encoding: utf-8
+
+Gem::Specification.new do |s|
+  s.name              = "busca"
+  s.version           = "0.0.2"
+  s.summary           = "Busca is a simple redis search"
+  s.description       = "Busca is a simple redis search that uses bitmaps to index words"
+  s.authors           = ["Julián Porta"]
+  s.email             = ["julian@porta.sh"]
+  s.homepage          = "https://github.com/Porta/busca"
+  s.files             = `git ls-files`.split("\n")
+  s.license           = "MIT"
+  s.add_development_dependency "cutest", '~>1.2'
+  s.add_runtime_dependency "redic", '~> 1.4'
+  s.add_runtime_dependency "msgpack", '~> 0.5'
+  s.add_runtime_dependency "filtra", '~> 0.0.2'
+  s.add_runtime_dependency "separa", '~> 0.0.3'
+end

data/lib/busca.rb

@@ -0,0 +1,78 @@
+# Encoding: utf-8
+require 'msgpack'
+require 'redic'
+require 'separa'
+require 'filtra'
+
+class Busca
+  NAMESPACE   = 'Busca' #TODO: Confirm gem name
+  LUA_CACHE   = Hash.new { |h, k| h[k] = Hash.new }
+  LUA_INDEX   = File.expand_path("./lib/lua/index.lua")
+  LUA_SEARCH  = File.expand_path("./lib/lua/search.lua")
+  LUA_REMOVE  = File.expand_path("./lib/lua/remove.lua")
+
+  attr_reader :namespace, :redis, :separa, :filtra
+
+  def initialize(opts = {})
+    @namespace = opts[:namespace] || NAMESPACE
+    @redis = opts[:redis] || Redic.new
+    @separa = opts[:separa] || Separa.new
+    @filtra = opts[:filtra] || Filtra.new
+  end
+
+  def split_and_filter(words)
+    terms = @separa.call(words)
+    filtered = @filtra.call(terms)
+    return filtered
+  end
+
+  def index(document_id, string)
+    words = split_and_filter(string)
+
+    index_id = script( LUA_INDEX, 0,
+       @namespace.to_msgpack,
+       document_id.to_msgpack,
+       words.to_msgpack
+      )
+    return index_id
+  end
+
+
+  def search(string)
+    words = split_and_filter(string)
+    document_ids = script( LUA_SEARCH, 0,
+       @namespace.to_msgpack,
+       words.to_msgpack
+      )
+    bitstring = document_ids.unpack('B*').join("")
+    ids = []
+    bitstring.each_char.each_with_index{|c, i| c == "1" ? ids.push(i) : nil}
+    ids
+    # document_ids
+  end
+
+
+  def remove(id)
+    result = script( LUA_REMOVE, 0,
+       @namespace.to_msgpack,
+       id.to_msgpack
+      )
+    return result
+  end
+
+  private
+
+  def script(file, *args)
+    cache = LUA_CACHE[redis.url]
+
+    if cache.key?(file)
+      sha = cache[file]
+    else
+      src = File.read(file)
+      sha = redis.call("SCRIPT", "LOAD", src)
+      cache[file] = sha
+    end
+    redis.call("EVALSHA", sha, *args)
+  end
+
+end

data/lib/lua/index.lua

@@ -0,0 +1,36 @@
+local namespace = cmsgpack.unpack(ARGV[1])
+local document_id  = cmsgpack.unpack(ARGV[2])
+local words     = cmsgpack.unpack(ARGV[3])
+
+local function check_for_vacants()
+  local id = redis.call('RPOP', namespace .. ':vacants')
+  return id
+end
+
+local function save(document_id, words)
+  -- first try to get one available id from the vacants list
+  local id = check_for_vacants()
+  
+  if id == false then
+    id = redis.call('INCR', namespace .. ':document:ids')
+  end
+
+  redis.call('SET', namespace .. ':document:' .. id, document_id)
+
+  return id
+end
+
+
+local function index(words, id)
+  for i, word in pairs(words) do
+    redis.call('SETBIT', namespace .. ':' .. word, id, 1)
+    redis.call('RPUSH', namespace .. ':document:' .. id .. ':words', word)
+  end
+end
+
+
+local id = save(document_id, words)
+
+index(words, id)
+
+return id

data/lib/lua/remove.lua

@@ -0,0 +1,34 @@
+local namespace = cmsgpack.unpack(ARGV[1])
+local id        = cmsgpack.unpack(ARGV[2])
+
+
+local function get_words(id)
+  local words = redis.call('LRANGE', namespace .. ':document:' .. id .. ':words', 0, -1)
+  return words
+end
+
+local function set_new_vacants(id)
+  local id = redis.call('RPUSH', namespace .. ':vacants', id)
+  return id
+end
+
+local function delete(id)
+  -- add document id to vacants list
+  set_new_vacants(id)  
+  redis.call('DEL', namespace .. ':document:' .. id)
+
+end
+
+local function remove_from_index(words, id)
+  for i, word in pairs(words) do
+    redis.call('SETBIT', namespace .. ':' .. word, id, 0)
+  end
+end
+
+
+local words = get_words(id)
+delete(id)
+
+remove_from_index(words, id)
+
+return true

data/lib/lua/search.lua

@@ -0,0 +1,51 @@
+local namespace = cmsgpack.unpack(ARGV[1])
+local words     = cmsgpack.unpack(ARGV[2])
+
+
+local function join(list, char)
+  local temp_str = ""
+  for k, v in pairs( list ) do
+     temp_str = temp_str .. char .. v
+  end
+  temp_str = string.sub(temp_str, 2) --remove first '_'
+  return temp_str
+end
+
+local function namespace_keys(list)
+  local keys = {}
+  for k, v in pairs( list ) do
+    table.insert(keys, k, namespace .. ':' .. v)
+  end
+  return keys
+end
+
+local function temp_dest_key(words)
+  local temp_key = join(words, '_')  
+  local cached_key = namespace .. ':query_cache:' .. temp_key
+  
+  local cached = redis.call('GET', cached_key)
+  
+  return cached_key, cached
+end
+
+local function cache_results(temp_dest_key, results)
+  -- no need to cache anything since the bitop operation
+  -- already writes the result to cache
+end
+
+
+local function search(words)
+  local cached_key, cached = temp_dest_key(words)
+  if cached == false then
+    redis.call('BITOP', 'AND', cached_key, unpack( namespace_keys(words) ) )
+    cached = redis.call('GET', cached_key)
+    if cached == false then
+      cached = ""
+    end
+  end  
+  return cached
+end
+
+
+local results = search(words)
+return results

data/makefile

@@ -0,0 +1,2 @@
+test:
+	cutest -r ./tests/helper.rb ./tests/*_test.rb

data/tests/busca_test.rb

@@ -0,0 +1,45 @@
+
+scope do
+    setup do
+        document = "this is some document that I've indexed"
+        busca = Busca.new()
+        busca.redis.call('flushdb')
+        [busca, document]
+    end
+
+    test "index a document and get it's id" do |busca, document|
+    document_id = busca.index(5, document)
+    assert_equal document_id, 1
+    end
+
+    test "index two documents" do |busca, document|
+        first = busca.index(3, document)
+        second = busca.index(7, "uno dos tres cuatro cinco seis")
+        assert_equal first, 1
+        assert_equal second, 2
+    end
+
+    test "trying a more complex separator" do |a, b|
+        separa = Separa.new(regexp: /,/)
+        busca = Busca.new(separa: separa)
+        document = "hey,there,this,is,sparta"
+        document_id = busca.index(1, document)
+        assert_equal 1, document_id
+        another_document = "i,cannot,believe,in,sparta"
+        busca.index(2, another_document)
+        sparta = busca.search("sparta")
+        assert_equal [1,2], sparta
+    end
+
+    test "trying a more complex document" do |a, b|
+        separa = Separa.new(Separa::Obj)
+        busca = Busca.new(separa: separa)
+        document = {uno: 'uno', dos: 'dos', tres: {cuatro: 4, cinco: 'SINCO'}}
+        document_id = busca.index(1, document)
+        assert_equal 1, document_id
+        cinco = busca.search(dos: 'dos')
+        assert_equal [1], cinco
+
+    end
+
+end

data/tests/helper.rb

@@ -0,0 +1,7 @@
+require "cutest"
+require_relative "../lib/busca"
+
+prepare do
+    busca = Busca.new
+    busca.redis.call("FLUSHDB")
+end

data/tests/remove_test.rb

@@ -0,0 +1,34 @@
+
+scope do
+  setup do
+      document = "this is some document that I've indexed"
+      busca = Busca.new()
+      busca.redis.call('flushdb')
+      [busca, document]
+  end
+
+  test "delete a document from index" do |busca, document|
+    first = busca.index(4, document)
+    second = busca.index(5, 'My name is julian porta and Im a good person')
+    busca.remove(second)
+    assert_equal busca.redis.call('GET', 'Busca:document:1'), "4"
+    assert_equal busca.redis.call('GET', 'Busca:document:2'), nil
+  end
+
+  test "should vacant an id upon document deletion" do |busca, document|
+    first = busca.index(11, document)
+    second = busca.index(12, 'My name is julian porta and Im a good person')
+    busca.remove(second)
+    assert_equal busca.redis.call('GET', 'Busca:document:2'), nil
+    assert_equal busca.redis.call('LINDEX', 'Busca:vacants', 0), "2"
+  end
+
+  test "should use the vacant id on new document index" do |busca, document|
+    first = busca.index(13, document)
+    assert_equal busca.redis.call('GET', 'Busca:document:1'), "13"
+    busca.redis.call('RPUSH', 'Busca:vacants', 3)
+    second = 'My name is julian porta and Im a good person'
+    busca.index(14, second)
+    assert_equal busca.redis.call('GET', 'Busca:document:3'), "14"
+  end
+end

data/tests/search_test.rb

@@ -0,0 +1,74 @@
+scope do
+
+  setup do
+    document = <<-eos
+Ay, marry, is't:
+But to my mind, though I am native here
+And to the manner born, it is a custom
+More honour'd in the breach than the observance.
+This heavy-headed revel east and west
+Makes us traduced and tax'd of other nations:
+They clepe us drunkards, and with swinish phrase
+Soil our addition; and indeed it takes
+From our achievements, though perform'd at height,
+The pith and marrow of our attribute.
+So, oft it chances in particular men,
+That for some vicious mole of nature in them,
+As, in their birth--wherein they are not guilty,
+Since nature cannot choose his origin--
+By the o'ergrowth of some complexion,
+Oft breaking down the pales and forts of reason,
+Or by some habit that too much o'er-leavens
+The form of plausive manners, that these men,
+Carrying, I say, the stamp of one defect,
+Being nature's livery, or fortune's star,--
+Their virtues else--be they as pure as grace,
+As infinite as man may undergo--
+Shall in the general censure take corruption
+From that particular fault: the dram of eale
+Doth all the noble substance of a doubt
+To his own scandal.
+    eos
+    compare = <<-eos
+Angels and ministers of grace defend us!
+Be thou a spirit of health or goblin damn'd,
+Bring with thee airs from heaven or blasts from hell,
+Be thy intents wicked or charitable,
+Thou comest in such a questionable shape
+That I will speak to thee: I'll call thee Hamlet,
+King, father, royal Dane: O, answer me!
+Let me not burst in ignorance; but tell
+Why thy canonized bones, hearsed in death,
+Have burst their cerements; why the sepulchre,
+Wherein we saw thee quietly inurn'd,
+Hath oped his ponderous and marble jaws,
+To cast thee up again. What may this mean,
+That thou, dead corse, again in complete steel
+Revisit'st thus the glimpses of the moon,
+Making night hideous; and we fools of nature
+So horridly to shake our disposition
+With thoughts beyond the reaches of our souls?
+Say, why is this? wherefore? what should we do?
+    eos
+    control = <<-eos
+This is a control string
+    eos
+    chongo = <<-eos
+Grace is here
+    eos
+    busca = Busca.new()
+    busca.redis.call('flushdb')
+
+    [busca, document, compare, control, chongo]
+  end
+
+  test "search a document" do |busca, document, compare, control, chongo|
+    first = busca.index(1, document)
+    second = busca.index(2, compare)
+    third = busca.index(3, control)
+    fourth = busca.index(4, chongo)
+    result = busca.search('grace')
+    assert_equal result, [1, 2, 4]
+  end
+  
+end

metadata

@@ -0,0 +1,129 @@
+--- !ruby/object:Gem::Specification
+name: busca
+version: !ruby/object:Gem::Version
+  version: 0.0.2
+platform: ruby
+authors:
+- Julián Porta
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-06-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: cutest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+- !ruby/object:Gem::Dependency
+  name: redic
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+- !ruby/object:Gem::Dependency
+  name: msgpack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+- !ruby/object:Gem::Dependency
+  name: filtra
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.2
+- !ruby/object:Gem::Dependency
+  name: separa
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.3
+description: Busca is a simple redis search that uses bitmaps to index words
+email:
+- julian@porta.sh
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- CHANGELOG.md
+- LICENSE
+- README.md
+- busca-0.0.1.gem
+- busca.gemspec
+- lib/busca.rb
+- lib/lua/index.lua
+- lib/lua/remove.lua
+- lib/lua/search.lua
+- makefile
+- tests/busca_test.rb
+- tests/helper.rb
+- tests/remove_test.rb
+- tests/search_test.rb
+homepage: https://github.com/Porta/busca
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.6
+signing_key: 
+specification_version: 4
+summary: Busca is a simple redis search
+test_files: []