lexster 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e58713a9965ca82acd9d981adaa2451bbe1c85f4
+  data.tar.gz: 336d87ffd60736d34967f33edbd33b51d952d866
+SHA512:
+  metadata.gz: 1df55ed5565f0c84e31764481c1fafbfbc608fac158eefe6282c87906dcb7dafd13f0cdba678e66a70559337ee1cdb38e64745a30c51292f34d53fe0a7494075
+  data.tar.gz: a1303d614c49e858196be1a8cf33fea1c9408ae2d38a5f6e662b116ee1ddb9f88231dfb13ba1d99f719a27c8180ede559589ff7a3267e4bbe9fbaf5750f69f5d

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+.DS_Store

data/.rspec

@@ -0,0 +1 @@
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+script: "bundle exec rake neo4j:install neo4j:start spec --trace"
+rvm:
+  - 1.9.2
+  - 1.9.3

data/CHANGELOG.md

@@ -0,0 +1,58 @@
+## v0.1.1
+
+* Added batch support, for much faster intiialization of current DB or reindexing all DB.
+* Dropped indexes per model, instead, using `node_auto_index` and `relationship_auto_index`, letting Neo4j auto index objects.
+* One `neo_save` method instead of `neo_create` and `neo_update`. It takes care of inserting or updating.
+
+### Breaking changes:
+
+Model indexes (such as `users_index`) are now turned off by default. Instead, Lexster uses Neo4j's auto indexing feature.
+
+In order to have the model indexes back, use this in your configuration:
+
+```ruby
+Lexster.configure do |c|
+  c.enable_per_model_indexes = true
+end
+```
+
+This will turn on for all models.
+
+You can turn off for a specific model with:
+
+```ruby
+class User < ActiveRecord::Base
+  include Lexster::Node
+  
+  lexsterable enable_model_index: false do |c|
+  end
+end
+```
+
+## v0.0.51
+
+* Releasing Lexster as a gem.
+
+## v0.0.41
+
+* fixed really annoying bug caused by Rails design -- Rails doesn't call `after_destroy` when assigning many to many relationships to a model, like `user.movies = [m1, m2, m3]` or `user.update_attributes(params[:user])` where it contains `params[:user][:movie_ids]` list (say from checkboxes), but it DOES CALL after_create for the new relationships. the fix adds after_remove callback to the has_many relationships, ensuring neo4j is up to date with all changes, no matter how they were committed
+
+## v0.0.4
+
+* rewrote seacrch. one index for all types instead of one for type. please run neo_search_index on all of your models.
+  search in multiple types at once with `Lexster.search(types_array, term)
+
+## v0.0.3
+
+* new configuration syntax (backwards compatible)
+* full text search index
+
+## v0.0.2
+
+* create node immediately after active record create
+* logging
+* bug fixes
+
+## v0.0.1
+
+* initial release

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in lexster.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2012 Elad Ossadon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,467 @@
+# Lexster
+
+[![Build Status](https://secure.travis-ci.org/elado/lexster.png)](http://travis-ci.org/elado/lexster)
+
+
+Make your ActiveRecords stored and searchable on Neo4j graph database, in order to make fast graph queries that MySQL would crawl while doing them.
+
+Lexster to Neo4j is like Sunspot to Solr. You get the benefits of Neo4j speed while keeping your schema on your plain old RDBMS.
+
+Lexster doesn't require JRuby. It's based on the great [Neography](https://github.com/maxdemarzi/neography) gem which uses Neo4j's REST API.
+
+Lexster offers querying Neo4j for IDs of objects and then fetch them from your RDBMS, or storing all desired data on Neo4j.
+
+**Important: Heroku Support is not available because Herokud doesn't support Gremlin. So until further notice, easiest way is to self host a Neo4j on EC2 in the same zone, and connect from your dyno to it**
+
+## Changelog
+
+[See Changelog](https://github.com/elado/lexster/blob/master/CHANGELOG.md). Including some breaking changes (and solutions) from previos versions.
+
+
+## Installation
+
+Add to your Gemfile and run the `bundle` command to install it.
+
+```ruby
+gem 'lexster', '~> 0.1.1'
+```
+
+**Requires Ruby 1.9.2 or later.**
+
+## Usage
+
+### Rails app configuration:
+
+In an initializer, such as `config/initializers/01_neo4j.rb`:
+
+```ruby
+ENV["NEO4J_URL"] ||= "http://localhost:7474"
+
+uri = URI.parse(ENV["NEO4J_URL"])
+
+$neo = Neography::Rest.new(uri.to_s)
+
+Neography.configure do |c|
+  c.server = uri.host
+  c.port = uri.port
+
+  if uri.user && uri.password
+    c.authentication = 'basic'
+    c.username = uri.user
+    c.password = uri.password
+  end
+end
+
+Lexster.db = $neo
+
+Lexster.configure do |c|
+  # should Lexster create sub-reference from the ref node (id#0) to every node-model? default: true
+  c.enable_subrefs = true
+end
+```
+
+`01_` in the file name is in order to get this file loaded first, before the models (initializers are loaded alphabetically).
+
+If you have a better idea (I bet you do!) please let me know.
+
+
+### ActiveRecord configuration
+
+#### Nodes
+
+For nodes, first include the `Lexster::Node` module in your model:
+
+
+```ruby
+class User < ActiveRecord::Base
+  include Lexster::Node
+end
+```
+
+This will help to create/update/destroy a corresponding node on Neo4j when changed are made a User model.
+
+Then, you can customize what fields will be saved on the node in Neo4j, inside `lexsterable` configuration, using `field`. You can also pass blocks to save content that's not a real column:
+
+```ruby
+class User < ActiveRecord::Base
+  include Lexster::Node
+  
+  lexsterable do |c|
+    c.field :slug
+    c.field :display_name
+    c.field :display_name_length do
+      self.display_name.length
+    end
+  end
+end
+```
+
+#### Relationships
+
+Let's assume that a `User` can `Like` `Movie`s:
+
+
+```ruby
+# user.rb
+
+class User < ActiveRecord::Base
+  include Lexster::Node
+
+  has_many :likes
+  has_many :movies, through: :likes
+
+  lexsterable do |c|
+    c.field :slug
+    c.field :display_name
+  end
+end
+
+
+# movie.rb
+
+class Movie < ActiveRecord::Base
+  include Lexster::Node
+
+  has_many :likes
+  has_many :users, through: :likes
+
+  lexsterable do |c|
+    c.field :slug
+    c.field :name
+  end
+end
+
+
+# like.rb
+
+class Like < ActiveRecord::Base
+  belongs_to :user
+  belongs_to :movie
+end
+```
+
+
+Now let's make the `Like` model a Lexster, by including the `Lexster::Relationship` module, and define the relationship (start & end nodes and relationship type) options with `lexsterable` config and `relationship` method:
+
+
+```ruby
+class Like < ActiveRecord::Base
+  belongs_to :user
+  belongs_to :movie
+
+  include Lexster::Relationship
+
+  lexsterable do |c|
+    c.relationship start_node: :user, end_node: :movie, type: :likes
+  end
+end
+```
+
+Lexster adds the metohds `neo_node` and `neo_relationships` to instances of nodes and relationships, respectively.
+
+So you could do:
+
+```ruby
+user = User.create!(display_name: "elado")
+user.movies << Movie.create("Memento")
+user.movies << Movie.create("Inception")
+
+user.neo_node                # => #<Neography::Node…>
+user.neo_node.display_name   # => "elado"
+
+rel = user.likes.first.neo_relationship
+rel.start_node  # user.neo_node
+rel.end_node    # user.movies.first.neo_node
+rel.rel_type    # 'likes'
+```
+
+#### Disabling auto saving to Neo4j:
+
+If you'd like to save nodes manually rather than after_save, use `auto_index: false`:
+
+```ruby
+class User < ActiveRecord::Base
+  include Lexster::Node
+  
+  lexsterable auto_index: false do |c|
+  end
+end
+
+user = User.create!(name: "Elad") # no node is created in Neo4j!
+
+user.neo_save # now there is!
+```
+
+## Querying
+
+You can query with all [Neography](https://github.com/maxdemarzi/neography)'s API: `traverse`, `execute_query` for Cypher, and `execute_script` for Gremlin.
+
+### Basics:
+
+#### Finding a node by ID
+
+Nodes and relationships are auto indexed in the `node_auto_index` and `relationship_auto_index` indexes, where the key is `Lexster::UNIQUE_ID_KEY` (which is 'lexster_unique_id') and the value is a combination of the class name and model id, `Movie:43`, this value is accessible with `model.neo_unique_id`. So use the constant and this method, never rely on assebling those values on your own because they might change in the future.
+
+That means, you can query like this:
+
+```ruby
+Lexster.db.get_node_auto_index(Lexster::UNIQUE_ID_KEY, user.neo_unique_id)
+# => returns a Neography hash
+
+Lexster::Node.from_hash(Lexster.db.get_node_auto_index(Lexster::UNIQUE_ID_KEY, user.neo_unique_id))
+# => returns a Neography::Node
+```
+
+#### Finding all nodes of type
+
+If Subreferences are enabled, you can get the subref node and then get all attached nodes:
+
+```ruby
+Lexster.ref_node.outgoing('users_subref').first.outgoing('users').to_a
+# => this, according to Neography, returns an array of Neography::Node so no conversion is needed
+```
+
+### Gremlin Example:
+
+These examples query Neo4j using Gremlin for IDs of objects, and then fetches them from ActiveRecord with an `in` query.
+
+Of course, you can store using the `lexsterable do |c| c.field ... end` all the data you need in Neo4j and avoid querying ActiveRecord.
+
+
+**Most liked movies**
+
+```ruby
+gremlin_query = <<-GREMLIN
+  m = [:]
+
+  g.v(0)
+    .out('movies_subref').out
+      .inE('likes')
+      .inV
+      .groupCount(m).iterate()
+
+  m.sort{-it.value}.collect{it.key.ar_id}
+GREMLIN
+
+movie_ids = Lexster.db.execute_script(gremlin_query)
+
+Movie.where(id: movie_ids)
+```
+
+*Side note: the resulted movies won't be sorted by like count because the RDBMS won't necessarily do it as we passed a list of IDs. You can sort it yourself with array manipulation, since you have the ids.*
+
+
+**Movies of user friends that the user doesn't have**
+
+Let's assume we have another `Friendship` model which is a relationship with start/end nodes of `user` and type of `friends`,
+
+```ruby
+user = User.find(1)
+
+gremlin_query = <<-GREMLIN
+  u = g.idx('node_auto_index').get(unique_id_key, user_unique_id).next()
+  movies = []
+
+  u
+    .out('likes').aggregate(movies).back(2)
+    .out('friends').out('likes')
+    .dedup
+    .except(movies).collect{it.ar_id}
+GREMLIN
+
+movie_ids = Lexster.db.execute_script(gremlin_query, unique_id_key: Lexster::UNIQUE_ID_KEY, user_unique_id: user.neo_unique_id)
+
+Movie.where(id: movie_ids)
+```
+
+## Full Text Search
+
+### Index for Full-Text Search
+
+Using `search` block inside a `lexsterable` block, you can store certain fields.
+
+```ruby
+# movie.rb
+
+class Movie < ActiveRecord::Base
+  include Lexster::Node
+
+  lexsterable do |c|
+    c.field :slug
+    c.field :name
+
+    c.search do |s|
+      # full-text index fields
+      s.fulltext :name
+      s.fulltext :description
+
+      # just index for exact matches
+      s.index :year
+    end
+  end
+end
+```
+
+Records will be automatically indexed when inserted or updated.
+
+### Querying a Full-Text Search index
+
+```ruby
+# will match all movies with full-text match for name/description. returns ActiveRecord instanced
+Movie.neo_search("*hello*").results
+
+# same as above but returns hashes with the values that were indexed on Neo4j
+Movie.search("*hello*").hits
+
+# search in multiple types
+Lexster.neo_search([Movie, User], "hello")
+
+# search with exact matches (pass a hash of field/value)
+Movie.neo_search(year: 2013).results
+```
+
+Full text search with Lexster is very limited and is likely not to develop more than this basic functionality. I strongly recommend using gems like Sunspot over Solr.
+
+## Batches
+
+Lexster has a batch ability, that is good for mass updateing/inserting of nodes/relationships. It sends batched requests to Neography, and takes care of type conversion (neography batch returns hashes and other primitive types) and "after" actions (via promises).
+
+A few examples, easy to complex:
+
+```ruby
+Lexster.batch(batch_size: 100) do
+  User.all.each(&:neo_save)
+end
+```
+With `then`:
+
+```ruby
+User.first.name # => "Elad"
+
+Lexster.batch(batch_size: 100) do
+  User.all.each(&:neo_save)
+end.then do |results|
+  # results is an array of the script results from neo4j REST.
+
+  results[0].name # => "Elad"
+end
+```
+
+*Nodes and relationships in the results are automatically converted to Neography::Node and Neography::Relationship, respectively.*
+
+With individual `then` as well as `then` for the entire batch:
+
+```ruby
+Lexster.batch(batch_size: 30) do |batch|
+  (1..90).each do |i|
+    (batch << [:create_node, { name: "Hello #{i}" }]).then { |result| puts result.name }
+  end
+end.then do |results|
+  puts results.collect(&:name)
+end
+```
+
+When in a batch, `neo_save` adds gremlin scripts to a batch, instead of running them immediately. The batch flushes whenever the `batch_size` option is met.
+So even if you have 20000 users, Lexster will insert/update in smaller batches. Default `batch_size` is 200.
+
+
+## Inserting records of existing app
+
+If you have an existing database and just want to integrate Lexster, configure the `lexsterable`s and run in a rake task or console.
+
+Use batches! It's free, and much faster. Also, you should use `includes` to incude the relationship edges on relationship entities, so it doesn't query the DB on each relationship.
+
+```ruby
+Lexster.batch do
+  [ Like.includes(:user).includes(:movie), OtherRelationshipModel.includes(:from_model).includes(:to_model) ].each { |model| model.all.each(&:neo_save) }
+
+  NodeModel.all.each(&:neo_save)
+end
+```
+
+This will loop through all of your relationship records and generate the two edge nodes along with a relationship (eager loading for better performance).
+The second line is for nodes without relationships.
+
+For large data sets use pagination.
+Better interface for that in the future.
+
+
+## Behind The Scenes
+
+Whenever the `neo_node` on nodes or `neo_relationship` on relationships is called, Lexster checks if there's a corresponding node/relationship in Neo4j (with the auto indexes). If not, it does the following:
+
+### For Nodes:
+
+1. Ensures there's a sub reference node (read [here](http://docs.neo4j.org/chunked/stable/tutorials-java-embedded-index.html) about sub references), if that option is on.
+2. Creates a node based on the ActiveRecord, with the `id` attribute and all other attributes from `lexsterable`'s field list
+3. Creates a relationship between the sub reference node and the newly created node
+4. Auto indexes a node in the auto index, for fast lookup in the future
+
+Then, when it needs to find it again, it just seeks the auto index with that ActiveRecord id.
+
+### For Relationships:
+
+Like Nodes, it uses an auto index, to look up a relationship by ActiveRecord id
+
+1. With the options passed in the `lexsterable`, it fetches the `start_node` and `end_node`
+2. Then, it calls `neo_node` on both, in order to create the Neo4j nodes if they're not created yet, and creates the relationship with the type from the options.
+3. Adds the relationship to the relationship index.
+
+## Testing
+
+In order to test your app or this gem, you need a running Neo4j database, dedicated to tests.
+
+I use port 7574 for testing.
+
+To run another database locally (read [here](http://docs.neo4j.org/chunked/1.9.M03/server-installation.html#_multiple_server_instances_on_one_machine) too):
+
+Copy the entire Neo4j database folder to a different location,
+
+**or**
+
+symlink `bin`, `lib`, `plugins`, `system`, copy `conf` to a single folder, and create an empty `data` folder.
+
+Then, edit `conf/neo4j-server.properties` and set the port (`org.neo4j.server.webserver.port`) from 7474 to 7574 and run the server with `bin/neo4j start`
+
+## Testing Your App with Lexster (RSpec)
+
+In `environments/test.rb`, add:
+
+```ruby
+ENV["NEO4J_URL"] = 'http://localhost:7574'
+```
+
+In your `spec_helper.rb`, add the following configurations:
+
+```ruby
+config.before :all do
+  Lexster.clean_db(:yes_i_am_sure)
+end
+
+config.before :each do
+  Lexster.reset_cached_variables
+end
+```
+
+## Testing This Gem
+
+Run the Neo4j DB on port 7574, and run `rake` from the gem folder.
+
+## Contributing
+
+Please create a [new issue](https://github.com/elado/lexster/issues) if you run into any bugs. Contribute patches via pull requests. Write tests and make sure all tests pass.
+
+
+## Heroku Support
+
+Unfortunately, as for now, Neo4j add-on on Heroku doesn't support Gremlin. Therefore, this gem won't work on Heroku's add on. You should self-host a Neo4j instance on an EC2 or any other server.
+
+
+## TO DO
+
+[TO DO](https://github.com/elado/lexster/blob/master/TODO.md)
+
+
+---
+
+Developed by [@elado](http://twitter.com/elado)