ladder 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9239651dbb526c85d77203a89abcaf134ad353a4
-  data.tar.gz: e6c5b9a8bf16d50899a9d1ba9b5457d4bbd6c47b
+  metadata.gz: e349ffdb2f8ef8f274a8f9f00e0b45c3f46b4a70
+  data.tar.gz: 1c2122ef4ef2b18dc2ff74ac2ce59a7e5d5e38fa
 SHA512:
-  metadata.gz: d05d9401849734c2292ad8a4d0e1caf252132ee75b76c5a8941c440e0c54297097a75bf6bdbee5f5509fad19eb0ba511ad4284bfb5d55b4ac190b08033898d5d
-  data.tar.gz: 2fa63d70f71f806d83212032b1acde913963ccdf07e4336ee05ab6a0d0b86aea45e2f1866dbc226a2ae3081cd04a3dc88d68def4952a559d1d64527ec77ad074
+  metadata.gz: 63cb325392cdd3525780f8a46c75c5c0a7e70b94521bb23aea92a44e984fdcdc3596304609502add4911e80c22c369b409d2a8f4803ab3196510369e9d01573c
+  data.tar.gz: b3654ba928cc613c5eb12955c1c29593215434de4a2cede575b4d6d3788fda62b715677f3c6b85ebae52558d5c38d33f8d44319621aa96bc37af636e80b66672

data/.travis.yml

@@ -1,6 +1,6 @@
 language: ruby
 cache: bundler
-sudo: false
+#sudo: false
 rvm:
   - 1.9.3
   - 2.1.1
@@ -13,4 +13,6 @@ services:
   - mongodb
   - elasticsearch
 before_script:
+  - sudo /usr/share/elasticsearch/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/2.4.1
+  - sudo service elasticsearch restart
   - sleep 15

data/README.md

@@ -1,24 +1,23 @@
 ![Ladder logo](https://github.com/ladder/ladder/blob/master/logo.png)
 
-[![Gem Version](http://img.shields.io/gem/v/ladder.svg)](https://rubygems.org/gems/ladder) [![Build Status](https://travis-ci.org/ladder/ladder.svg)](https://travis-ci.org/ladder/ladder)
+[![Gem Version](http://img.shields.io/gem/v/ladder.svg)](https://rubygems.org/gems/ladder)  [![Inline docs](http://inch-ci.org/github/ladder/ladder.svg?branch=master)](http://inch-ci.org/github/ladder/ladder)  [![Build Status](https://travis-ci.org/ladder/ladder.svg)](https://travis-ci.org/ladder/ladder)
 
 # Ladder
 
-Ladder is a dynamic [Linked Data](http://en.wikipedia.org/wiki/Linked_data) framework for ActiveModel implemented as a series of Ruby modules that can be used individually and incorporated within existing frameworks (eg. [Project Hydra](http://projecthydra.org)), or combined as a comprehensive stack.
+Ladder is a dynamic framework for [Linked Data](http://en.wikipedia.org/wiki/Linked_data) modelling, persistence, and full-text indexing. It is implemented as a series of Ruby modules that can be used individually and incorporated within existing ActiveModel frameworks (eg. [Project Hydra](http://projecthydra.org)), or combined as a comprehensive stack.
 
-## History
+Ladder is intended to encourage the [GLAM](http://en.wikipedia.org/wiki/GLAM_(industry_sector)) community to think less dogmatically about established (often monolithic and/or niche) tools and instead embrace a broader vision of adopting more widely-used technologies.
 
-Ladder was loosely conceived over the course of several years prior to 2011.  In early 2012, Ladder began existence as an opportunity to escape from a decade of LAMP development and become familiar with Ruby.  From 2012 to late 2013, a closed prototype was built under the auspices of [Deliberate Data](http://deliberatedata.com) as a proof-of-concept to test the feasibility of the design.  Ladder is intended to encourage the [GLAM](http://en.wikipedia.org/wiki/GLAM_(industry_sector)) community to think less dogmatically about established (often monolithic and/or niche) tools and instead embrace a broader vision of adopting more widely-used technologies.
+### Components
 
-For those interested in the historical code, the original [prototype](https://github.com/ladder/ladder/tree/prototype) branch is available, as is an [experimental](https://github.com/ladder/ladder/tree/l2) branch.
+- [Mongoid](http://mongoid.org) for persistence
+- [ElasticSearch](http://www.elasticsearch.org) for full-text indexing
+- [ActiveTriples](https://github.com/no-reply/ActiveTriples) for linked data
+- [ActiveJob](https://github.com/rails/rails/tree/master/activejob) for background job execution
 
-## Components
+## History
 
-- Persistence ([Mongoid](http://mongoid.org)/MongoDB)
-- Full-text indexing ([ElasticSearch](http://www.elasticsearch.org))
-- RDF ([ActiveTriples](https://github.com/no-reply/ActiveTriples)/RDF.rb)
-- Asynchronous job execution ([Sidekiq](http://sidekiq.org)/Redis)
-- HTTP interaction ([Padrino](http://www.padrinorb.com)/Sinatra)
+Ladder was loosely conceived over the course of several years prior to 2011.  In early 2012, Ladder began existence as an opportunity to escape from a decade of LAMP development and become familiar with Ruby.  From 2012 to late 2013, a closed prototype was built under the auspices of [Deliberate Data](http://deliberatedata.com) as a proof-of-concept to test the feasibility of the design.  For those interested in the historical code, the original [prototype](https://github.com/ladder/ladder/tree/prototype) branch is available, as is an [experimental](https://github.com/ladder/ladder/tree/l2) branch.
 
 ## Installation
 
@@ -41,9 +40,11 @@ Or install it yourself as:
 * [Resources](#resources)
   * [Configuring Resources](#configuring-resources)
   * [Dynamic Resources](#dynamic-resources)
-* [Indexing for Search](#indexing-for-search)
 * [Files](#files)
+* [Indexing](#indexing)
+  * [Indexing Resources](#indexing-resources)
   * [Indexing Files](#indexing-files)
+  * [Background Indexing](#background-indexing)
 
 ### Resources
 
@@ -341,35 +342,101 @@ steve.as_jsonld
  #	}
 ```
 
-Note that due to the way Mongoid handles dynamic fields, dynamic properties **can not** be localized.  They can be any kind of literal, but they **can not** be a related object. They can, however, contain a reference to the related object's URI.
+Note that due to the way Mongoid handles dynamic fields, dynamic properties **can not** be localized.  They can be any kind of literal, but they **can not** be a related object. They can, however, contain the related object's URI.
 
-### Indexing for Search
+### Files
 
-You can also index your model classes for keyword searching through ElasticSearch by mixing in the Ladder::Searchable module:
+Files are bytestreams that store binary content using MongoDB's GridFS storage system.  They are still identifiable by a URI, and contain technical metadata about the File's contents.
 
 ```ruby
 class Person
   include Ladder::Resource
-  include Ladder::Searchable
 
   configure type: RDF::FOAF.Person
 
   property :first_name, predicate: RDF::FOAF.name
-  property :description, predicate: RDF::DC.description
+  property :thumbnails, predicate: RDF::FOAF.depiction, class_name: 'Image', inverse_of: nil
 end
 
-kimchy = Person.new(first_name: 'Shay', description: 'Real genius')
-=> #<Person _id: 543b457b41697231c5000000, first_name: {"en"=>"Shay"}, description: {"en"=>"Real genius"}>
+class Image
+  include Ladder::File
+end
 ```
 
-In order to enable indexing, call the `#index_for_search` method on the class:
+Similar to Resources, using `#property` will create a has-many relation for a File by default; however, because Files must be the target of a one-way relation, the `inverse_of: nil` option is required. Note that due to the way GridFS is designed, Files **can not** be embedded.
 
 ```ruby
-Person.index_for_search
-=> :as_indexed_json
+steve = Person.new(first_name: 'Steve')
+=> #<Person _id: 549d83c64169720b32010000, first_name: {"en"=>"Steve"}>
 
-kimchy.as_indexed_json
-=> {"description"=>"Real genius", "first_name"=>"Shay"}
+thumb = Image.new(file: open('http://some.image/pic.jpg'))
+=> #<Image _id: 549d83c24169720b32000000>
+
+steve.thumbnails << thumb
+=> [#<Image _id: 549d83c24169720b32000000, >]
+
+steve.as_jsonld
+ # => {
+ #     "@context": {
+ #         "foaf": "http://xmlns.com/foaf/0.1/"
+ #     },
+ #     "@id": "http://example.org/people/549d83c64169720b32010000",
+ #     "@type": "foaf:Person",
+ #     "foaf:depiction": {
+ #         "@id": "http://example.org/images/549d83c24169720b32000000"
+ #     },
+ #     "foaf:name": {
+ #         "@language": "en",
+ #         "@value": "Steve"
+ #     }
+ # }
+
+steve.save
+ # ... File is stored to GridFS ...
+=> true
+```
+
+Files have all the attributes of a GridFS file, and the stored binary content is accessed using `#data`.
+
+```ruby
+thumb.reload
+=> #<Image _id: 549d86184169720b6a000000, >
+
+thumb.as_document
+=> {"_id"=>BSON::ObjectId('549d86184169720b6a000000'),
+ "length"=>59709,
+ "chunkSize"=>4194304,
+ "uploadDate"=>2014-12-26 16:00:29 UTC,
+ "md5"=>"0d4a486e2cd71c51b7a92cfe96f29324",
+ "contentType"=>"image/jpeg",
+ "filename"=>"549d86184169720b6a000000/open-uri20141226-2922-u66ap6"}
+
+thumb.length
+=> 59709
+
+thumb.data
+=> # ... binary data ...
+```
+
+### Indexing
+
+#### Indexing Resources
+
+You can index Resources for keyword searching by mixing in the Ladder::Searchable module:
+
+```ruby
+class Person
+  include Ladder::Resource
+  include Ladder::Searchable
+
+  configure type: RDF::FOAF.Person
+
+  property :first_name, predicate: RDF::FOAF.name
+  property :description, predicate: RDF::DC.description
+end
+
+kimchy = Person.new(first_name: 'Shay', description: 'Real genius')
+=> #<Person _id: 543b457b41697231c5000000, first_name: {"en"=>"Shay"}, description: {"en"=>"Real genius"}>
 
 kimchy.save
 => true
@@ -393,10 +460,10 @@ results.records.first == kimchy
 => true
 ```
 
-When indexing, you can control how your model is stored in the index by supplying the `as: :jsonld` or `as: :qname` options:
+When indexing, you can control how your model is stored in the index by calling `#index_for_search` and supplying a block that returns a serializable hash:
 
 ```ruby
-Person.index_for_search as: :jsonld
+Person.index_for_search { as_jsonld }
 => :as_indexed_json
 
 kimchy.as_indexed_json
@@ -417,7 +484,7 @@ kimchy.as_indexed_json
  #   }
  # }
 
-Person.index_for_search as: :qname
+Person.index_for_search { as_qname }
 => :as_indexed_json
 
 kimchy.as_indexed_json
@@ -434,7 +501,7 @@ kimchy.as_indexed_json
  # }
 ```
 
-You can also index related objects as framed JSON-LD or hierarchical qname, by again using the `related: true` option:
+You can also index related objects as framed JSON-LD using `#as_framed_jsonld` or by using the `related: true` option with `#as_qname`:
 
 ```ruby
 class Project
@@ -456,10 +523,10 @@ es = Project.new(project_name: 'ElasticSearch', description: 'You know, for sear
 es.developers << kimchy
 => [#<Person _id: 543b457b41697231c5000000, first_name: {"en"=>"Shay"}, description: {"en"=>"Real genius"}, project_ids: [BSON::ObjectId('544562c24169728b4e010000')]>]
 
-Person.index_for_search as: :jsonld, related: true
+Person.index_for_search { as_framed_jsonld }
 => :as_indexed_json
 
-Project.index_for_search as: :jsonld, related: true
+Project.index_for_search { as_framed_jsonld }
 => :as_indexed_json
 
 kimchy.as_indexed_json
@@ -530,10 +597,10 @@ es.as_indexed_json
  #    }
  # }
 
-Person.index_for_search as: :qname, related: true
+Person.index_for_search { as_qname related: true }
 => :as_indexed_json
 
-Project.index_for_search as: :qname, related: true
+Project.index_for_search { as_qname related: true }
 => :as_indexed_json
 
 kimchy.as_indexed_json
@@ -587,83 +654,9 @@ es.as_indexed_json
  # }
 ```
 
-### Files
-
-Files are bytestreams that store binary content using MongoDB's GridFS storage system.  They are still identifiable by a URI, and contain technical metadata about the File's contents.
-
-```ruby
-class Person
-  include Ladder::Resource
-
-  configure type: RDF::FOAF.Person
-
-  property :first_name, predicate: RDF::FOAF.name
-  property :thumbnails, predicate: RDF::FOAF.depiction, class_name: 'Image', inverse_of: nil
-end
-
-class Image
-  include Ladder::File
-end
-```
-
-Similar to Resources, using `#property` as above will create a has-many relation for a File by default; however, because Files must be the target of a one-way relation, the `inverse_of: nil` option is required. Note that due to the way GridFS is designed, Files **can not** be embedded.
-
-```ruby
-steve = Person.new(first_name: 'Steve')
-=> #<Person _id: 549d83c64169720b32010000, first_name: {"en"=>"Steve"}>
-
-thumb = Image.new(file: open('http://some.image/pic.jpg'))
-=> #<Image _id: 549d83c24169720b32000000>
-
-steve.thumbnails << thumb
-=> [#<Image _id: 549d83c24169720b32000000, >]
-
-steve.as_jsonld
- # => {
- #     "@context": {
- #         "foaf": "http://xmlns.com/foaf/0.1/"
- #     },
- #     "@id": "http://example.org/people/549d83c64169720b32010000",
- #     "@type": "foaf:Person",
- #     "foaf:depiction": {
- #         "@id": "http://example.org/images/549d83c24169720b32000000"
- #     },
- #     "foaf:name": {
- #         "@language": "en",
- #         "@value": "Steve"
- #     }
- # }
-
-steve.save
- # ... File is stored to GridFS ...
-=> true
-```
-
-Files have all the attributes of a GridFS file, and the stored binary content is accessed using `#data`.
-
-```ruby
-thumb.reload
-=> #<Image _id: 549d86184169720b6a000000, >
-
-thumb.as_document
-=> {"_id"=>BSON::ObjectId('549d86184169720b6a000000'),
- "length"=>59709,
- "chunkSize"=>4194304,
- "uploadDate"=>2014-12-26 16:00:29 UTC,
- "md5"=>"0d4a486e2cd71c51b7a92cfe96f29324",
- "contentType"=>"image/jpeg",
- "filename"=>"549d86184169720b6a000000/open-uri20141226-2922-u66ap6"}
-
-thumb.length
-=> 59709
-
-thumb.data
-=> # ... binary data ...
-```
-
 #### Indexing Files
 
-Files that contain textual content (eg. HTML, PDF, ePub, DOC, etc) can be automatically indexed when they are persisted, again just by mixing in the Ladder::Searchable module (there is no need to call `#index_for_search` on the class).  Note that this requires the [Mapper Attachments Plugin for Elasticsearch](https://github.com/elasticsearch/elasticsearch-mapper-attachments) to be installed.
+Files that contain textual content (eg. HTML, PDF, ePub, DOC, etc) can be indexed when they are stored, again by mixing in the Ladder::Searchable module.  This can be useful if you want to retrieve a File by searching for the textual content that it contains.  Note that this requires the [Mapper Attachments Plugin for Elasticsearch](https://github.com/elasticsearch/elasticsearch-mapper-attachments) to be installed.
 
 ```ruby
 class OCR
@@ -705,7 +698,7 @@ results.records.first.data
 => # ... binary data ...
 ```
 
-This can be useful if you want to retrieve a File by searching for the textual content that it contains.  Note the use of `#records` to access the Ladder::File instances directly ([see here for more information](https://github.com/elasticsearch/elasticsearch-rails/tree/master/elasticsearch-model#search-results-as-database-records)).  However, if you want to get information about the file characteristics (including the extracted textual content), you can use a modified search query:
+Note the use of `#records` to access the Ladder::File instances directly ([see here for more information](https://github.com/elasticsearch/elasticsearch-rails/tree/master/elasticsearch-model#search-results-as-database-records)).  However, if you want to get information about the file characteristics (including the extracted textual content), you can use a modified search query:
 
 ```ruby
 results = OCR.search 'Moomintroll', fields: '*'
@@ -754,7 +747,37 @@ results.first.highlight.file
     ", but at the same time <em>his</em> nose caught a new smell. It was a more \nserious smell than any he had met"]
 ```
 
-More information about performing highlighting queries is available in the [Elasticsearch documentation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html).
+More information about highlighting queries is available in the [Elasticsearch documentation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html).
+
+#### Background Indexing
+
+In large-scale production environments, sending an HTTP request to Elasticsearch during the database transaction isn't optimal (especially for large Files), so Ladder uses ActiveJob to queue and process indexing operations in the background.  Just use the Ladder::Searchable::Background module in your model:
+
+```ruby
+class OCR
+  include Ladder::File
+  include Ladder::Searchable::Background
+end
+
+ # ...
+ 
+class Person
+  include Ladder::Resource
+  include Ladder::Searchable::Background
+
+  configure type: RDF::FOAF.Person
+end
+
+ # ...
+```
+
+You'll also have to set the queue adapter in your application, depending on which backend you're using:
+
+```ruby
+ActiveJob::Base.queue_adapter = :sidekiq
+```
+
+For more information on available queueing adapters and their features, see the [ActiveJob documentation](http://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html).
 
 ## Contributing
 
@@ -779,4 +802,4 @@ Many thanks to Christopher Knight [@NomadicKnight](https://twitter.com/Nomadic_K
 ## License
 
 Apache License Version 2.0
-http://apache.org/licenses/LICENSE-2.0.txt
+http://apache.org/licenses/LICENSE-2.0.txt

data/ladder.gemspec

@@ -20,17 +20,18 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
+  spec.add_dependency "active-triples", "~> 0.6"
+  spec.add_dependency "activejob", "~> 4.2"
+  spec.add_dependency "elasticsearch-model", "~> 0.1"
   spec.add_dependency "mongoid", "~> 4.0"
   spec.add_dependency "mongoid-grid_fs", "~> 2.1"
-  spec.add_dependency "active-triples", "~> 0.4"
-  spec.add_dependency "elasticsearch-model", "~> 0.1"
-  spec.add_dependency "mimemagic", "~> 0.2"
 
   spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "mimemagic", "~> 0.2"
   spec.add_development_dependency "pry", "~> 0.10"
-  spec.add_development_dependency "wirble", "~> 0.1"
   spec.add_development_dependency "rspec", "~> 3.1"
   spec.add_development_dependency "rake", "~> 10.4"
-  spec.add_development_dependency "yard", "~> 0.8"
   spec.add_development_dependency "simplecov", "~> 0.9"
+  spec.add_development_dependency "wirble", "~> 0.1"
+  spec.add_development_dependency "yard", "~> 0.8"
 end

data/lib/ladder/file.rb

@@ -10,7 +10,7 @@ module Ladder::File
   included do
     configure base_uri: RDF::URI.new(LADDER_BASE_URI) / name.underscore.pluralize if defined? LADDER_BASE_URI
 
-    store_in :collection => "#{ grid.prefix }.files"
+    store_in collection: "#{ grid.prefix }.files"
 
     # Define accessor methods for attributes
     define_method(:content_type) { read_attribute(:contentType) }
@@ -18,24 +18,12 @@ module Ladder::File
     grid::File.fields.keys.map(&:to_sym).each do |attr|
       define_method(attr) { read_attribute(attr) }
     end
-  end
-
-  attr_accessor :file
-
-  ##
-  # Make save behave like Mongoid::Document as much as possible
-  def save
-    # FIXME: this raises on a freshly-retrieved document; should either set @file or catch @grid_file
-    raise Mongoid::Errors::InvalidValue.new(IO, NilClass) if file.nil?
 
-    persisted? ? run_callbacks(:update) : run_callbacks(:create)
-
-    run_callbacks(:save) do
-      attributes[:content_type] = file.content_type if file.respond_to? :content_type
-      @grid_file ? @grid_file.save : !! @grid_file = self.class.grid.put(file, attributes.symbolize_keys)
-    end
+    around_save :save_file
   end
 
+  attr_accessor :file
+  
   ##
   # Output content of object from stored file or readable input
   def data
@@ -51,6 +39,17 @@ module Ladder::File
   def update_resource
     resource
   end
+  
+  private
+
+    ##
+    # Make save behave like Mongoid::Document as much as possible
+    def save_file(&block)
+      attributes[:content_type] = file.content_type if file.respond_to? :content_type
+      @grid_file ? @grid_file.save : !! @grid_file = self.class.grid.put(file, attributes.symbolize_keys)
+
+      persisted? ? run_callbacks(:update) : run_callbacks(:create)
+    end
 
   module ClassMethods
     ##
@@ -58,7 +57,6 @@ module Ladder::File
     def grid
      @grid ||= Mongoid::GridFs.build_namespace_for name
     end
-
   end
 
 end