ansr 0.0.1 → 0.0.3

checksums.yaml

@@ -0,0 +1,15 @@
+---
+!binary "U0hBMQ==":
+  metadata.gz: !binary |-
+    YjdhOTQ4ZmY2ZTY3OTFkM2YwZjQwNDI3MWI4NmEzYTYwMTc3YzkwZg==
+  data.tar.gz: !binary |-
+    ZDI1ODllMmMxOGMxMjg5MDI3MWEyYjIwZjQ5ODdkYWNhOGZjZTZhOA==
+SHA512:
+  metadata.gz: !binary |-
+    MzhhMjYwMTA5YjljZjFhZmU3M2RmOWViNjZkMzg3YzA2ODIxOTRlYzc2M2Yy
+    NWUyNDI2NzY1ZTY5YzQ1ZTJhMTNmNzlmOWJjYWM1MzJlYjUyY2RiZjNkNzIx
+    ZjVlNzMyZjBlNzBjM2ZjOTNkMmI5NmRkOTZhZGQxNjJiYmI5YzE=
+  data.tar.gz: !binary |-
+    ZTlhMjE0NjczMjk0NDk1ZThmMzlkMGY0NWYwYzQ1OTU5NWVhMDA2NDY0N2Ux
+    YjE1ZGI5NTdiYTRmNGRjZmMyMmNjMzQ2ZWVlYjU4OTllN2FhNTlhYzNhMjk4
+    NDc1NjJmYTZhMGM0YTEzOTNkMjRmYmJmZDQzZTcxODQxMmI2Mjg=

data/README.md

@@ -0,0 +1,35 @@
+ansr
+====
+
+ActiveRecord(No-SQL)::Relation + Blacklight
+
+Ansr is a library for building ActiveRecord-style models and Relation implementations that query no-SQL data sources.
+Ansr is motivated by a proposed refactoring of Blacklight at Code4Lib 2014.
+
+[Blacklight](https://github.com/projectblacklight/blacklight) (BL) defines itself as “an open source Solr user interface discovery platform.” The coupling to Solr is evident in the structure: Solr querying facilities are sprinkled throughout several mixins that are included in BL controllers. This results in a codebase that cannot, as is regularly asked on the mailing lists, be used in front of another document store (eg ElasticSeach).  But this is not necessarily the case.
+
+BL might be refactored to locate the actual Solr querying machinery behind the core model of BL apps (currently called SolrDocument).  Refactoring the codebase this way would realize several benefits:
+
+1.   Adherence to the Principle of Least Surprise: The BL document model would behave more like a Rails model backed by RDBMS. When bringing new developers into a BL project, familiarity with the standard patterns of Rails would translate more immediately to the BL context.
+
+2.   Flexible abstraction of the document store: Moving the specifics of querying the document store would make the introduction of models interacting with other stores possible. For example, the DPLA REST API exposes some Solr-like concepts, and a proof-of-concept model for an ActiveRecord-like approach to searching them can be seen at (https://github.com/barmintor/ansr/tree/master/ansr_dpla)
+
+3.   Clearer testing strategies and ease of console debugging
+
+4.   Clarification of BL’s relationship to RSolr as the provider to an analog for ActiveRecord::Relation
+
+What would such a refactor require?
+
+1.   A definition of the backend requirements of BL beyond a reference to Solr per se: indexed documents with fields, a concept of facets corresponding to the Solr/Lucene definitions, the ability to expose Hash-like representations of results.
+
+2.   A relocation of the searching methods from Blacklight::Catalog and Blacklight::SolrHelper into a model generated to include Solr code
+
+3.   An accommodation of controller-specific Solr configuration, possibly resolved by having the BL config register Solr parms with the model a la SolrDocument extensions in BL 4
+
+4.   An abstraction of the fielded/faceted search parameters to mimic ActiveRecord limits
+
+5.   A partner institution capable of producing integration and system testing support for, at minimum, another Lucene-backed document store (ElasticSearch)
+
+Since these changes are incompatible with current BL, they are proposed as a principal feature of BL 6.0.
+
+An example of the kinds of models to be implemented can be seen in a [DPLA proof of concept](https://github.com/barmintor/ansr/tree/master/ansr_dpla).

data/ansr.gemspec

@@ -14,7 +14,8 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_dependency 'loggable'
-  spec.add_dependency 'blacklight', '>=5.1.0'
+  spec.add_dependency 'arel'
+  spec.add_dependency 'activerecord'
 
   spec.add_development_dependency("rake")
   spec.add_development_dependency("bundler", ">= 1.0.14")

data/ansr_blacklight/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/ansr_blacklight/README.md

@@ -0,0 +1,37 @@
+Ansr::Blacklight
+=================
+
+A re-implementation of Blacklight's Solr model with find/search functionality moved behind ActiveRecord::Relation subclasses.
+
+QUESTIONS
+
+Is a closer conformation to the expectations from ActiveRecord valuable enough to forego use of Sunspot (https://github.com/sunspot/sunspot)?
+
+REQUEST REQUIREMENTS
+
+Considering the following block from the BL Solr request code:
+  SINGULAR_KEYS = %W{ facet fl q qt rows start spellcheck spellcheck.q sort 
+  per_page wt hl group defType}
+  ARRAY_KEYS = %W{facet.field facet.query facet.pivot fq hl.fl }
+
+facet : a boolean field indicating the requested presence of facet info in response
+fl : the selected fields
+q : the query (fielding?)
+qt : query type; indicates queryHandler in Solr
+rows : corresponds to limit
+start : corresponds to offset
+spellcheck : boolean?
+spellcheck.q : ?
+sort : ?
+facet.field : the fields for which facet info is requested
+facet.query : ?
+facet.pivot : ?
+fq : ?
+hl.fl : field to highlight
+How is facet query different from filter query (fq)?
+
+Relations must be configurable with default parameters; this is fairly easy to do with a template Relation to spawn the default scope from.
+
+RESPONSE REQUIREMENTS
+
+tbd

data/ansr_blacklight/ansr_blacklight.gemspec

@@ -0,0 +1,28 @@
+require File.join(File.dirname(__FILE__), '../lib/ansr/version')
+version = Ansr.version
+Gem::Specification.new do |spec|
+  spec.name = 'ansr_blacklight'
+  spec.version = version
+  spec.platform = Gem::Platform::RUBY
+  spec.authors = ["Benjamin Armintor"]
+  spec.email = ["armintor@gmail.com"]
+  spec.summary = 'ActiveRecord-style models and relations for Blacklight'
+  spec.description = 'Wrapping the Blacklight/RSolr in Rails-like models and relations'
+  spec.homepage = 'https://github.com/barmintor/ansr/tree/master/ansr_blacklight'
+  spec.files         = `git ls-files`.split("\n")
+  spec.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  spec.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'ansr', version
+  spec.add_dependency 'json-ld'
+  spec.add_dependency 'rest-client'
+  spec.add_dependency 'loggable'
+  spec.add_dependency "rails",     ">= 3.2.6", "< 5"
+#  spec.add_dependency 'blacklight', '>=5.1.0'
+  spec.add_dependency 'sass-rails'
+  spec.add_development_dependency("rake")
+  spec.add_development_dependency("bundler", ">= 1.0.14")
+  spec.add_development_dependency "rspec-rails"
+  spec.add_development_dependency("yard")
+end

data/ansr_blacklight/lib/ansr_blacklight.rb

@@ -0,0 +1,57 @@
+require 'ansr'
+require 'rsolr'
+module Ansr::Blacklight
+  extend ActiveSupport::Autoload
+  autoload :SolrProjectionMethods, 'ansr_blacklight/relation/solr_projection_methods'
+  require 'ansr_blacklight/solr'
+  require 'ansr_blacklight/request_builders'
+  require 'ansr_blacklight/arel'
+  require 'ansr_blacklight/connection_adapters/no_sql_adapter'
+  require 'ansr_blacklight/relation'
+  require 'ansr_blacklight/model/querying'
+  require 'ansr_blacklight/base'
+
+    def self.solr_file
+    "#{::Rails.root.to_s}/config/solr.yml"
+  end
+  
+  def self.solr
+    @solr ||=  RSolr.connect(Ansr::Blacklight.solr_config)
+  end
+
+  def self.solr_config
+    @solr_config ||= begin
+        raise "The #{::Rails.env} environment settings were not found in the solr.yml config" unless solr_yml[::Rails.env]
+        solr_yml[::Rails.env].symbolize_keys
+      end
+  end
+
+  def self.solr_yml
+    require 'erb'
+    require 'yaml'
+
+    return @solr_yml if @solr_yml
+    unless File.exists?(solr_file)
+      raise "You are missing a solr configuration file: #{solr_file}. Have you run \"rails generate blacklight:install\"?"  
+    end
+
+    begin
+      @solr_erb = ERB.new(IO.read(solr_file)).result(binding)
+    rescue Exception => e
+      raise("solr.yml was found, but could not be parsed with ERB. \n#{$!.inspect}")
+    end
+
+    begin
+      @solr_yml = YAML::load(@solr_erb)
+    rescue StandardError => e
+      raise("solr.yml was found, but could not be parsed.\n")
+    end
+
+    if @solr_yml.nil? || !@solr_yml.is_a?(Hash)
+      raise("solr.yml was found, but was blank or malformed.\n")
+    end
+
+    return @solr_yml
+  end
+
+end

data/ansr_blacklight/lib/ansr_blacklight/arel.rb

@@ -0,0 +1,6 @@
+module Ansr::Blacklight
+  module Arel
+    require 'ansr_blacklight/arel/visitors'
+    require 'ansr_blacklight/arel/big_table'
+  end
+end

data/ansr_blacklight/lib/ansr_blacklight/arel/big_table.rb

@@ -0,0 +1,57 @@
+module Ansr::Blacklight::Arel
+  class BigTable < Ansr::Arel::BigTable
+    attr_accessor :name
+
+    def initialize(klass, engine=nil, config=nil)
+      super(klass, engine)
+      @name = 'select'
+      self.config(config)
+    end
+    
+    delegate :index_fields, to: :config
+    delegate :show_fields, to: :config
+    delegate :sort_fields, to: :config
+
+    def filterable
+      config.facet_fields.keys
+    end
+
+    alias_method :facets, :filterable
+
+    def filterable?(field)
+      filterable.include? field
+    end
+
+    def constrainable
+      index_fields.keys
+    end
+
+    def constrainable?(field)
+      index_fields.include?(field)
+    end
+
+    def selectable
+      show_fields.keys + index_fields.keys
+    end
+
+    def selectable?(field)
+      show_fields.include? field
+    end
+
+    def fields
+      (constrainable + selectable + filterable).uniq
+    end
+
+    def sortable
+      sort_fields.keys
+    end
+
+    def sortable?(field)
+      sort_fields.include? field
+    end
+
+    def primary_key
+      @primary_key ||= ::Arel::Attribute.new(self, config.document_unique_id_param.to_s)
+    end
+  end
+end

data/ansr_blacklight/lib/ansr_blacklight/arel/visitors.rb

@@ -0,0 +1,6 @@
+module Ansr::Blacklight::Arel
+  module Visitors
+    require 'ansr_blacklight/arel/visitors/query_builder'
+    require 'ansr_blacklight/arel/visitors/to_no_sql'
+  end
+end

data/ansr_blacklight/lib/ansr_blacklight/arel/visitors/query_builder.rb

@@ -0,0 +1,217 @@
+module Ansr::Blacklight::Arel::Visitors
+  class QueryBuilder < Ansr::Arel::Visitors::QueryBuilder
+  	include Ansr::Blacklight::RequestBuilders
+    attr_reader :solr_request
+    
+    def initialize(table)
+      super(table)
+      @solr_request = Ansr::Blacklight::Solr::Request.new
+      table.configure_fields.each do |k,v|
+        unless v[:select].blank?
+          v[:select].each do |sk, sv|
+            key = "f.#{k}.#{sk}".to_sym
+            @solr_request[key] = sv
+          end
+        end
+      end
+    end
+
+    public
+    def query_opts
+    	solr_request
+    end
+
+    # determines whether multiple values should accumulate or overwrite in merges
+    def multiple?(field_key)
+      true
+    end
+
+    def visit_String o, a
+      case a
+      when Ansr::Arel::Visitors::From
+        query_opts.path = o
+      when Ansr::Arel::Visitors::Filter
+        filter_field(o.to_sym)
+      when Ansr::Arel::Visitors::Order
+        order(o)
+      else
+        raise "visited String \"#{o}\" with #{a.to_s}"
+      end
+    end
+
+
+    def visit_Arel_Nodes_TableAlias(object, attribute)
+      solr_request[:qt] = object.name.to_s
+      opts = {qt: object.name.to_s}
+      if (cf = table[object.name]).is_a? Ansr::Arel::ConfiguredField
+        opts.merge!(cf.config.fetch(:query,{}))
+      end
+      solr_request.merge!(opts)
+      visit object.relation, attribute
+    end
+
+    def visit_Ansr_Arel_Nodes_ProjectionTraits(object, attribute)
+      solr_request[:wt] = object.wt if object.wt
+      solr_request[:defType] = object.defType if object.defType
+      visit(object.expr, attribute)
+    end
+
+    def visit_Arel_SqlLiteral(n, attribute)
+      select_val = n.to_s.split(" AS ")
+      if Ansr::Arel::Visitors::Filter === attribute
+        solr_request.append_facet_fields(select_val[0].to_sym)
+      else
+        field(select_val[0].to_sym)
+        if select_val[1]
+          query_opts.aliases ||= {}
+          query_opts.aliases[select_val[0]] = select_val[1]
+        end
+      end
+    end
+
+    def from(value)
+      if value.respond_to? :name
+        solr_request.path = value.name
+      else
+        solr_request.path = value.to_s
+      end
+      self.table=value if (value.is_a? Ansr::Arel::BigTable)
+    end
+
+    def field(field_name)
+      return unless field_name
+      old = query_opts[:fields] ? Array(query_opts[:fields]) : []
+      field_names = (old + Array(field_name)).uniq
+      if field_names[0]
+        query_opts[:fields] = field_names[1] ? field_names : field_names[0]
+      end
+    end
+
+    def filter_field(field_name)
+      return unless field_name
+      old = solr_request[:"facet.field"] ? Array(solr_request[:"facet.field"]) : []
+      fields = Array(field_name).delete_if {|x| old.include? x}
+      solr_request.append_facet_fields(fields)
+    end
+
+    def visit_Arel_Nodes_Equality(object, attribute)
+      field_key = (object.left.respond_to? :expr) ? field_key_from_node(object.left.expr) : field_key_from_node(object.left)
+      opts = {}
+      opts.merge!(local_field_params(field_key))
+      opts.merge!(object.left.config.fetch(:local,{})) if object.left.respond_to? :config
+      if Ansr::Arel::Visitors::Filter === attribute or Ansr::Arel::Nodes::Filter === object.left
+        add_filter_fq_to_solr(solr_request, f: {field_key => object.right}, opts: opts)
+      else
+        # check the table for configured fields
+        add_query_to_solr(field_key, object.right, opts)
+      end
+    end
+
+    def visit_Arel_Nodes_NotEqual(object, attribute)
+    end
+
+    def visit_Arel_Nodes_Or(object, attribute)
+    end
+
+    def visit_Arel_Nodes_Grouping(object, attribute)
+      visit object.expr, attribute
+    end
+
+    def visit_Arel_Nodes_Group(object, attribute)
+      solr_request[:group] = object.expr.to_s
+    end
+
+    def visit_Ansr_Arel_Nodes_Facet(object, attribute)
+      name = object.expr
+      name = name.name if name.respond_to? :name
+      default = false
+      if name == ::Arel.star
+        prefix = "facet."
+        default = true
+      else
+        filter_field(name.to_sym) unless default
+        solr_request.append_facet_fields(name.to_sym) unless default
+        prefix = "f.#{name}.facet."
+      end
+      # there's got to be a helper for this
+      if object.pivot
+        solr_request.append_facet_pivot with_ex_local_param(object.ex, object.pivot.join(","))
+      elsif object.query
+        solr_request.append_facet_query object.query.map { |k, x| with_ex_local_param(object.ex, x[:fq]) }
+      else
+        object.opts.each do |att, value|
+          solr_request["#{prefix}#{att.to_s}".to_sym] = value.to_s unless att == :ex
+        end
+        solr_request.append_facet_fields with_ex_local_param(object.ex, name.to_sym) unless default
+      end
+    end
+
+    def visit_Ansr_Arel_Nodes_Spellcheck(object, attribute)
+      unless object.expr == false
+        solr_request[:spellcheck] = object.expr.to_s
+      end
+      object.opts.each do |att, val|
+        solr_request["spellcheck.#{att.to_s}".to_sym] = val if att != :select
+      end
+    end
+
+    def visit_Ansr_Arel_Nodes_Highlight(object, attribute)
+      unless object.expr == false or object.expr == true
+        solr_request[:hl] = object.expr.to_s
+      end
+      object.opts.each do |att, val|
+        solr_request["hl.#{att.to_s}".to_sym] = val if att != :select
+      end
+    end
+
+    def order(*arel_nodes)
+      direction = nil
+      nodes = []
+      arel_nodes.inject(nodes) do |c, n|
+        if ::Arel::Nodes::Ordering === n
+          c << n
+        elsif n.is_a? String
+          _ns = n.split(',')
+          _ns.each do |_n| 
+            _p = _n.split(/\s+/)
+            if (_p[1])
+              _p[1] = _p[1].downcase.to_sym
+            else
+              _p[1] = :asc
+            end
+            c << table[_p[0].to_sym].send(_p[1])
+          end
+        end
+        c
+      end
+      nodes.each do |node|
+        if ::Arel::Nodes::Ordering === node
+          if solr_request[:sort_by]
+            solr_request[:sort_by] = Array[solr_request[:sort_by]] << node.expr.name
+          else
+            solr_request[:sort_by] = node.expr.name
+          end
+          direction = :asc if (::Arel::Nodes::Ascending === node and direction)
+          direction = :desc if (::Arel::Nodes::Descending === node)
+        end
+      end
+      solr_request[:sort_order] = direction if direction
+    end
+
+    def visit_Arel_Nodes_Limit(object, attribute)
+      value = object.expr
+      if value and (value = value.to_i)
+        raise "Page size cannot be > 500 (#{value}" if value > 500
+        solr_request[:rows] = value.to_s
+      end
+    end
+
+    def visit_Arel_Nodes_Offset(object, attribute)
+      value = object.expr
+      if value
+        solr_request[:start] = value.to_s
+      end
+    end        
+
+  end    
+end