solr_ead 0.4.5 → 0.5.0

checksums.yaml

@@ -0,0 +1,15 @@
+---
+!binary "U0hBMQ==":
+  metadata.gz: !binary |-
+    YzBlY2QzY2Y2MzdlNDhjNDg3NjRjOWJkOWNkNWUyNzZkM2M2OWYwMw==
+  data.tar.gz: !binary |-
+    MWYzMjQ0YTc1NDRiZmNmZTg2MTBlZWE1YmE0YzA1OTJhMzM5Yzg2OA==
+SHA512:
+  metadata.gz: !binary |-
+    Yzk1ZWI5NTE4ZWE4ZDUzNGU5ZGQ4MWFjOTQxNDViZjc3OWY1MThmNjM5NmQ0
+    NTY4YjQ0YzYxYmNlOTlmMjhkNDE3YjczY2ZlMmNhN2E1NGFiYzQxMGEzMjA1
+    MWU0OTc1ZDJlNmI0ODk0MTVmYzNhMjJiNjg2MGFkMTI4NjJiN2E=
+  data.tar.gz: !binary |-
+    Njg0MTY0Y2Q0YTQxMzViZmUwNTY3MzkwNjEwMTNlMjJlODZlYjE1ZWVlZWMy
+    ODZlNzMzZDM4YjhiZjg3NWIwNWY0Zjk2OTc4NGZmNzQ4OGM0MjUwNjI3ODZm
+    YWQxZDc3ZjQ4ZGVjM2ZhZWRkMmVjNjEwNjczM2Q5ODIxYmEzOTY=

data/.gitignore

@@ -16,3 +16,5 @@ test/version_tmp
 tmp
 .DS_Store
 bin
+Gemfile.lock
+jetty

data/.travis.yml

@@ -0,0 +1,7 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+branches:
+  only:
+    - master

data/README.md

@@ -1,5 +1,8 @@
 # SolrEad
 
+[![Build Status](https://travis-ci.org/awead/solr_ead.png?branch=master)](https://travis-ci.org/awead/solr_ead)
+[![Gem Version](https://badge.fury.io/rb/solr_ead.png)](http://badge.fury.io/rb/solr_ead)
+
 SolrEad is a gem that indexes your ead documents into Solr.  From there, you can use other
 Solr-based applications to search and display your finding aids.  It originated as some
 code that I used to index ead into Blacklight, but this gem does not require you to use
@@ -47,32 +50,10 @@ You can also do this via the command line:
     > indexer = SolrEad::Indexer.new
     > indexer.create(File.new("path/to/your/ead.xml))
 
-### Usage with Blacklight
-
-This code originated in a Blacklight application and some of its default solr fields
-reflect a Blacklight-style solr implementation.  For example, certain facet fields such as
-subject_topic_facet and title_display will appear in your solr index by default.  If you
-are trying out the gem within a default Blacklight installation, you will need to make
-the following changes to your schema.xml file:
-
-    <dynamicField name="*_id" type="string" indexed="true" stored="true" multiValued="false" />
-    <dynamicField name="*_s"  type="string" indexed="true" stored="true" multiValued="true"  />
-
-Fields ending in "_id" are used for identifying components within finding aids as well as
-the finding aid itself, which may not always be the same as the default "id" field for
-the solr document.  "_s" fields are not multivalued by default in Blacklight, so you'll
-need to edit the existing entry for this dyamic field to reflect the change.
-
-Other than that, your solr configuration should require no futher modifications;
-however,  the only fields that will appear in your search results will be format
-and title.  In order to make this into working solution, you'll need to modify
-both the definitions of documents and components within SolrEad and configure
-Blacklight's own display and facet fields accordingly.
-
 ## Applications
 
 SolrEad is intended to work at the indexing layer of an application, but it can also work
-at the display/presentation layer as well.  You can use the solr fields defined in your OM
+at the display and presentation layer as well.  You can use the solr fields defined in your OM
 terminology for display; however, formatting information such as italics and boldface is
 not preserved from the original EAD xml.
 
@@ -85,9 +66,9 @@ pages created using XSLT, or create the html when the page is requested.  If you
 the latter, you can store the ead xml in a solr field.  To do this, add a new solr field
 under the to_solr method of your OM terminology for the ead document:
 
-    solr_doc.merge!({"xml_display" => self.to_xml})
+    Solrizer.insert_field(solr_doc, "xml", self.to_xml, :displayable)
 
-This will create the solr field "xml_display" containing the complete ead xml. Then you
+This will create the solr field "xml_ssm" containing the complete ead xml. Then you
 will be able to apply any xslt processing you wish.  Other solutions are possible using
 xml from the document as well as the component, depending on the needs of your
 application.
@@ -104,6 +85,15 @@ following content:
 
     class CustomDocument < SolrEad::Document
 
+      # Use the existing terminology
+      use_terminology SolrEad::Document
+
+      # And extend it with terms of your own
+      extend_terminology do |t|
+        ...
+      end
+
+      # Or, just define your own from scratch
       set_terminology do |t|
         t.root(:path="ead", :index_as = [:not_searchable])
         t.eadid
@@ -167,31 +157,50 @@ Then, include your module in your own custom document and call the method during
 
       include MyEadBehaviors
 
-      # terminology goes here...
+      use_terminology SolrEad::Document
 
       def to_solr(solr_doc = Hash.new)
         super(solr_doc)
-        solr_doc.merge!({"solr_field" => special_process(self.field_name)})
+        Solrizer.insert_field(solr_doc, "field", special_process(self.field), :displayable)
       end
 
     end
 
-Your solr document will now include the field "solr_field" that has taken the term
-"field_name" and processed it with the special_process method.
+Your solr document will now include the field "field_ssm" that has taken the term
+"field" and processed it with the special_process method.
 
 ### Solr schema configurations
 
-SolrEad is designed to work with the solr jetty application that comes with Blacklight.
-However, this doesn't prevent you from using your own solr application.  You can alter the
-way SolrEad creates its solr fields by creating your own mapper.  See the ead_mapper.rb
-file for more info and the solrizer gem for more information on configuring how SolrEad
-creates solr fields.
-
-By default, SolrEad will display series and subseries component documents.  You may,
-however, want to surpress this from search results.  To do this, add the following line to
-your solrconfig.xml file, under the "search" request handler:
-
-    <lst name="appends"><str name="fq">-component_children_b:[TRUE TO *]</str></lst>
+SolrEad uses Solrizer's default field descriptors to create the names of solr fields.  A complete
+listing of these fields is found under 
+[Solrizer::DefaultDescriptors](https://github.com/projecthydra/solrizer/blob/master/lib/solrizer/default_descriptors.rb)
+but the options that are used here are specifically:
+
+    :displayable
+    :stored_sortable
+    :type => :integer
+    :type => :boolean
+    :facetable
+    :sortable, :type => :integer
+    :searchable
+
+These result in a specific set of dynamic field names that will need to be present in your schema.xml file in
+solr.  In order to have these fields index correctly, include the following in your schema.xml file:
+
+    <dynamicField name="*_teim"  type="text_en"   stored="false" indexed="true"  multiValued="true"  />
+    <dynamicField name="*_si"    type="string"    stored="false" indexed="true"  multiValued="false" />
+    <dynamicField name="*_sim"   type="string"    stored="false" indexed="true"  multiValued="true"  />
+    <dynamicField name="*_ssm"   type="string"    stored="true"  indexed="false" multiValued="true"  />
+    <dynamicField name="*_ssi"   type="string"    stored="true"  indexed="true"  multiValued="false" />
+    <dynamicField name="*_ssim"  type="string"    stored="true"  indexed="true"  multiValued="true"  />
+    <dynamicField name="*_dtsi"  type="date"      stored="true"  indexed="true"  multiValued="false" />
+    <dynamicField name="*_dtsim" type="date"      stored="true"  indexed="true"  multiValued="true"  />
+    <dynamicField name="*_bsi"   type="boolean"   stored="true"  indexed="true"  multiValued="false" />
+    <dynamicField name="*_isim"  type="int"       stored="true"  indexed="true"  multiValued="true"  />
+    <dynamicField name="*_ii"    type="int"       stored="false" indexed="true"  multiValued="false" />
+
+Note that the type "text_en" is dependent on your particular solr application, but the others should be
+included in the default installation.
 
 ## Issues
 
@@ -199,13 +208,22 @@ your solrconfig.xml file, under the "search" request handler:
 
 solr_ead uses the <eadid> node to create unique ids for documents.  Consequently, if you're using
 a rails app, this id will be a part of the url.  If your eadid has .xml or some other combination
-of characters preceeded by a period, this will cause Rails to interpret these characters as a 
+of characters preceded by a period, this will cause Rails to interpret these characters as a 
 format, which you don't want.  You may need to edit your eadid nodes if this is the case.
 
 ## Contributing
 
-If you have questions or have specific needs, let me know. If you have other ideas or
-solutions, please contribute code!
+### Testing with Jettywrapper
+
+SolrEad uses jettywrapper to download a solr application for testing.  To get setup for developing
+additional features for SolrEad:
+
+    git clone https://github.com/awead/solr_ead
+    bundle install
+    rake ci
+
+This will download jetty, start it up and run the spec tests. If you have questions or have specific needs, let me know. 
+If you have other ideas or solutions, please contribute code!
 
 1. Fork SolrEad
 2. Create your feature branch (`git checkout -b my-new-feature`)

data/Rakefile

@@ -1,8 +1,9 @@
 #!/usr/bin/env rake
 require "bundler/gem_tasks"
 require "rspec/core/rake_task"
+require "jettywrapper"
 Bundler::GemHelper.install_tasks
 # load rake tasks in lib/tasks
 Dir.glob('lib/tasks/*.rake').each { |r| import r }
 RSpec::Core::RakeTask.new(:spec)
-task :default => :spec
+task :default => :ci

data/config/jetty.yml

@@ -0,0 +1,9 @@
+default:
+  jetty_port: <%= ENV['TEST_JETTY_PORT'] || 8983 %>
+  startup_wait: 90
+  java_opts:
+    - "-Xmx256m" 
+    - "-XX:MaxPermSize=128m"
+    - "-XX:+CMSPermGenSweepingEnabled"
+    - "-XX:+CMSClassUnloadingEnabled"
+    - "-XX:PermSize=64M"

data/config/solr.yml

@@ -1,6 +1,6 @@
 development:
-  url: http://127.0.0.1:8985/solr/blacklight-dev
+  url: http://127.0.0.1:8983/solr/hydra-dev
 test: &test
-  url: http://127.0.0.1:8985/solr/blacklight-dev
+  url: http://127.0.0.1:8983/solr/hydra-dev
 production:
-  url: http://127.0.0.1:8985/solr/blacklight-dev
+  url: http://127.0.0.1:8983/solr/hydra-dev

data/lib/solr_ead/behaviors.rb

@@ -58,14 +58,14 @@ module SolrEad::Behaviors
   # These fields are used so that we may reconstruct placement of a single component
   # within the hierarchy of the original ead.
   def additional_component_fields(node, addl_fields = Hash.new)
-    addl_fields["id"]                         = [node.xpath("//eadid").text, node.attr("id")].join
-    addl_fields["ead_id"]                     = node.xpath("//eadid").text
-    addl_fields["parent_id"]                  = node.parent.attr("id") unless node.parent.attr("id").nil?
-    addl_fields["parent_id_s"]                = parent_id_list(node)
-    addl_fields["parent_unittitles_display"]  = parent_unittitle_list(node)
-    addl_fields["component_level_i"]          = parent_id_list(node).length + 1
-    addl_fields["component_children_b"]       = component_children?(node)
-    addl_fields["collection_facet"]           = node.xpath("//archdesc/did/unittitle").text
+    addl_fields["id"]                                                        = [node.xpath("//eadid").text, node.attr("id")].join
+    addl_fields[Solrizer.solr_name("ead", :stored_sortable)]                 = node.xpath("//eadid").text
+    addl_fields[Solrizer.solr_name("parent", :stored_sortable)]              = node.parent.attr("id") unless node.parent.attr("id").nil?
+    addl_fields[Solrizer.solr_name("parent", :displayable)]                  = parent_id_list(node)
+    addl_fields[Solrizer.solr_name("parent_unittitles", :displayable)]       = parent_unittitle_list(node)
+    addl_fields[Solrizer.solr_name("component_level", :type => :integer)]    = parent_id_list(node).length + 1
+    addl_fields[Solrizer.solr_name("component_children", :type => :boolean)] = component_children?(node)
+    addl_fields[Solrizer.solr_name("collection", :facetable)]                = node.xpath("//archdesc/did/unittitle").text
     return addl_fields
   end
 

data/lib/solr_ead/component.rb

@@ -1,7 +1,7 @@
 class SolrEad::Component
 
   include OM::XML::Document
-  include Solrizer::XML::TerminologyBasedSolrizer
+  include OM::XML::TerminologyBasedSolrizer
 
   # Define each term in your ead that you want put into the solr document
   set_terminology do |t|
@@ -84,9 +84,21 @@ class SolrEad::Component
 
   def to_solr(solr_doc = Hash.new)
     super(solr_doc)
-    solr_doc.merge!({"format" => "Archival Item"})
-    solr_doc["parent_unittitles_display"].length > 0 ? solr_doc.merge!({"heading_display" => [ solr_doc["parent_unittitles_display"], self.title.first].join(" >> ")  }) : solr_doc.merge!({"heading_display" => self.title.first  })
-    solr_doc.merge!({"ref_id" => self.ref.first.strip})
+    Solrizer.insert_field(solr_doc, "format", "Archival Item", :facetable)
+    heading = get_heading solr_doc[Solrizer.solr_name("parent_unittitles", :displayable)]
+    Solrizer.insert_field(solr_doc, "heading", heading, :displayable) unless heading.nil?
+    Solrizer.insert_field(solr_doc, "ref", self.ref.first.strip, :stored_sortable)
+  end
+
+  protected
+
+  def get_heading parent_titles = Array.new
+    return nil if parent_titles.nil?
+    if parent_titles.length > 0
+      [parent_titles, self.title.first].join(" >> ")
+    else
+      self.title.first
+    end
   end
 
 end

data/lib/solr_ead/document.rb

@@ -1,7 +1,7 @@
 class SolrEad::Document
 
   include OM::XML::Document
-  include Solrizer::XML::TerminologyBasedSolrizer
+  include OM::XML::TerminologyBasedSolrizer
   include SolrEad::OmBehaviors
 
   # Define each term in your ead that you want put into the solr document
@@ -27,9 +27,9 @@ class SolrEad::Document
     t.title_filing(:path=>"titleproper", :attributes=>{ :type => "filing" }, :index_as=>[:sortable])
     t.title_num(:path=>"archdesc/did/unitid")
     t.extent(:path=>"archdesc/did/physdesc/extent")
-    t.unitdate(:path=>"archdesc/did/unitdate[not(@type)]", :index_as=>[:unstemmed])
-    t.unitdate_bulk(:path=>"archdesc/did/unitdate[@type='bulk']", :index_as=>[:unstemmed])
-    t.unitdate_inclusive(:path=>"archdesc/did/unitdate[@type='inclusive']", :index_as=>[:unstemmed])
+    t.unitdate(:path=>"archdesc/did/unitdate[not(@type)]", :index_as=>[:searchable])
+    t.unitdate_bulk(:path=>"archdesc/did/unitdate[@type='bulk']", :index_as=>[:searchable])
+    t.unitdate_inclusive(:path=>"archdesc/did/unitdate[@type='inclusive']", :index_as=>[:searchable])
     t.language(:path=>"archdesc/did/langmaterial", :index_as=>[:displayable])
     t.langcode(:path=>"did/langmaterial/language/@langcode")
     t.abstract(:path=>"archdesc/did/abstract", :index_as=>[:searchable])
@@ -78,9 +78,12 @@ class SolrEad::Document
   def to_solr(solr_doc = Hash.new)
     super(solr_doc)
     solr_doc.merge!({"id"              => self.eadid.first.strip})
-    solr_doc.merge!({"ead_id"          => self.eadid.first.strip})
-    solr_doc.merge!({"format"          => "Archival Collection"})
-    solr_doc.merge!({"heading_display" => ("Guide to the " + self.title.first + " (" + self.title_num.first + ")")}) unless self.title_num.empty?
+    Solrizer.insert_field(solr_doc, "format", "Archival Collection", :facetable)
+    Solrizer.insert_field(solr_doc, "ead", self.eadid.first.strip, :stored_sortable)
+    unless self.title_num.empty?
+      heading = "Guide to the " + self.title.first + " (" + self.title_num.first + ")"
+      Solrizer.insert_field(solr_doc, "heading", heading, :displayable)
+    end
     return solr_doc
   end
 

data/lib/solr_ead/indexer.rb

@@ -39,28 +39,14 @@ class Indexer
   attr_accessor :solr, :options
 
   # Creates a new instance of SolrEad::Indexer and connects to your solr server
-  # using the url supplied in your config/solr.yml file.
-  def initialize(opts={})
-    Solrizer.default_field_mapper = EadMapper.new
-    if ENV['SOLR_URL']
-      url = ENV['SOLR_URL']
-    elsif defined?(Rails.root)
-      url = YAML.load(ERB.new(File.read(File.join(Rails.root,"config","solr.yml"))).result)[Rails.env]['url']
-    elsif ENV['RAILS_ENV']
-      url = YAML.load(ERB.new(File.read("config/solr.yml")).result)[ENV['RAILS_ENV']]['url']
-    else
-      url = YAML.load(ERB.new(File.read("config/solr.yml")).result)['development']['url']
-    end
-    self.solr = RSolr.connect :url => url
+  def initialize opts={}
+    self.solr = solr_connection
     self.options = opts
   end
   
-  # Indexes your ead and additional component documents with the supplied file, then
-  # commits the results to your solr server.
-  def create(file)
-    doc = om_document(File.new(file))
-    solr_doc = doc.to_solr
-    solr.add solr_doc
+  # Indexes ead xml and commits the results to your solr server.
+  def create file
+    solr.add om_document(File.new(file)).to_solr
     add_components(file) unless options[:simple]
     solr.commit
   end
@@ -68,10 +54,9 @@ class Indexer
   # Updates your ead from a given file by first deleting the existing ead document and
   # any component documents, then creating a new index from the supplied file.
   # This method will also commit the results to your solr server when complete.
-  def update(file)
-    doc = om_document(File.new(file))
-    solr_doc = doc.to_solr
-    solr.delete_by_query( 'ead_id:"' + solr_doc["id"] + '"' )
+  def update file
+    solr_doc = om_document(File.new(file)).to_solr
+    delete solr_doc["id"]
     solr.add solr_doc
     add_components(file) unless options[:simple]
     solr.commit
@@ -79,19 +64,19 @@ class Indexer
 
   # Deletes the ead document and any component documents from your solr index and
   # commits the results.
-  def delete(id)
-    solr.delete_by_query( 'ead_id:"' + id + '"')
+  def delete id
+    solr.delete_by_query( Solrizer.solr_name("ead", :stored_sortable)+':"' + id + '"')
     solr.commit
   end
 
-  protected
+  private
 
   # Returns an OM document from a given file.
   #
   # Determines if you have specified a custom definition for your ead document.
   # If you've defined a class CustomDocument, and have passed it as an option
   # to your indexer, then SolrEad will use that class instead of SolrEad::Document.
-  def om_document(file)
+  def om_document file
     options[:document] ? options[:document].from_xml(File.new(file)) : SolrEad::Document.from_xml(File.new(file))
   end
 
@@ -100,7 +85,7 @@ class Indexer
   # Determines if you have specified a custom definition for your ead component.
   # If you've defined a class CustomComponent, and have passed it as an option
   # to your indexer, then SolrEad will use that class instead of SolrEad::Component.
-  def om_component_from_node(node)
+  def om_component_from_node node
     options[:component] ? options[:component].from_xml(prep(node)) : SolrEad::Component.from_xml(prep(node))
   end
 
@@ -117,15 +102,34 @@ class Indexer
   # A sorting field *sort_i* is added to the document using the index values from the array
   # of <c> nodes.  This allows us to preserve the order of <c> nodes as they appear
   # in the original ead document.
-  def add_components(file)
-    counter = 1
+  def add_components file, counter = 1
     components(file).each do |node|
       solr_doc = om_component_from_node(node).to_solr(additional_component_fields(node))
-      solr_doc.merge!({"sort_i" => counter.to_s})
+      solr_doc.merge!({Solrizer.solr_name("sort", :sortable, :type => :integer) => counter.to_s})
       solr.add solr_doc
       counter = counter + 1
     end
   end
 
+  # Returns a connection to solr using Rsolr
+  def solr_connection
+    if ENV['SOLR_URL']
+      RSolr.connect :url => ENV['SOLR_URL']
+    else
+      RSolr.connect :url => solr_url
+    end
+  end
+
+  # Determines the url to our solr service by consulting yaml files
+  def solr_url
+    if defined?(Rails.root)
+      YAML.load(ERB.new(File.read(File.join(Rails.root,"config","solr.yml"))).result)[Rails.env]['url']
+    elsif ENV['RAILS_ENV']
+      YAML.load(ERB.new(File.read("config/solr.yml")).result)[ENV['RAILS_ENV']]['url']
+    else
+      YAML.load(ERB.new(File.read("config/solr.yml")).result)['development']['url']
+    end
+  end
+
 end
 end

data/lib/solr_ead/version.rb

@@ -1,3 +1,3 @@
 module SolrEad
-  VERSION = "0.4.5"
+  VERSION = "0.5.0"
 end

data/lib/solr_ead.rb

@@ -1,19 +1,21 @@
-require "nokogiri"
-require "solrizer"
-require "om"
-require "rsolr"
+require 'nokogiri'
+require 'solrizer'
+require 'om'
+require 'rsolr'
+require 'active_support'
 
 module SolrEad
+  extend ActiveSupport::Autoload
+
+  autoload :Behaviors
+  autoload :OmBehaviors
+  autoload :Indexer
+  autoload :Document
+  autoload :Component
+  autoload :Railtie if defined?(Rails)
+
+
   def self.version
     SolrEad::VERSION
   end
-end
-
-require "ead_mapper"
-require "solr_ead/behaviors"
-require "solr_ead/om_behaviors"
-require "solr_ead/indexer"
-require "solr_ead/document"
-require "solr_ead/component"
-
-require 'solr_ead/railtie' if defined?(Rails)
+end

data/lib/tasks/solr_ead_dev.rake

@@ -0,0 +1,22 @@
+desc "Travis continuous integration task"
+task :ci do
+  Rake::Task["jetty:stop"].invoke
+  Rake::Task["jetty:clean"].invoke
+  Rake::Task["jetty:start"].invoke
+  Rake::Task["spec"].invoke
+end
+
+namespace :jetty do
+
+  desc "Unpack a clean version of solr-jetty"
+  task :clean do
+    Jettywrapper.url = "https://github.com/awead/solr-jetty/archive/v1.zip"
+    Jettywrapper.clean
+  end
+
+  desc "Configure solr-jetty instance"
+  task :config do
+    `cp solr/schema.xml jetty/solr/blacklight-dev-core/conf/schema.xml`
+  end
+
+end