traject-marc4j_reader 1.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 781377f3f2adb35b8345224b0d04f424355b899f
+  data.tar.gz: 5941faff6ae8c86282d1acad116c2079f6840d65
+SHA512:
+  metadata.gz: 2deabe30e44a00b5f232660bf36db22bddeb0f0577ba75bfaa662d5ad98aeaca36493b2d69104e7ab3b6362b1f1ac2daf76a6c5cccd31cb0e4bae9a362901468
+  data.tar.gz: b9259af133a130101c0b3e6a2a47bd4eac2a6dac790c79ff9fea3062cf1ab8222873761684c641d2a7145dab20ed8a6acf263e054be925575d17c15bebe39815

data/.gitignore

@@ -0,0 +1,22 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.travis.yml

@@ -0,0 +1,3 @@
+language: ruby
+rvm:
+  - 1.9.3

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in traject_marc4j_reader.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Bill Dueber
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,100 @@
+# Traject::Marc4JReader
+
+**Note**: `Traject::Marc4JReader` is for JRuby only.
+
+`Traject::Marc4JReader` is a reader for the [traject](https://github.com/traject-project/traject) ETL system
+that allows the use of [marc4j](https://github.com/marc4j/marc4j) as a reader when dealing with MARC
+binary or MARC-XML files. It is of no use outside of `traject` run under JRuby.
+
+It leverages [marc-marc4j](https://github.com/billdueber/ruby-marc-marc4j), which is a paper-thin wrapper around
+the Marc4J `.jar` that is shipped with it.
+
+**The output of the reader is a vanilla ruby-marc object**. You can hang onto the
+original marc4j java object with the `marc4j_reader.keep_marc4j` setting.
+
+
+## Why use this?
+
+
+The biggest reason would be for faster MARC/MARC-XML parsing and generation than the vanilla
+[marc](https://github.com/ruby-marc/ruby-marc) gem can provide, or if you need to do something wacky with the marc4j
+internal structure (such as feed it to legacy java code you have lying around).
+
+In general, the marc4j library will parse marc21 (binary) and MARC-XML roughly twice
+as fast as the pure-ruby library. While MARC parsing tends to not be a huge part
+of the workload in a traject run, you'll almost certainly see performance gains.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'traject-marc4j_reader'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install traject-marc4j_reader
+
+## Traject::Marc4jReader settings
+
+For more about the traject `settings` object, see [the traject settings documentation](https://github.com/traject-project/traject/blob/master/doc/settings.md)
+
+
+Note that the standard Marc4JReader always converts to UTF8,
+so output will always reflect that conversion.
+
+* `marc4j.jar_dir`:   Path to a directory containing Marc4J jar file to use. All .jar's in dir will
+                      be loaded. If unset, uses marc4j.jar bundled with traject.
+
+* `marc4j_reader.permissive`: Used by Marc4JReader only when marc.source_type is 'binary', boolean, argument to the underlying MarcPermissiveStreamReader. Default true.
+
+* `marc4j_reader.source_encoding`: Used by Marc4JReader only when marc.source_type is 'binary', encoding strings accepted
+  by marc4j MarcPermissiveStreamReader. Default "BESTGUESS", also "UTF-8", "MARC"
+
+* `marc4j_reader.keep_marc4j`: After translating the marc4j record into a normal ruby-marc object,
+provides access to the former via `record#original_marc4j`.
+
+
+## Sample use
+
+A simple example that reads in via marc4j and outputs to the newline-delimited-json writer.
+
+Use would be:
+
+```bash
+traject -c id_title.rb my_marc_file.mrc
+```
+
+```ruby
+# File id_title.rb
+
+require 'traject'
+require 'traject/marc4j_reader'
+require 'traject/json_writer'
+
+require 'traject/macros/marc21_semantics'
+extend  Traject::Macros::Marc21Semantics
+
+settings do
+  provide "reader_class_name", "Traject::Marc4JReader"
+  provide "marc4j_reader.keep_marc4j", true
+  provide "writer_class_name", "Traject::JsonWriter"
+  provide "output_file", "ids_and_titles.ndj"
+end
+
+to_field "id", extract_marc("001", :first => true)
+to_field "title", extract_marc_filing_version('245abdefghknp', :include_original => true)
+
+```
+
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/traject_marc4j_reader/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+task :default => [:test]
+
+Rake::TestTask.new do |t|
+  t.pattern = 'test/**/*_test.rb'
+  t.libs.push 'test', 'test_support'
+end

data/lib/traject/marc4j_reader/version.rb

@@ -0,0 +1,5 @@
+module Traject
+  class Marc4JReader
+    VERSION = "1.0.0"
+  end
+end

data/lib/traject/marc4j_reader.rb

@@ -0,0 +1,153 @@
+require 'traject'
+require 'marc'
+require 'marc/marc4j'
+
+# `Traject::Marc4JReader` uses the marc4j java package to parse the MARC records
+# into standard ruby-marc MARC::Record objects. This reader may be faster than
+# Traject::MarcReader, especially for XML.
+#
+# Marc4JReader can read MARC ISO 2709 ("binary") or MARCXML. We use the Marc4J MarcPermissiveStreamReader
+# for reading binary, but sometimes in non-permissive mode, according to settings. We use the Marc4j MarcXmlReader
+# for reading xml. The actual code for dealing with Marc4J is in the separate
+# [marc-marc4j gem](https://github.com/billdueber/ruby-marc-marc4j).
+#
+# See also the pure ruby Traject::MarcReader as an alternative, if you need to read
+# marc-in-json, or if you don't need binary Marc8 support, it may in some cases
+# be faster.
+#
+# ## Settings
+#
+# * marc_source.type:     serialization type. default 'binary', also 'xml' (TODO: json/marc-in-json)
+#
+# * marc4j_reader.permissive:   default true, false to turn off permissive reading. Used as
+#                             value to 'permissive' arg of MarcPermissiveStreamReader constructor.
+#                             Only used for 'binary'
+#
+# * marc_source.encoding: Only used for 'binary', otherwise always UTF-8.
+#         String of the values MarcPermissiveStreamReader accepts:
+#         * BESTGUESS  (default: not entirely clear what Marc4J does with this)
+#         * ISO-8859-1 (also accepted: ISO8859_1)
+#         * UTF-8
+#         * MARC-8 (also accepted: MARC8)
+#         Default 'BESTGUESS', but HIGHLY recommend setting
+#         to avoid some Marc4J unpredictability, Marc4J "BESTGUESS" can be unpredictable
+#         in a variety of ways.
+#         (will ALWAYS be transcoded to UTF-8 on the way out. We insist.)
+#
+# * marc4j_reader.jar_dir: Path to a directory containing Marc4J jar file to use. All .jar's in dir will
+#                          be loaded. If unset, uses marc4j.jar bundled with traject.
+#
+# * marc4j_reader.keep_marc4j: Keeps the original marc4j record accessible from
+#   the eventual ruby-marc record via record#original_marc4j. Intended for
+#   those that have legacy java code for which a marc4j object is needed. .
+#
+#
+# ## Example
+#
+# In a configuration file:
+#
+#     require 'traject/marc4j_reader
+#     settings do
+#       provide "reader_class_name", "Traject::Marc4JReader"
+#
+#       #for MarcXML:
+#       # provide "marc_source.type", "xml"
+#
+#       # Or instead for binary:
+#       provide "marc4j_reader.permissive", true
+#       provide "marc_source.encoding", "MARC8"
+#     end
+class Traject::Marc4JReader
+  include Enumerable
+
+  attr_reader :settings, :input_stream
+
+  def initialize(input_stream, settings)
+    @settings     = Traject::Indexer::Settings.new settings
+    @input_stream = input_stream
+
+    if @settings['marc4j_reader.keep_marc4j'] &&
+        ! (MARC::Record.instance_methods.include?(:original_marc4j) &&
+            MARC::Record.instance_methods.include?(:"original_marc4j="))
+      MARC::Record.class_eval('attr_accessor :original_marc4j')
+    end
+
+    # Creating a converter will do the following:
+    #  - nothing, if it detects that the marc4j jar is already loaded
+    #  - load all the .jar files in settings['marc4j_reader.jar_dir'] if set
+    #  - load the marc4j jar file bundled with MARC::MARC4J otherwise
+
+    @converter = MARC::MARC4J.new(:jardir => settings['marc4j_reader.jar_dir'], :logger => logger)
+
+    # Convenience
+    java_import org.marc4j.MarcPermissiveStreamReader
+    java_import org.marc4j.MarcXmlReader
+
+  end
+
+
+  def internal_reader
+    @internal_reader ||= create_marc_reader!
+  end
+
+  def input_type
+    # maybe later add some guessing somehow
+    settings["marc_source.type"]
+  end
+
+  def specified_source_encoding
+    #settings["marc4j_reader.source_encoding"]
+    enc = settings["marc_source.encoding"]
+
+    # one is standard for ruby and we want to support,
+    # the other is used by Marc4J and we have to pass it to Marc4J
+    enc = "ISO8859_1" if enc == "ISO-8859-1"
+
+    # default
+    enc = "BESTGUESS" if enc.nil? || enc.empty?
+
+    return enc
+  end
+
+  def create_marc_reader!
+    case input_type
+    when "binary"
+      permissive = settings["marc4j_reader.permissive"].to_s == "true"
+
+      # #to_inputstream turns our ruby IO into a Java InputStream
+      # third arg means 'convert to UTF-8, yes'
+      MarcPermissiveStreamReader.new(input_stream.to_inputstream, permissive, true, specified_source_encoding)
+    when "xml"
+      MarcXmlReader.new(input_stream.to_inputstream)
+    else
+      raise IllegalArgument.new("Unrecgonized marc_source.type: #{input_type}")
+    end
+  end
+
+  def each
+    while (internal_reader.hasNext)
+      begin
+        marc4j = internal_reader.next
+        rubymarc = @converter.marc4j_to_rubymarc(marc4j)
+        if @settings['marc4j_reader.keep_marc4j']
+          rubymarc.original_marc4j = marc4j
+        end
+      rescue Exception =>e
+        msg = "MARC4JReader: Error reading MARC, fatal, re-raising"
+        if marc4j
+          msg += "\n    001 id: #{marc4j.getControlNumber}"
+        end
+        msg += "\n    #{Traject::Util.exception_to_log_message(e)}"
+        logger.fatal msg
+        raise e
+      end
+
+      yield rubymarc
+    end
+  end
+
+  def logger
+    @logger ||= (settings[:logger] || Yell.new(STDERR, :level => "gt.fatal")) # null logger)
+  end
+
+end

data/test/marc4j_reader_test.rb

@@ -0,0 +1,138 @@
+# Encoding: UTF-8
+
+require 'test_helper'
+
+require 'traject'
+require 'traject/indexer'
+require 'traject/marc4j_reader'
+
+require 'marc'
+
+describe "Marc4JReader" do
+  it "reads Marc binary" do
+    file = File.new(support_file_path("test_data.utf8.mrc"))
+    settings = Traject::Indexer::Settings.new() # binary type is default
+    reader = Traject::Marc4JReader.new(file, settings)
+
+    array = reader.to_a
+
+    assert_equal 30, array.length
+    first = array.first
+
+    assert_kind_of MARC::Record, first
+    assert_equal first['245']['a'].encoding.name, "UTF-8"
+  end
+
+  it "can skip a bad subfield code" do
+    file = File.new(support_file_path("bad_subfield_code.marc"))
+    settings = Traject::Indexer::Settings.new() # binary type is default
+    reader = Traject::Marc4JReader.new(file, settings)
+
+    array = reader.to_a
+
+    assert_equal 1, array.length
+    assert_kind_of MARC::Record, array.first
+    assert_length 2, array.first['260'].subfields
+  end
+
+  it "reads Marc binary in Marc8 encoding" do
+    file = File.new(support_file_path("one-marc8.mrc"))
+    settings = Traject::Indexer::Settings.new("marc_source.encoding" => "MARC8")
+    reader = Traject::Marc4JReader.new(file, settings)
+
+    array = reader.to_a
+
+    assert_length 1, array
+
+
+    assert_kind_of MARC::Record, array.first
+    a245a = array.first['245']['a']
+
+    assert a245a.encoding.name, "UTF-8"
+    assert a245a.valid_encoding?
+    # marc4j converts to denormalized unicode, bah. Although
+    # it's legal, it probably looks weird as a string literal
+    # below, depending on your editor.
+    assert_equal "Por uma outra globalização :", a245a
+
+    # Set leader byte to proper for unicode
+    assert_equal 'a', array.first.leader[9]
+  end
+
+
+  it "reads XML" do
+    file = File.new(support_file_path "test_data.utf8.marc.xml")
+    settings = Traject::Indexer::Settings.new("marc_source.type" => "xml")
+    reader = Traject::Marc4JReader.new(file, settings)
+
+    array = reader.to_a
+
+    assert_equal 30, array.length
+
+    first = array.first
+
+    assert_kind_of MARC::Record, first
+    assert first['245']['a'].encoding.name, "UTF-8"
+    assert_equal "Fikr-i Ayāz /", first['245']['a']
+  end
+
+  it "keeps marc4j object when asked" do
+    file = File.new(support_file_path "test_data.utf8.marc.xml")
+    settings = Traject::Indexer::Settings.new("marc_source.type" => "xml", 'marc4j_reader.keep_marc4j' => true)
+    record = Traject::Marc4JReader.new(file, settings).to_a.first
+    assert_kind_of MARC::Record, record
+    assert_kind_of Java::org.marc4j.marc.impl::RecordImpl, record.original_marc4j
+  end
+
+  it "replaces unicode character reference in Marc8 transcode" do
+    file = File.new(support_file_path "escaped_character_reference.marc8.marc")
+    # due to marc4j idiosyncracies, this test will NOT pass with default source_encoding
+    # of "BESTGUESS", it only works if you explicitly set to MARC8. Doh. 
+    settings = Traject::Indexer::Settings.new("marc_source.encoding" => "MARC8") # binary type is default
+    record = Traject::Marc4JReader.new(file, settings).to_a.first
+
+    assert_equal "Rio de Janeiro escaped replacement char: \uFFFD .", record['260']['a']
+  end
+
+  describe "Marc4J Java Permissive Stream Reader" do 
+    # needed for sanity check when our tests fail to see if Marc4J
+    # is not behaving how we think it should. 
+    it "converts character references" do
+      settings = Traject::Indexer::Settings.new("marc4j_reader.permissive" => true)
+
+      file = File.new(support_file_path "escaped_character_reference.marc8.marc")
+      reader = MarcPermissiveStreamReader.new(file.to_inputstream, true, true, "MARC-8")
+      record = reader.next
+
+      field = record.getVariableField("260")
+      subfield = field.getSubfield('a'.ord)
+      value = subfield.getData
+      
+      assert_equal "Rio de Janeiro escaped replacement char: \uFFFD .", value        
+    end
+  end
+
+  it "replaces bad byte in UTF8 marc" do
+
+    # Note this only works because the marc file DOES correctly
+    # have leader byte 9 set to 'a' for UTF8, otherwise Marc4J can't do it. 
+    file = File.new(support_file_path "bad_utf_byte.utf8.marc")
+
+    # marc4j doesn't do this in permissive mode
+    settings = Traject::Indexer::Settings.new("marc4j_reader.permissive" => false)
+    reader = Traject::Marc4JReader.new(file, settings)
+
+    record = reader.to_a.first
+    
+    value = record['300']['a']
+
+    assert_equal value.encoding.name, "UTF-8"
+    assert value.valid_encoding?, "Has valid encoding"
+    assert_equal "This is a bad byte: '\uFFFD' and another: '\uFFFD'", record['300']['a']
+  end
+
+
+
+
+
+end

data/test/test_helper.rb

@@ -0,0 +1,88 @@
+gem 'minitest' # I feel like this messes with bundler, but only way to get minitest to shut up
+require 'minitest/autorun'
+require 'minitest/spec'
+
+require 'traject'
+require 'marc'
+
+# keeps things from complaining about "yell-1.4.0/lib/yell/adapters/io.rb:66 warning: syswrite for buffered IO"
+# for reasons I don't entirely understand, involving yell using syswrite and tests sometimes
+# using $stderr.puts. https://github.com/TwP/logging/issues/31
+STDERR.sync = true
+
+# Hacky way to turn off Indexer logging by default, say only
+# log things higher than fatal, which is nothing.
+require 'traject/indexer/settings'
+Traject::Indexer::Settings.defaults["log.level"] = "gt.fatal"
+
+def support_file_path(relative_path)
+  return File.expand_path(File.join("test_support", relative_path), File.dirname(__FILE__))
+end
+
+# The 'assert' method I don't know why it's not there
+def assert_length(length, obj, msg = nil)
+  unless obj.respond_to? :length
+    raise ArgumentError, "object with assert_length must respond_to? :length", obj
+  end
+
+
+  msg ||= "Expected length of #{obj} to be #{length}, but was #{obj.length}"
+
+  assert_equal(length, obj.length, msg.to_s )
+end
+
+def assert_start_with(start_with, obj, msg = nil)
+  msg ||= "expected #{obj} to start with #{start_with}"
+
+  assert obj.start_with?(start_with), msg
+end
+
+
+# An empty record, for making sure extractors and macros work when
+# the fields they're looking for aren't there
+
+def empty_record
+  rec = MARC::Record.new
+  rec.append(MARC::ControlField.new('001', '000000000'))
+  rec
+end
+
+# pretends to be a SolrJ HTTPServer-like thing, just kind of mocks it up
+# and records what happens and simulates errors in some cases.
+class MockSolrServer
+  attr_accessor :things_added, :url, :committed, :parser, :shutted_down
+
+  def initialize(url)
+    @url =  url
+    @things_added = []
+    @add_mutex = Mutex.new
+  end
+
+  def add(thing)
+    @add_mutex.synchronize do # easy peasy threadsafety for our mock
+      if @url == "http://no.such.place"
+        raise org.apache.solr.client.solrj.SolrServerException.new("mock bad uri", java.io.IOException.new)
+      end
+
+      # simulate a multiple id error please
+      if [thing].flatten.find {|doc| doc.getField("id").getValueCount() != 1}
+        raise org.apache.solr.client.solrj.SolrServerException.new("mock non-1 size of 'id'")
+      else
+        things_added << thing
+      end
+    end
+  end
+
+  def commit
+    @committed = true
+  end
+
+  def setParser(parser)
+    @parser = parser
+  end
+
+  def shutdown
+    @shutted_down = true
+  end
+
+end

data/test/test_support/bad_subfield_code.marc

@@ -0,0 +1 @@
+00582cam  2200217   4500001000800000005001700008008004100025035001200066035001400078035001900092035002300111040001300134049000800147050002000155100001800175245003300193260003900226300001000265910002600275991006300301117499920020328125800.0670101s1959    fr            000 0 fre    a1174999  aABK6934EI  a(MdBJ)99005528  a(CaOTULAS)00190819  aOTUbENG00aJHE 4aPQ2605 A873bC61 aCayrol, Jean.14aLes corps �etrangers, roman.  aParis��bEditions du Seuilc[1959]  a188p.  a1174999bHorizon bib#  aPH4605.C36 C6 1959flcbelccc. 1q0i2240984leoffsmelsc

data/test/test_support/bad_utf_byte.utf8.marc

@@ -0,0 +1 @@
+00083    a2200037   4500300004500000  aThis is a bad byte: '�' and another: '�'

data/test/test_support/escaped_character_reference.marc8.marc

@@ -0,0 +1 @@
+00138cam  2200049Ia 45000010008000002600080000082196384  aRio de Janeiro escaped replacement char: � .bEditora Record,c2000.

data/test/test_support/one-marc8.mrc

@@ -0,0 +1 @@
+00663cam  2200217Ia 4500001000800000005001700008008004100025020001500066035001600081040001300097049000900110090002200119100002000141245010100161260004500262300002100307650001800328910002600346991006100372994001200433219638420010919155148.0010626s2000    bl            000 0 por d  a8501058785  aocm47198253  aIXAcIXA  aJHEE  aJZ1308b.S26 20001 aSantos, M�ilton10aPor uma outra globaliza�c�ao :bdo pensamemto �unico �a consci�encia universal /cMilton Santos.  aRio de Janeiro :bEditora Record,c2000.  a174 p. ;c21 cm. 0aGlobalization  a2196384bHorizon bib#  aJZ1308.S26 2000flcbelccc. 1q0i3557489lemainmemsel  aX0bJHE