traject 0.10.0 → 0.11.0

data/README.md

@@ -321,9 +321,10 @@ You can supply more than one conf file with repeated `-c` arguments.
 
     traject -c connection_conf.rb -c indexing_conf.rb marc_file.mrc
 
-If you leave off the marc_file, traject will try to read from stdin. You can only supply one marc file at a time, but we can take advantage of stdin to get around this:
+If you supply a `--stdin` argument, traject will try to read from stdin.
+You can only supply one marc file at a time, but we can take advantage of stdin to get around this:
 
-    cat some/dir/*.marc | traject -c conf_file.rb
+    cat some/dir/*.marc | traject -c conf_file.rb --stdin
 
 You can set any setting on the command line with `-s key=value`.
 This will over-ride any settings set with `provide` in conf files.
@@ -343,7 +344,7 @@ Use `-u` as a shortcut for `s solr.url=X`
 
     traject -c conf_file.rb -u http://example.com/solr marc_file.mrc
 
-Also see `-I load_path` and `-g Gemfile` options under Extending With Your Own Code.
+Also see `-I load_path` and `-G Gemfile` options under Extending With Your Own Code.
 
 See also [Hints for batch and cronjob use](./doc/batch_execution.md) of traject.
 
@@ -374,9 +375,9 @@ Own Code](./doc/extending.md)
   * translation map files found on the load path or in a
     "./translation_maps" subdir on the load path will be found
     for Traject translation maps.
-* Traject `-g` command line can be used to tell traject to use
+* Traject `-G` command line can be used to tell traject to use
   bundler with a `Gemfile` located at current working dirctory
-  (or give an argument to `-g ./some/myGemfile`)
+  (or give an argument to `-G ./some/myGemfile`)
 
 ## More
 

data/doc/extending.md

@@ -19,9 +19,9 @@ of a couple traject features meant to make it easier.
   * translation map files found on the load path or in a
     "./translation_maps" subdir on the load path will be found
     for Traject translation maps.
-* Traject `-g` command line can be used to tell traject to use
+* Traject `-G` command line can be used to tell traject to use
   bundler with a `Gemfile` located at current working dirctory
-  (or give an argument to `-g ./some/myGemfile`)
+  (or give an argument to `-G ./some/myGemfile`)
 
 ## Custom code local to your project
 
@@ -161,9 +161,9 @@ possibly with version restrictions, in the [Gemfile](http://bundler.io/v1.3/gemf
 Run `bundle install` from the directory with the Gemfile, on any system
 at any time, to make sure specified gems are installed.
 
-**Run traject** with the `-g` flag to tell it to use the Gemfile:
+**Run traject** with the `-G` flag to tell it to use the Gemfile:
 
-   traject -g -c some_traject_config.rb ...
+   traject -G -c some_traject_config.rb ...
 
 Traject will use bundler to setup with the Gemfile, making sure
 the specified versions of all gems are used (and also making sure

data/doc/settings.md

@@ -54,6 +54,11 @@ for commonly used settings, see `traject -h`.
 * `marc4j_reader.source_encoding`: Used by Marc4JReader only when marc.source_type is 'binary', encoding strings accepted
   by marc4j MarcPermissiveStreamReader. Default "BESTGUESS", also "UTF-8", "MARC"
 
+* `marcout.allow_oversized`: Used with `-x marcout` command to output marc when outputting
+     as ISO 2709 binary, set to true or string "true", and the MARC::Writer will have
+     allow_oversized=true set, allowing oversized records to be serialized with length
+    bytes zero'd out -- technically illegal, but can be read by MARC::Reader in permissive mode.
+
 * `output_file`: Output file to write to for operations that write to files: For instance the `marcout` command,
                  or Writer classes that write to files, like Traject::JsonWriter. Has an shortcut
                  `-o` on command line. 

data/lib/traject/command_line.rb

@@ -1,6 +1,7 @@
+# Require as little as possible at top, so we can bundle require later
+# if needed, before requiring anything from the bundle. Can't avoid slop
+# though, to get our bundle arg out, sorry. 
 require 'slop'
-require 'traject'
-require 'traject/indexer'
 
 module Traject
   # The class that executes for the Traject command line utility.
@@ -8,7 +9,7 @@ module Traject
   # Warning, does do things like exit entire program on error at present.
   # You probably don't want to use this class for anything but an actual
   # shell command line, if you want to execute indexing directly, just
-  # use the Traject::Indexer directly. 
+  # use the Traject::Indexer directly.
   #
   # A CommandLine object has a single persistent Indexer object it uses
   class CommandLine
@@ -43,10 +44,16 @@ module Traject
 
       # have to use Slop object to tell diff between
       # no arg supplied and no option -g given at all
-      if slop.present? :gemfile
-        require_bundler_setup(options[:gemfile])
+      if slop.present? :Gemfile
+        require_bundler_setup(options[:Gemfile])
       end
 
+      # We require them here instead of top of file,
+      # so we have done bundler require before we require these.
+      require 'traject'
+      require 'traject/indexer'
+
+
       (options[:load_path] || []).each do |path|
         $LOAD_PATH << path unless $LOAD_PATH.include? path
       end
@@ -114,7 +121,7 @@ module Traject
       when "xml"
         writer = MARC::XMLWriter.new(output_arg)
       when "human"
-        writer = output_arg.kind_of?(String) ? File.open(output_arg, "w:binary") : output_arg
+        writer = output_arg.kind_of?(String) ? File.open(output_arg, "w:binary") : output_arg        
       else
         raise ArgumentError.new("traject marcout unrecognized marcout.type: #{output_type}")
       end
@@ -142,14 +149,17 @@ module Traject
       # * It INSISTS on reading from ARGFV, making it hard to test, or use when you want to give
       #   it a list of files on something other than ARGV.
       #
-      # So for now we do just one file, or stdin if none given. Sorry!
-      if argv.length > 1
-        self.console.puts "Sorry, traject can only handle one input file at a time right now. `#{argv}` Exiting..."
-        exit 1
-      end
-      if argv.length == 0
+      # So for now we do just one file, or stdin if specified. Sorry!
+
+      if options[:stdin]
         indexer.logger.info "Reading from STDIN..."
         io = $stdin
+      elsif argv.length > 1
+        self.console.puts "Sorry, traject can only handle one input file at a time right now. `#{argv}` Exiting..."
+        exit 1
+      elsif argv.length == 0
+        indexer.logger.warn "Warning, no file input given..."
+        io = File.open(File::NULL, 'r')
       else
         indexer.logger.info "Reading from #{argv.first}"
         io = File.open(argv.first, 'r')
@@ -160,15 +170,24 @@ module Traject
     def load_configuration_files!(my_indexer, conf_files)
       conf_files.each do |conf_path|
         begin
-          my_indexer.instance_eval(File.open(conf_path).read, conf_path)
+          file_io = File.open(conf_path)
         rescue Errno::ENOENT => e
           self.console.puts "Could not find configuration file '#{conf_path}', exiting..."
           exit 2
+        end
+
+        begin
+          my_indexer.instance_eval(file_io.read, conf_path)
         rescue Exception => e
-          self.console.puts "Could not parse configuration file '#{conf_path}'"
-          self.console.puts "  #{e.message}"
+          if (conf_trace = e.backtrace.find {|l| l.start_with? conf_path}) &&
+             (conf_trace =~ /\A.*\:(\d+)\:in/)
+            line_number = $1
+          end
+
+          self.console.puts "Error processing configuration file '#{conf_path}' at line #{line_number}"
+          self.console.puts "  #{e.class}: #{e.message}"
           if e.backtrace.first =~ /\A(.*)\:in/
-            self.console.puts "  #{$1}"
+            self.console.puts "  from #{$1}"
           end
           exit 3
         end
@@ -191,10 +210,10 @@ module Traject
     def require_bundler_setup(gemfile=nil)
       if gemfile
         # tell bundler what gemfile to use
-        gem_path = File.expand_path( options[:gemfile] )
+        gem_path = File.expand_path( gemfile )
         # bundler not good at error reporting, we check ourselves
         unless File.exists? gem_path
-          self.console.puts "Gemfile `#{options[:gemfile]}` does not exist, exiting..."
+          self.console.puts "Gemfile `#{gemfile}` does not exist, exiting..."
           self.console.puts
           self.console.puts slop.help
           exit 2
@@ -263,9 +282,11 @@ module Traject
         on :j, "output as pretty printed json, shortcut for -s writer_class_name=JsonWriter -s json_writer.pretty_print=true"
         on :t, :marc_type, "xml, json or binary. shortcut for -s marc_source.type=", :argument => true
         on :I, "load_path", "append paths to ruby $LOAD_PATH", :argument => true, :as => Array, :delimiter => ":"
-        on :g, "gemfile", "run with bundler and optionally specified Gemfile", :argument => :optional, :default => ""
+        on :G, "Gemfile", "run with bundler and optionally specified Gemfile", :argument => :optional, :default => ""
 
         on :x, "command", "alternate traject command: process (default); marcout", :argument => true, :default => "process"
+
+        on "stdin", "read input from stdin"
       end
     end
 

data/lib/traject/macros/marc21.rb

@@ -78,19 +78,27 @@ module Traject::Macros
     # [json] marc-in-json (http://dilettantes.code4lib.org/blog/2010/09/a-proposal-to-serialize-marc-in-json/)
     # [binary] Standard ISO 2709 binary marc. By default WILL be base64-encoded,
     #          assumed destination a solr 'binary' field.
-    #          add option `:binary_escape => false` to do straight binary -- unclear
+    #          * add option `:binary_escape => false` to do straight binary -- unclear
     #          what Solr's documented behavior is when you do this, and add a string
     #          with binary control chars to solr. May do different things in diff
     #          Solr versions, including raising exceptions.
+    #          * add option `:allow_oversized => true` to pass that flat
+    #          to the MARC::Writer. Oversized records will then still be
+    #          serialized, with certain header bytes filled with ascii 0's
+    #          -- technically illegal MARC, but can still be read by
+    #          ruby MARC::Reader in permissive mode.
     def serialized_marc(options)
-      options[:format] = options[:format].to_s
-      raise ArgumentError.new("Need :format => [binary|xml|json] arg") unless %w{binary xml json}.include?(options[:format])
+      format          = options[:format].to_s
+      binary_escape   = (options[:binary_escape] != false)
+      allow_oversized = (options[:allow_oversized] == true)
+
+      raise ArgumentError.new("Need :format => [binary|xml|json] arg") unless %w{binary xml json}.include?(format)
 
       lambda do |record, accumulator, context|
-        case options[:format]
+        case format
         when "binary"
-          binary = record.to_marc
-          binary = Base64.encode64(binary) unless options[:binary_escape] == false
+          binary = MARC::Writer.encode(record, allow_oversized)
+          binary = Base64.encode64(binary) if binary_escape
           accumulator << binary
         when "xml"
           # ruby-marc #to_xml returns a REXML object at time of this writing, bah!@

data/lib/traject/marc4j_reader.rb

@@ -35,8 +35,12 @@ require 'marc'
 #         Default 'BESTGUESS', but marc records in the wild are so wrong here, recommend setting.
 #         (will ALWAYS be transcoded to UTF-8 on the way out. We insist.)
 #
-# * marc4j_reader.jar_dir:   Path to a directory containing Marc4J jar file to use. All .jar's in dir will
+# * marc4j_reader.jar_dir: Path to a directory containing Marc4J jar file to use. All .jar's in dir will
 #                          be loaded. If unset, uses marc4j.jar bundled with traject.
+#
+# * marc4j_reader.keep_marc4j: Keeps the original marc4j record accessible from 
+#                              the eventual ruby-marc record via record#original_marc4j
+
 class Traject::Marc4JReader
   include Enumerable
 
@@ -47,6 +51,13 @@ class Traject::Marc4JReader
     @input_stream = input_stream
 
     ensure_marc4j_loaded!
+
+    if @settings['marc4j_reader.keep_marc4j'] &&
+      ! (MARC::Record.instance_methods.include?(:original_marc4j) &&
+         MARC::Record.instance_methods.include?(:"original_marc4j="))
+      MARC::Record.class_eval('attr_accessor :original_marc4j')
+    end
+
   end
 
   # Loads solrj unless it appears to already be loaded.
@@ -91,6 +102,9 @@ class Traject::Marc4JReader
       begin
         marc4j = internal_reader.next
         rubymarc = convert_marc4j_to_rubymarc(marc4j)
+        if @settings['marc4j_reader.keep_marc4j']
+          rubymarc.original_marc4j = marc4j
+        end
       rescue Exception =>e
         msg = "MARC4JReader: Error reading MARC, fatal, re-raising"
         if marc4j

data/lib/traject/mock_reader.rb

@@ -2,7 +2,7 @@
 require 'marc'
 require 'json'
 module Traject
-  
+
   # A mock reader, designed to do almost no work during a run to provide better benchmarking
   #
   # It pulls in 20 records from the end of this file and then just returns
@@ -12,42 +12,53 @@ module Traject
   #
     # require 'traject/mock_writer'
     # require 'traject/mock_reader'
-    #   
+    #
     # settings do
     #   store "reader_class_name", "Traject::MockReader"
     #   store "writer_class_name", "Traject::MockWriter"
     #   store "mock_reader.limit", 4_000 # default is 10_000
     # end
-  
+
   class MockReader
-    
+
     attr_accessor :limit
-    
+
     # @param [Ignored] input_stream (ignored)
     # @param [Hash] settings (looks only for an integer in 'mock_reader.limit')
     def initialize(input_stream, settings = {})
-      @records = []
-      @limit = settings["mock_reader.limit"]  || 10_000
-      
-      this_file_iter = File.open(__FILE__).each_line
-      
+      @limit = (settings["mock_reader.limit"]  || 10_000).to_i
+
+      @records = load_ndjson(File.open(__FILE__))
+
+      # freeze it immutable for thread safety and performance
+      @records.each {|r| r.fields.freeze}
+    end
+
+    # newline delimited json, assuming no internal unescaped
+    # newlines in json too!
+    def load_ndjson(file_io)
+      records = []
+
+      this_file_iter = file_io.each_line
+
       while true
         line = this_file_iter.next
         break if line =~ /^\_\_END\_\_/
       end
-      
+
       begin
         while true
           json = this_file_iter.next
           next unless json =~ /\S/
-          @records << MARC::Record.new_from_hash(JSON.parse(json))
+          records << MARC::Record.new_from_hash(JSON.parse(json))          
         end
       rescue StopIteration
       end
-      
+
+      return records
     end
 
-    
+
     def each
       unless block_given?
         enum_for(:each)
@@ -58,8 +69,8 @@ module Traject
         end
       end
     end
-    
-    
+
+
   end
 end
 

data/lib/traject/mock_writer.rb

@@ -1,17 +1,5 @@
-# A writer for Traject::Indexer, that just writes out
-# all the output as serialized text with #puts. 
-#
-# Should be thread-safe (ie, multiple worker threads can be calling #put
-# concurrently), by wrapping write to actual output file in a mutex synchronize.
-# This does not seem to effect performance much, as far as I could tell
-# benchmarking.
-#
-# Output will be sent to settings["output_file"] string path, or else
-# settings["output_stream"] (ruby IO object), or else stdout.
-#
-# This class can be sub-classed to write out different serialized
-# reprentations -- subclasses will just override the #serialize
-# method. For instance, see JsonWriter. 
+# A Null writer that does absolutely nothing with records given to it,
+# just drops em on the floor.
 class Traject::MockWriter
   attr_reader :settings
 
@@ -21,7 +9,7 @@ class Traject::MockWriter
 
   def serialize(context)
     # null
-  end    
+  end
 
   def put(context)
     # null

data/lib/traject/solrj_writer.rb

@@ -257,6 +257,12 @@ class Traject::SolrJWriter
       return true
     end
 
+    # Consider Solr server returning HTTP 500 Internal Server Error to be fatal.
+    # This can mean, for instance, that disk space is exhausted on solr server.
+    if e.kind_of?(Java::OrgApacheSolrCommon::SolrException) && e.code == 500
+      return true
+    end
+
     return false
   end
 

data/lib/traject/translation_map.rb

@@ -184,7 +184,7 @@ module Traject
 
     class NotFound < Exception
       def initialize(path)
-        super("No translation map definition file found at '#{path}[.rb|.yaml]' in load path")
+        super("No translation map definition file found at '#{path}[.rb|.yaml]' in load path: #{$LOAD_PATH}")
       end
     end
 

data/lib/traject/version.rb

@@ -1,3 +1,3 @@
 module Traject
-  VERSION = "0.10.0"
+  VERSION = "0.11.0"
 end

data/test/marc4j_reader_test.rb

@@ -75,4 +75,14 @@ describe "Marc4JReader" do
     assert first['245']['a'].encoding.name, "UTF-8"
     assert_equal "Fikr-i Ayāz /", first['245']['a']
   end
+  
+  it "keeps marc4j object when asked" do
+    file = File.new(support_file_path "test_data.utf8.marc.xml")
+    settings = Traject::Indexer::Settings.new("marc_source.type" => "xml", 'marc4j_reader.keep_marc4j' => true)
+    record = Traject::Marc4JReader.new(file, settings).to_a.first
+    assert_kind_of MARC::Record, record
+    assert_kind_of Java::org.marc4j.marc.impl::RecordImpl, record.original_marc4j
+  end
+    
+  
 end

data/traject.gemspec

@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
 
-  spec.add_dependency "marc", ">= 0.7.0"
+  spec.add_dependency "marc", ">= 0.7.1"
   spec.add_dependency "hashie", ">= 2.0.5", "< 2.1" # used for Indexer#settings
   spec.add_dependency "slop", ">= 3.4.5", "< 4.0"   # command line parsing
   spec.add_dependency "yell" # logging

metadata

@@ -2,14 +2,14 @@
 name: traject
 version: !ruby/object:Gem::Version
   prerelease:
-  version: 0.10.0
+  version: 0.11.0
 platform: ruby
 authors:
 - Jonathan Rochkind
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2013-09-04 00:00:00.000000000 Z
+date: 2013-09-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: marc
@@ -17,13 +17,13 @@ dependencies:
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 0.7.0
+        version: 0.7.1
     none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 0.7.0
+        version: 0.7.1
     none: false
   prerelease: false
   type: :runtime