traject 3.1.0.rc1 → 3.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 06c28d37f9aafafe709a146c7612e5b5d8a5c58a61fd1502823a38dc52b9d05b
-  data.tar.gz: 2e38b2b8c4030456f3757ae6062231268110d68ef07e10cab722b4074ccd570c
+  metadata.gz: 01ca968682bb3fc2a8313131ef6344bfc9e5418b767b2900c3d799caa356d016
+  data.tar.gz: a3fd6c9a3bec88c6ba592500ea170357b059533f28c5ba3fb2fe72de39702a2a
 SHA512:
-  metadata.gz: 04561a77a3e6f2073198983b5bf7d4e35cc9f52bccc1211487cc4c850b0f0b0fc9395a7c87e6ed90061f4a15af57516434d260c649fbc43ea65a0c6435194818
-  data.tar.gz: c7312156c3be556218e319e35ae76aa97fbae5fad6720dbce2e4a046ec90603f5de34fe2cb055425fb3da499922fba50c7d4a6445858793bb0a4fb26cf8f7b29
+  metadata.gz: 93547927e90b7947588c1983bd37de4651722bebaff7aaad2d3965ec46eed8c647971f1ce093beb86a5c47c2efa999516d00fb6956682c98913cd54ea5a1a2b8
+  data.tar.gz: 128ea6e2517711f2324541215f814430f963b0042ae321ebc3f29646398990dc09d2a307acc32fb89a70142d74763fdfee6a61d220043622b5b8b11fbcd645d8

data/.github/workflows/ruby.yml

@@ -0,0 +1,26 @@
+name: CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: ['**']
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby: [ '2.4', '2.5', '2.6', '2.7', 'jruby-9.1', 'jruby-9.2' ]
+    name: Ruby ${{ matrix.ruby }}
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Ruby
+      uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: ${{ matrix.ruby }}
+    - name: Install dependencies
+      run: bundle install --jobs 4 --retry 3
+    - name: Run tests
+      run: bundle exec rake

data/CHANGES.md

@@ -1,5 +1,47 @@
 # Changes
 
+## Next
+
+*
+
+*
+
+*
+
+## 3.5.0
+
+* `traject -v` and `traject -h` correctly return 0 exit code indicating success.
+
+* upgrade to slop gem  4.x, which carries with it a slightly different format of human-readable command-line arg errors, should be otherwise invisible.
+
+* the SolrJsonWriter now supports HTTP basic auth credentials embedded in `solr.url` or `solr.update_url`, eg `http://user:pass@example.org/solr` https://github.com/traject/traject/pull/262
+
+
+## 3.4.0
+
+* XML-mode `extract_xpath` now supports extracting attribute values with xpath @attr syntax.
+
+## 3.3.0
+
+* `Traject::Macros::Marc21Semantics.publication_date` now gets date from 264 before 260. https://github.com/traject/traject/pull/233
+
+* Allow hashie 4.x in gemspec https://github.com/traject/traject/pull/234
+
+* Allow `http` gem 4.x versions. https://github.com/traject/traject/pull/236
+
+* Can now call class-level Indexer.configure multiple times https://github.com/sciencehistory/scihist_digicoll/pull/525
+
+## 3.2.0
+
+* NokogiriReader has a "nokogiri.strict_mode" setting. Set to true or string 'true' to ask Nokogori to parse in strict mode, so it will immediately raise on ill-formed XML, instead of nokogiri's default to do what it can with it. https://github.com/traject/traject/pull/226
+
+* SolrJsonWriter
+
+  * Utility method `delete_all!` sends a delete all query to the Solr URL endpoint. https://github.com/traject/traject/pull/227
+
+  * Allow basic auth configuration of the default http client via `solr_writer.basic_auth_user` and `solr_writer.basic_auth_password`. https://github.com/traject/traject/pull/231
+
+
 ## 3.1.0
 
 ### Added
@@ -24,6 +66,10 @@
 
   * SolrJsonWriter now respects a `solr_writer.http_timeout` setting, in seconds, to be passed to HTTPClient instance. https://github.com/traject/traject/pull/219
 
+  * Only runs thread pool shutdown code (and logging) if there is a `solr_writer.batch_size` greater than 0. Keep it out of the logs if it was a no-op anyway.
+
+  * Logs at DEBUG level every time it sends an update request to solr
+
 * Nokogiri dependency for the NokogiriReader increased to `~> 1.9`. When using Jruby `each_record_xpath`, resulting yielded documents may have xmlns declarations on different nodes than in MRI (and previous versions of nokogiri), but we could find now way around this with nokogiri >= 1.9.0. The documents should still be semantically equivalent for namespace use. This was necessary to keep JRuby Nokogiri XML working with recent Nokogiri releases.  https://github.com/traject/traject/pull/209
 
 * LineWriter guesses better about when to auto-close, and provides an optional explicit setting in case it guesses wrong. (thanks @justinlittman) https://github.com/traject/traject/pull/211

data/README.md

@@ -9,7 +9,7 @@ Traject can also be generalized to a set of tools for getting structured data fr
 **Traject is stable, mature software, that is already being used in production by its authors and several other institutions.**
 
 [![Gem Version](https://badge.fury.io/rb/traject.png)](http://badge.fury.io/rb/traject)
-[![Build Status](https://travis-ci.org/traject/traject.png)](https://travis-ci.org/traject/traject)
+[![CI Status](https://github.com/traject/traject/workflows/CI/badge.svg?branch=master)](https://github.com/traject/traject/actions?query=workflow%3ACI+branch%3Amaster)
 
 
 ## Background/Goals
@@ -175,6 +175,8 @@ TranslationMap use above is just one example of a transformation macro, that tra
 * `append("--after each value")`
 * `gsub(/regex/, "replacement")`
 * `split(" ")`: take values and split them, possibly result in multiple values.
+* `transform(proc)`: transform each existing macro using a proc, kind of like `map`.
+   eg `to_field "something", extract_xml("//author"), transform( ->(author) { "#{author.last}, #{author.first}" })
 
 You can add on as many transformation macros as you want, they will be applied to output in order.
 

data/doc/settings.md

@@ -83,7 +83,8 @@ settings are applied first of all. It's recommended you use `provide`.
 ### Writing to solr
 
 * `json_writer.pretty_print`: used by the JsonWriter, if set to true, will output pretty printed json (with added whitespace) for easier human readability. Default false.
-* `solr.url`: URL to connect to a solr instance for indexing, eg http://example.org:8983/solr . Command-line short-cut `-u`.
+
+* `solr.url`: URL to connect to a solr instance for indexing, eg http://example.org:8983/solr . Command-line short-cut `-u`. (Can include embedded HTTP basic auth as eg `http://user:pass@example.org/solr`)
 
 * `solr.version`: Set to eg "1.4.0", "4.3.0"; currently un-used, but in the future will control some default settings, and/or sanity check and warn you if you're doing something that might not work with that version of solr. Set now for help in the future.
 
@@ -93,6 +94,9 @@ settings are applied first of all. It's recommended you use `provide`.
 
 * `solr_writer.thread_pool`: defaults to 1 (single bg thread). A thread pool is used for submitting docs to solr. Set to 0 or nil to disable threading. Set to 1, there will still be a single bg thread doing the adds. May make sense to set higher than number of cores on your indexing machine, as these threads will mostly be waiting on Solr. Speed/capacity of your solr might be more relevant. Note that processing_thread_pool threads can end up submitting to solr too, if solr_json_writer.thread_pool is full.
 
+* `solr_writer.basic_auth_user`, `solr_writer.basic_auth_password`: Not set by default but when both are set the default writer is configured with basic auth. You can also just embed basic
+auth credentials in `solr.url` using standard URI syntax.
+
 
 ### Dealing with MARC data
 

data/doc/xml.md

@@ -72,6 +72,16 @@ You can use all the standard transforation macros in Traject::Macros::Transforma
 to_field "something", extract_xpath("//value"), first_only, translation_map("some_map"), default("no value")
 ```
 
+### selecting attribute values
+
+Just works, using xpath syntax for selecting an attribute:
+
+
+```ruby
+# gets status value in:  <oai:header status="something">
+to_field "status", extract_xpath("//oai:record/oai:header/@status")
+```
+
 
 ### selecting non-text nodes
 

data/lib/traject/command_line.rb

@@ -29,10 +29,10 @@ module Traject
       self.console = $stderr
 
       self.orig_argv      = argv.dup
-      self.remaining_argv = argv
 
-      self.slop    = create_slop!
-      self.options = parse_options(self.remaining_argv)
+      self.slop    = create_slop!(argv)
+      self.options = self.slop
+      self.remaining_argv = self.slop.arguments
     end
 
     # Returns true on success or false on failure; may also raise exceptions;
@@ -40,11 +40,11 @@ module Traject
     def execute
       if options[:version]
         self.console.puts "traject version #{Traject::VERSION}"
-        return
+        return true
       end
       if options[:help]
-        self.console.puts slop.help
-        return
+        self.console.puts slop.to_s
+        return true
       end
 
 
@@ -179,11 +179,11 @@ module Traject
     end
 
     def arg_check!
-      if options[:command] == "process" && (options[:conf].nil? || options[:conf].length == 0)
+      if options[:command] == "process" && (!options[:conf] || options[:conf].length == 0)
         self.console.puts "Error: Missing required configuration file"
         self.console.puts "Exiting..."
         self.console.puts
-        self.console.puts self.slop.help
+        self.console.puts self.slop.to_s
         exit 2
       end
     end
@@ -234,28 +234,36 @@ module Traject
     end
 
 
-    def create_slop!
-      return Slop.new(:strict => true) do
-        banner "traject [options] -c configuration.rb [-c config2.rb] file.mrc"
+    def create_slop!(argv)
+      options = Slop::Options.new do |o|
+        o.banner = "traject [options] -c configuration.rb [-c config2.rb] file.mrc"
 
-        on 'v', 'version', "print version information to stderr"
-        on 'd', 'debug', "Include debug log, -s log.level=debug"
-        on 'h', 'help', "print usage information to stderr"
-        on 'c', 'conf', 'configuration file path (repeatable)', :argument => true, :as => Array
-        on :i, 'indexer', "Traject indexer class name or shortcut", :argument => true, default: "marc"
-        on :s, :setting, "settings: `-s key=value` (repeatable)", :argument => true, :as => Array
-        on :r, :reader, "Set reader class, shortcut for -s reader_class_name=", :argument => true
-        on :o, "output_file", "output file for Writer classes that write to files", :argument => true
-        on :w, :writer, "Set writer class, shortcut for -s writer_class_name=", :argument => true
-        on :u, :solr, "Set solr url, shortcut for -s solr.url=", :argument => true
-        on :t, :marc_type, "xml, json or binary. shortcut for -s marc_source.type=", :argument => true
-        on :I, "load_path", "append paths to ruby $LOAD_PATH", :argument => true, :as => Array, :delimiter => ":"
+        o.on '-v', '--version', "print version information to stderr"
+        o.on '-d', '--debug', "Include debug log, -s log.level=debug"
+        o.on '-h', '--help', "print usage information to stderr"
+        o.array '-c', '--conf', 'configuration file path (repeatable)', :delimiter => nil
+        o.string "-i", '--indexer', "Traject indexer class name or shortcut", :default => "marc"
+        o.array "-s", "--setting", "settings: `-s key=value` (repeatable)", :delimiter => nil
+        o.string "-r", "--reader", "Set reader class, shortcut for -s reader_class_name="
+        o.string "-o", "--output_file", "output file for Writer classes that write to files"
+        o.string "-w", "--writer", "Set writer class, shortcut for -s writer_class_name="
+        o.string "-u", "--solr", "Set solr url, shortcut for -s solr.url="
+        o.string "-t", "--marc_type", "xml, json or binary. shortcut for -s marc_source.type="
+        o.array "-I", "--load_path", "append paths to ruby $LOAD_PATH", :delimiter => ":"
 
-        on :x, "command", "alternate traject command: process (default); marcout; commit", :argument => true, :default => "process"
+        o.string "-x", "--command", "alternate traject command: process (default); marcout; commit", :default => "process"
 
-        on "stdin", "read input from stdin"
-        on "debug-mode", "debug logging, single threaded, output human readable hashes"
+        o.on "--stdin", "read input from stdin"
+        o.on "--debug-mode", "debug logging, single threaded, output human readable hashes"
       end
+
+      options.parse(argv)
+    rescue Slop::Error => e
+      self.console.puts "Error: #{e.message}"
+      self.console.puts "Exiting..."
+      self.console.puts
+      self.console.puts options.to_s
+      exit 1
     end
 
     def initialize_indexer!
@@ -267,22 +275,5 @@ module Traject
 
       return indexer
     end
-
-    def parse_options(argv)
-
-      begin
-        self.slop.parse!(argv)
-      rescue Slop::Error => e
-        self.console.puts "Error: #{e.message}"
-        self.console.puts "Exiting..."
-        self.console.puts
-        self.console.puts slop.help
-        exit 1
-      end
-
-      return self.slop.to_hash
-    end
-
-
   end
 end

data/lib/traject/indexer.rb

@@ -190,7 +190,7 @@ class Traject::Indexer
     instance_eval(&block)
   end
 
-  ## Class level configure block accepted too, and applied at instantiation
+  ## Class level configure block(s) accepted too, and applied at instantiation
   #  before instance-level configuration.
   #
   #  EXPERIMENTAL, implementation may change in ways that effect some uses.
@@ -199,8 +199,14 @@ class Traject::Indexer
   #  Note that settings set by 'provide' in subclass can not really be overridden
   #  by 'provide' in a next level subclass. Use self.default_settings instead, with
   #  call to super.
+  #
+  #  You can call this .configure multiple times, blocks are added to a list, and
+  #  will be used to initialize an instance in order.
+  #
+  #  The main downside of this workaround implementation is performance, even though
+  #  defined at load-time on class level, blocks are all executed on every instantiation.
   def self.configure(&block)
-    @class_configure_block = block
+    (@class_configure_blocks ||= []) << block
   end
 
   def self.apply_class_configure_block(instance)
@@ -208,8 +214,10 @@ class Traject::Indexer
     if self.superclass.respond_to?(:apply_class_configure_block)
       self.superclass.apply_class_configure_block(instance)
     end
-    if @class_configure_block
-      instance.configure(&@class_configure_block)
+    if @class_configure_blocks && !@class_configure_blocks.empty?
+      @class_configure_blocks.each do |block|
+        instance.configure(&block)
+      end
     end
   end
 

data/lib/traject/macros/marc21.rb

@@ -15,7 +15,7 @@ module Traject::Macros
     # field/substring specification.
     #
     # First argument is a string spec suitable for the MarcExtractor, see
-    # MarcExtractor::parse_string_spec.
+    # Traject::MarcExtractor::Spec.
     #
     # Second arg is optional options, including options valid on MarcExtractor.new,
     # and others. By default, will de-duplicate results, but see :allow_duplicates
@@ -42,11 +42,11 @@ module Traject::Macros
     #
     # * :translation_map => String: translate with named translation map looked up in load
     #       path, uses Tranject::TranslationMap.new(translation_map_arg).
-    #       **Instead**, use `extract_marc(whatever), translation_map(translation_map_arg)
+    #       **Instead**, use `extract_marc(whatever), translation_map(translation_map_arg)`
     #
     # * :trim_punctuation => true; trims leading/trailing punctuation using standard algorithms that
     #     have shown themselves useful with Marc, using Marc21.trim_punctuation. **Instead**, use
-    #    `extract_marc(whatever), trim_punctuation
+    #    `extract_marc(whatever), trim_punctuation`
     #
     # * :default => String: if otherwise empty, add default value. **Instead**, use `extract_marc(whatever), default("default value")`
     #

data/lib/traject/macros/marc21_semantics.rb

@@ -26,19 +26,19 @@ module Traject::Macros
         accumulator.concat list.uniq if list
       end
     end
-   
+
     # If a num begins with a known OCLC prefix, return it without the prefix.
     # otherwise nil.
     #
-    # Allow (OCoLC) and/or ocn/ocm/on 
-    
+    # Allow (OCoLC) and/or ocn/ocm/on
+
     OCLCPAT = /
       \A\s*
       (?:(?:\(OCoLC\)) |
          (?:\(OCoLC\))?(?:(?:ocm)|(?:ocn)|(?:on))
          )(\d+)
          /x
-         
+
     def self.oclcnum_extract(num)
       if m = OCLCPAT.match(num)
         return m[1]
@@ -364,13 +364,16 @@ module Traject::Macros
           end
         end
       end
-      # Okay, nothing from 008, try 260
+      # Okay, nothing from 008, first try 264, then try 260
       if found_date.nil?
+        v264c = MarcExtractor.cached("264c", :separator => nil).extract(record).first
         v260c = MarcExtractor.cached("260c", :separator => nil).extract(record).first
         # just try to take the first four digits out of there, we're not going to try
         # anything crazy.
-        if m = /(\d{4})/.match(v260c)
+        if m = /(\d{4})/.match(v264c)
           found_date = m[1].to_i
+        elsif m = /(\d{4})/.match(v260c)
+            found_date = m[1].to_i
         end
       end
 
@@ -519,11 +522,11 @@ module Traject::Macros
 
     # Extracts LCSH-carrying fields, and formatting them
     # as a pre-coordinated LCSH string, for instance suitable for including
-    # in a facet. 
+    # in a facet.
     #
     # You can supply your own list of fields as a spec, but for significant
     # customization you probably just want to write your own method in
-    # terms of the Marc21Semantics.assemble_lcsh method. 
+    # terms of the Marc21Semantics.assemble_lcsh method.
     def marc_lcsh_formatted(options = {})
       spec            = options[:spec] || "600:610:611:630:648:650:651:654:662"
       subd_separator  = options[:subdivison_separator] || " — "
@@ -540,17 +543,17 @@ module Traject::Macros
     end
 
     # Takes a MARC::Field and formats it into a pre-coordinated LCSH string
-    # with subdivision seperators in the right place. 
+    # with subdivision seperators in the right place.
     #
     # For 600 fields especially, need to not just join with subdivision seperator
     # to take acount of $a$d$t -- for other fields, might be able to just
-    # join subfields, not sure. 
+    # join subfields, not sure.
     #
     # WILL strip trailing period from generated string, contrary to some LCSH practice.
     # Our data is inconsistent on whether it has period or not, this was
-    # the easiest way to standardize. 
+    # the easiest way to standardize.
     #
-    # Default subdivision seperator is em-dash with spaces, set to '--' if you want. 
+    # Default subdivision seperator is em-dash with spaces, set to '--' if you want.
     #
     # Cite: "Dash (-) that precedes a subdivision in an extended 600 subject heading
     # is not carried in the MARC record. It may be system generated as a display constant

data/lib/traject/macros/nokogiri_macros.rb

@@ -26,9 +26,15 @@ module Traject
             # Make sure to avoid text content that was all blank, which is "between the children"
             # whitespace.
             result = result.collect do |n|
-              n.xpath('.//text()').collect(&:text).tap do |arr|
-                arr.reject! { |s| s =~ (/\A\s+\z/) }
-              end.join(" ")
+              if n.kind_of?(Nokogiri::XML::Attr)
+                # attribute value
+                n.value
+              else
+                # text from node
+                n.xpath('.//text()').collect(&:text).tap do |arr|
+                  arr.reject! { |s| s =~ (/\A\s+\z/) }
+                end.join(" ")
+              end
             end
           else
             # just put all matches in accumulator as Nokogiri::XML::Node's

data/lib/traject/marc_extractor.rb

@@ -2,9 +2,9 @@ require 'traject/marc_extractor_spec'
 
 module Traject
   # MarcExtractor is a class for extracting lists of strings from a MARC::Record,
-  # according to specifications. See #parse_string_spec for description of string
-  # string arguments used to specify extraction. See #initialize for options
-  # that can be set controlling extraction.
+  # according to specifications. See Traject::MarcExtractor::Spec for description
+  # of string string arguments used to specify extraction. See #initialize for
+  # options that can be set controlling extraction.
   #
   # Examples:
   #

data/lib/traject/nokogiri_reader.rb

@@ -21,6 +21,9 @@ module Traject
   #   If you need to use namespaces here, you need to have them registered with
   #   `nokogiri.default_namespaces`. If your source docs use namespaces, you DO need
   #   to use them in your each_record_xpath.
+  # * nokogiri.strict_mode: if set to `true` or `"true"`, ask Nokogiri to parse in 'strict'
+  #   mode, it will raise a `Nokogiri::XML::SyntaxError` if the XML is not well-formed, instead
+  #   of trying to take it's best-guess correction. https://nokogiri.org/tutorials/ensuring_well_formed_markup.html
   # * nokogiri_reader.extra_xpath_hooks: Experimental in progress, see below.
   #
   # ## nokogiri_reader.extra_xpath_hooks: For handling nodes outside of your each_record_xpath
@@ -87,7 +90,11 @@ module Traject
     end
 
     def each
-      whole_input_doc = Nokogiri::XML.parse(input_stream)
+      config_proc = if settings["nokogiri.strict_mode"]
+        proc { |config| config.strict }
+      end
+
+      whole_input_doc = Nokogiri::XML.parse(input_stream, &config_proc)
 
       if each_record_xpath
         whole_input_doc.xpath(each_record_xpath, default_namespaces).each do |matching_node|