traject 3.1.0 → 3.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7bc0afc820efb8a2479d96913ae67552556978d264e20a55d0a4d4cef5ff9a2a
-  data.tar.gz: 773529d885d46d2ba0afd323d680724678225bd3616a720dba92809479395dce
+  metadata.gz: 2e47e6648ed9fc963d18e10c9be48a30273147c4920cb4b7e448d078fd2398ac
+  data.tar.gz: efa549ebcbd87e599b56b955b4bd26422dfe7de67697aed6b39cb421c3b80677
 SHA512:
-  metadata.gz: 395f1a3cf62cfd0cccbdcd428830feb098586c9ca1edc7adecff31483545a62dbc06631e7c166ea0ce1b229afeddcec7d994245d78871f1a6644c591d6cc432c
-  data.tar.gz: 647f9cfbf0a7a0876cef29b5a90a9fd6d3dd1f6a1010b332d58e132875b5c4a7f8be9bdab1afabb5ec0bc583af9ea0d8a422cc87eec6c04d93c9fe9f4b4d1e9a
+  metadata.gz: 6acdd2b8cfc888b221a1f19cd5197127006be81d0525169d531fc9bf43fe02cc9ec87401e6b2442c57ff0cd483d9884504ac75be92e3718cbbc49208dc97024f
+  data.tar.gz: 30abefa7af9e1c170ae8570aa59b6c571a9acc1eb7b0abf6efd64d97550b678c21c72d76a8156ef3844ab01154111fd3747f96d6346ee8a8d76e747b2cf92e1f

data/.github/workflows/ruby.yml

@@ -0,0 +1,35 @@
+name: CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: ['**']
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby: [ '2.4', '2.5', '2.6', '2.7', '3.0', 'jruby-9.1', 'jruby-9.2' ]
+    name: Ruby ${{ matrix.ruby }}
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Ruby
+      uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: ${{ matrix.ruby }}
+
+    - name: set JAVA_OPTS for jruby-9.1
+      run: echo 'JAVA_OPTS="--add-opens java.base/java.security.cert=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.util.zip=ALL-UNNAMED"' >> $GITHUB_ENV
+      if: ${{ matrix.ruby == 'jruby-9.1' }}
+      # https://github.com/jruby/jruby/issues/4834
+      # Still seems to be an issue in jruby-9.1, but not 9.2
+      # https://github.community/t/conditional-setting-of-env-variables-in-gh-actions/179650
+
+    - name: Install dependencies
+      run: bundle install --jobs 4 --retry 3
+    - name: Run tests
+      run: bundle exec rake

data/CHANGES.md

@@ -1,5 +1,51 @@
 # Changes
 
+## Next
+
+*
+
+*
+
+## 3.6.0
+
+* Tiny backward compat changes for ruby 3.0 compat. https://github.com/traject/traject/pull/263
+
+* Allow gem `http` 5.x in gemspec. https://github.com/traject/traject/pull/269
+
+## 3.5.0
+
+* `traject -v` and `traject -h` correctly return 0 exit code indicating success.
+
+* upgrade to slop gem  4.x, which carries with it a slightly different format of human-readable command-line arg errors, should be otherwise invisible.
+
+* the SolrJsonWriter now supports HTTP basic auth credentials embedded in `solr.url` or `solr.update_url`, eg `http://user:pass@example.org/solr` https://github.com/traject/traject/pull/262
+
+
+## 3.4.0
+
+* XML-mode `extract_xpath` now supports extracting attribute values with xpath @attr syntax.
+
+## 3.3.0
+
+* `Traject::Macros::Marc21Semantics.publication_date` now gets date from 264 before 260. https://github.com/traject/traject/pull/233
+
+* Allow hashie 4.x in gemspec https://github.com/traject/traject/pull/234
+
+* Allow `http` gem 4.x versions. https://github.com/traject/traject/pull/236
+
+* Can now call class-level Indexer.configure multiple times https://github.com/sciencehistory/scihist_digicoll/pull/525
+
+## 3.2.0
+
+* NokogiriReader has a "nokogiri.strict_mode" setting. Set to true or string 'true' to ask Nokogori to parse in strict mode, so it will immediately raise on ill-formed XML, instead of nokogiri's default to do what it can with it. https://github.com/traject/traject/pull/226
+
+* SolrJsonWriter
+
+  * Utility method `delete_all!` sends a delete all query to the Solr URL endpoint. https://github.com/traject/traject/pull/227
+
+  * Allow basic auth configuration of the default http client via `solr_writer.basic_auth_user` and `solr_writer.basic_auth_password`. https://github.com/traject/traject/pull/231
+
+
 ## 3.1.0
 
 ### Added

data/README.md

@@ -8,8 +8,8 @@ Traject can also be generalized to a set of tools for getting structured data fr
 
 **Traject is stable, mature software, that is already being used in production by its authors and several other institutions.**
 
-[![Gem Version](https://badge.fury.io/rb/traject.png)](http://badge.fury.io/rb/traject)
-[![Build Status](https://travis-ci.org/traject/traject.png)](https://travis-ci.org/traject/traject)
+[![Gem Version](https://badge.fury.io/rb/traject.svg)](http://badge.fury.io/rb/traject)
+[![CI Status](https://github.com/traject/traject/workflows/CI/badge.svg?branch=master)](https://github.com/traject/traject/actions?query=workflow%3ACI+branch%3Amaster)
 
 
 ## Background/Goals
@@ -468,6 +468,22 @@ Also see `-I load_path` option and suggestions for Bundler use under Extending W
 See also [Hints for batch and cronjob use](./doc/batch_execution.md) of traject.
 
 
+## A small but complete example
+
+To process a MARC XML file with the data shown in [./examples/marc/tiny.xml](./examples/marc/tiny.xml) you can use save the following configuration as `config.rb`:
+
+```
+to_field 'title', extract_marc('245a', first: true)
+```
+
+and run Traject as follows:
+
+```
+traject -t xml -c config.rb -w Traject::DebugWriter tiny.xml
+```
+
+`-t xml` indicates that the file is a MARC XML file. `-w Traject::DebugWriter` outputs the results to the console (e.g. without saving to Solr).
+
 ## Extending With Your Own Code
 
 Traject config files are full live ruby files, where you can do anything,

data/doc/settings.md

@@ -83,7 +83,8 @@ settings are applied first of all. It's recommended you use `provide`.
 ### Writing to solr
 
 * `json_writer.pretty_print`: used by the JsonWriter, if set to true, will output pretty printed json (with added whitespace) for easier human readability. Default false.
-* `solr.url`: URL to connect to a solr instance for indexing, eg http://example.org:8983/solr . Command-line short-cut `-u`.
+
+* `solr.url`: URL to connect to a solr instance for indexing, eg http://example.org:8983/solr . Command-line short-cut `-u`. (Can include embedded HTTP basic auth as eg `http://user:pass@example.org/solr`)
 
 * `solr.version`: Set to eg "1.4.0", "4.3.0"; currently un-used, but in the future will control some default settings, and/or sanity check and warn you if you're doing something that might not work with that version of solr. Set now for help in the future.
 
@@ -93,6 +94,9 @@ settings are applied first of all. It's recommended you use `provide`.
 
 * `solr_writer.thread_pool`: defaults to 1 (single bg thread). A thread pool is used for submitting docs to solr. Set to 0 or nil to disable threading. Set to 1, there will still be a single bg thread doing the adds. May make sense to set higher than number of cores on your indexing machine, as these threads will mostly be waiting on Solr. Speed/capacity of your solr might be more relevant. Note that processing_thread_pool threads can end up submitting to solr too, if solr_json_writer.thread_pool is full.
 
+* `solr_writer.basic_auth_user`, `solr_writer.basic_auth_password`: Not set by default but when both are set the default writer is configured with basic auth. You can also just embed basic
+auth credentials in `solr.url` using standard URI syntax.
+
 
 ### Dealing with MARC data
 

data/doc/xml.md

@@ -4,6 +4,8 @@ The [NokogiriIndexer](../lib/traject/nokogiri_indexer.md) is a Traject::Indexer
 
 It by default uses the NokogiriReader to read XML and read Nokogiri::XML::Documents, and includes the NokogiriMacros mix-in, with some macros for operating on Nokogiri::XML::Documents.
 
+Plese notice that the recommened mechanism to parse MARC XML files with Traject is via the `-t` parameter (or the via the `provide "marc_source.type", "xml"` setting). The documentation in this page is for those parsing other (non MARC) XML files.
+
 ## On the command-line
 
 You can tell the traject command-line to use the NokogiriIndexer with the `-i xml` flag:
@@ -72,6 +74,16 @@ You can use all the standard transforation macros in Traject::Macros::Transforma
 to_field "something", extract_xpath("//value"), first_only, translation_map("some_map"), default("no value")
 ```
 
+### selecting attribute values
+
+Just works, using xpath syntax for selecting an attribute:
+
+
+```ruby
+# gets status value in:  <oai:header status="something">
+to_field "status", extract_xpath("//oai:record/oai:header/@status")
+```
+
 
 ### selecting non-text nodes
 

data/examples/marc/tiny.xml

@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<collection xmlns="http://www.loc.gov/MARC21/slim" xmlns:marc="http://www.loc.gov/MARC21/slim">
+ <record>
+  <leader>01352cam a2200349 a 4500</leader>
+  <datafield tag="245" ind1="0" ind2="0">
+   <subfield code="6">880-01</subfield>
+   <subfield code="a">Kazoku kankei no shakai shinrigaku /</subfield>
+   <subfield code="c">Osada Masayoshi hen.</subfield>
+  </datafield>
+ </record>
+ <record>
+  <leader>01121ccm a2200289z  4500</leader>
+  <datafield tag="245" ind1="1" ind2="0">
+   <subfield code="a">Powhatan&#39;s daughter :</subfield>
+   <subfield code="b">march</subfield>
+  </datafield>
+  <datafield tag="100" ind1="1" ind2=" ">
+   <subfield code="a">Sousa, John Philip,</subfield>
+   <subfield code="d">1854-1932,</subfield>
+   <subfield code="e">composer.</subfield>
+  </datafield>
+ </record>
+ <record>
+  <leader>01137cam a2200301 a 4500</leader>
+  <datafield tag="245" ind1="1" ind2="0">
+   <subfield code="a">Two pieces /</subfield>
+   <subfield code="c">by Frank O&#39;Hara.</subfield>
+  </datafield>
+  <datafield tag="100" ind1="1" ind2=" ">
+   <subfield code="a">O&#39;Hara, Frank,</subfield>
+   <subfield code="d">1926-1966.</subfield>
+   <subfield code="0">http://id.loc.gov/authorities/names/n79042130</subfield>
+  </datafield>
+ </record>
+</collection>

data/lib/traject/command_line.rb

@@ -29,10 +29,10 @@ module Traject
       self.console = $stderr
 
       self.orig_argv      = argv.dup
-      self.remaining_argv = argv
 
-      self.slop    = create_slop!
-      self.options = parse_options(self.remaining_argv)
+      self.slop    = create_slop!(argv)
+      self.options = self.slop
+      self.remaining_argv = self.slop.arguments
     end
 
     # Returns true on success or false on failure; may also raise exceptions;
@@ -40,11 +40,11 @@ module Traject
     def execute
       if options[:version]
         self.console.puts "traject version #{Traject::VERSION}"
-        return
+        return true
       end
       if options[:help]
-        self.console.puts slop.help
-        return
+        self.console.puts slop.to_s
+        return true
       end
 
 
@@ -179,11 +179,11 @@ module Traject
     end
 
     def arg_check!
-      if options[:command] == "process" && (options[:conf].nil? || options[:conf].length == 0)
+      if options[:command] == "process" && (!options[:conf] || options[:conf].length == 0)
         self.console.puts "Error: Missing required configuration file"
         self.console.puts "Exiting..."
         self.console.puts
-        self.console.puts self.slop.help
+        self.console.puts self.slop.to_s
         exit 2
       end
     end
@@ -234,28 +234,36 @@ module Traject
     end
 
 
-    def create_slop!
-      return Slop.new(:strict => true) do
-        banner "traject [options] -c configuration.rb [-c config2.rb] file.mrc"
+    def create_slop!(argv)
+      options = Slop::Options.new do |o|
+        o.banner = "traject [options] -c configuration.rb [-c config2.rb] file.mrc"
 
-        on 'v', 'version', "print version information to stderr"
-        on 'd', 'debug', "Include debug log, -s log.level=debug"
-        on 'h', 'help', "print usage information to stderr"
-        on 'c', 'conf', 'configuration file path (repeatable)', :argument => true, :as => Array
-        on :i, 'indexer', "Traject indexer class name or shortcut", :argument => true, default: "marc"
-        on :s, :setting, "settings: `-s key=value` (repeatable)", :argument => true, :as => Array
-        on :r, :reader, "Set reader class, shortcut for -s reader_class_name=", :argument => true
-        on :o, "output_file", "output file for Writer classes that write to files", :argument => true
-        on :w, :writer, "Set writer class, shortcut for -s writer_class_name=", :argument => true
-        on :u, :solr, "Set solr url, shortcut for -s solr.url=", :argument => true
-        on :t, :marc_type, "xml, json or binary. shortcut for -s marc_source.type=", :argument => true
-        on :I, "load_path", "append paths to ruby $LOAD_PATH", :argument => true, :as => Array, :delimiter => ":"
+        o.on '-v', '--version', "print version information to stderr"
+        o.on '-d', '--debug', "Include debug log, -s log.level=debug"
+        o.on '-h', '--help', "print usage information to stderr"
+        o.array '-c', '--conf', 'configuration file path (repeatable)', :delimiter => nil
+        o.string "-i", '--indexer', "Traject indexer class name or shortcut", :default => "marc"
+        o.array "-s", "--setting", "settings: `-s key=value` (repeatable)", :delimiter => nil
+        o.string "-r", "--reader", "Set reader class, shortcut for -s reader_class_name="
+        o.string "-o", "--output_file", "output file for Writer classes that write to files"
+        o.string "-w", "--writer", "Set writer class, shortcut for -s writer_class_name="
+        o.string "-u", "--solr", "Set solr url, shortcut for -s solr.url="
+        o.string "-t", "--marc_type", "xml, json or binary. shortcut for -s marc_source.type="
+        o.array "-I", "--load_path", "append paths to ruby $LOAD_PATH", :delimiter => ":"
 
-        on :x, "command", "alternate traject command: process (default); marcout; commit", :argument => true, :default => "process"
+        o.string "-x", "--command", "alternate traject command: process (default); marcout; commit", :default => "process"
 
-        on "stdin", "read input from stdin"
-        on "debug-mode", "debug logging, single threaded, output human readable hashes"
+        o.on "--stdin", "read input from stdin"
+        o.on "--debug-mode", "debug logging, single threaded, output human readable hashes"
       end
+
+      options.parse(argv)
+    rescue Slop::Error => e
+      self.console.puts "Error: #{e.message}"
+      self.console.puts "Exiting..."
+      self.console.puts
+      self.console.puts options.to_s
+      exit 1
     end
 
     def initialize_indexer!
@@ -267,22 +275,5 @@ module Traject
 
       return indexer
     end
-
-    def parse_options(argv)
-
-      begin
-        self.slop.parse!(argv)
-      rescue Slop::Error => e
-        self.console.puts "Error: #{e.message}"
-        self.console.puts "Exiting..."
-        self.console.puts
-        self.console.puts slop.help
-        exit 1
-      end
-
-      return self.slop.to_hash
-    end
-
-
   end
 end

data/lib/traject/debug_writer.rb

@@ -62,7 +62,7 @@ All records are assumed to have a unique id. You can set which field to look in
   def serialize(context)
     h       = context.output_hash
     rec_key = record_number(context)
-    lines   = h.keys.sort.map { |k| @format % [rec_key, k, h[k].join(' | ')] }
+    lines   = h.keys.sort.map { |k| @format % [rec_key, k, (h[k] || []).join(' | ')] }
     lines.push "\n"
     lines.join("\n")
   end

data/lib/traject/indexer.rb

@@ -190,7 +190,7 @@ class Traject::Indexer
     instance_eval(&block)
   end
 
-  ## Class level configure block accepted too, and applied at instantiation
+  ## Class level configure block(s) accepted too, and applied at instantiation
   #  before instance-level configuration.
   #
   #  EXPERIMENTAL, implementation may change in ways that effect some uses.
@@ -199,8 +199,14 @@ class Traject::Indexer
   #  Note that settings set by 'provide' in subclass can not really be overridden
   #  by 'provide' in a next level subclass. Use self.default_settings instead, with
   #  call to super.
+  #
+  #  You can call this .configure multiple times, blocks are added to a list, and
+  #  will be used to initialize an instance in order.
+  #
+  #  The main downside of this workaround implementation is performance, even though
+  #  defined at load-time on class level, blocks are all executed on every instantiation.
   def self.configure(&block)
-    @class_configure_block = block
+    (@class_configure_blocks ||= []) << block
   end
 
   def self.apply_class_configure_block(instance)
@@ -208,8 +214,10 @@ class Traject::Indexer
     if self.superclass.respond_to?(:apply_class_configure_block)
       self.superclass.apply_class_configure_block(instance)
     end
-    if @class_configure_block
-      instance.configure(&@class_configure_block)
+    if @class_configure_blocks && !@class_configure_blocks.empty?
+      @class_configure_blocks.each do |block|
+        instance.configure(&block)
+      end
     end
   end
 

data/lib/traject/macros/marc21.rb

@@ -15,7 +15,7 @@ module Traject::Macros
     # field/substring specification.
     #
     # First argument is a string spec suitable for the MarcExtractor, see
-    # MarcExtractor::parse_string_spec.
+    # Traject::MarcExtractor::Spec.
     #
     # Second arg is optional options, including options valid on MarcExtractor.new,
     # and others. By default, will de-duplicate results, but see :allow_duplicates
@@ -42,11 +42,11 @@ module Traject::Macros
     #
     # * :translation_map => String: translate with named translation map looked up in load
     #       path, uses Tranject::TranslationMap.new(translation_map_arg).
-    #       **Instead**, use `extract_marc(whatever), translation_map(translation_map_arg)
+    #       **Instead**, use `extract_marc(whatever), translation_map(translation_map_arg)`
     #
     # * :trim_punctuation => true; trims leading/trailing punctuation using standard algorithms that
     #     have shown themselves useful with Marc, using Marc21.trim_punctuation. **Instead**, use
-    #    `extract_marc(whatever), trim_punctuation
+    #    `extract_marc(whatever), trim_punctuation`
     #
     # * :default => String: if otherwise empty, add default value. **Instead**, use `extract_marc(whatever), default("default value")`
     #

data/lib/traject/macros/marc21_semantics.rb

@@ -26,19 +26,19 @@ module Traject::Macros
         accumulator.concat list.uniq if list
       end
     end
-   
+
     # If a num begins with a known OCLC prefix, return it without the prefix.
     # otherwise nil.
     #
-    # Allow (OCoLC) and/or ocn/ocm/on 
-    
+    # Allow (OCoLC) and/or ocn/ocm/on
+
     OCLCPAT = /
       \A\s*
       (?:(?:\(OCoLC\)) |
          (?:\(OCoLC\))?(?:(?:ocm)|(?:ocn)|(?:on))
          )(\d+)
          /x
-         
+
     def self.oclcnum_extract(num)
       if m = OCLCPAT.match(num)
         return m[1]
@@ -364,13 +364,16 @@ module Traject::Macros
           end
         end
       end
-      # Okay, nothing from 008, try 260
+      # Okay, nothing from 008, first try 264, then try 260
       if found_date.nil?
+        v264c = MarcExtractor.cached("264c", :separator => nil).extract(record).first
         v260c = MarcExtractor.cached("260c", :separator => nil).extract(record).first
         # just try to take the first four digits out of there, we're not going to try
         # anything crazy.
-        if m = /(\d{4})/.match(v260c)
+        if m = /(\d{4})/.match(v264c)
           found_date = m[1].to_i
+        elsif m = /(\d{4})/.match(v260c)
+            found_date = m[1].to_i
         end
       end
 
@@ -519,11 +522,11 @@ module Traject::Macros
 
     # Extracts LCSH-carrying fields, and formatting them
     # as a pre-coordinated LCSH string, for instance suitable for including
-    # in a facet. 
+    # in a facet.
     #
     # You can supply your own list of fields as a spec, but for significant
     # customization you probably just want to write your own method in
-    # terms of the Marc21Semantics.assemble_lcsh method. 
+    # terms of the Marc21Semantics.assemble_lcsh method.
     def marc_lcsh_formatted(options = {})
       spec            = options[:spec] || "600:610:611:630:648:650:651:654:662"
       subd_separator  = options[:subdivison_separator] || " — "
@@ -540,17 +543,17 @@ module Traject::Macros
     end
 
     # Takes a MARC::Field and formats it into a pre-coordinated LCSH string
-    # with subdivision seperators in the right place. 
+    # with subdivision seperators in the right place.
     #
     # For 600 fields especially, need to not just join with subdivision seperator
     # to take acount of $a$d$t -- for other fields, might be able to just
-    # join subfields, not sure. 
+    # join subfields, not sure.
     #
     # WILL strip trailing period from generated string, contrary to some LCSH practice.
     # Our data is inconsistent on whether it has period or not, this was
-    # the easiest way to standardize. 
+    # the easiest way to standardize.
     #
-    # Default subdivision seperator is em-dash with spaces, set to '--' if you want. 
+    # Default subdivision seperator is em-dash with spaces, set to '--' if you want.
     #
     # Cite: "Dash (-) that precedes a subdivision in an extended 600 subject heading
     # is not carried in the MARC record. It may be system generated as a display constant