traject 0.15.0 → 0.16.0

data/README.md

@@ -49,8 +49,9 @@ The traject command-line utility requires you to supply it with a configuration
 
 Configuration files are actually just ruby -- so by convention they end in `.rb`.
 
-Don't worry, you don't neccesarily need to know ruby well to write them, they give you a subset of ruby to work with. But the full power
-of ruby is available to you.
+We hope you can write basic useful configuration files without being a ruby expert,
+they give you a subset of ruby to work with. But the full power
+of ruby is available to you if needed. 
 
 **rubyist tip**: Technically, config files are executed with `instance_eval` in a Traject::Indexer instance, so the special commands you see are just methods on Traject::Indexer (or mixed into it). But you can
 call ordinary ruby `require` in config files, etc., too, to load
@@ -84,9 +85,6 @@ settings do
   # you have to tell it.
   provide "marc_source.type", "xml"
 
-  # settings can be set on command line instead of
-  # config file too.
-
   # various others...
   provide "solrj_writer.commit_on_close", "true"
 
@@ -163,39 +161,44 @@ Other examples of the specification string, which can include multiple tag menti
   # "*" is a wildcard in indicator spec.  So
   # 856 with first indicator '0', subfield u.
   to_field "email_addresses", extract_marc("856|0*|u")
-
-  # Instead of joining subfields from the same field
-  # into one string, joined by spaces, leave them
-  # each in separate strings:
-  to_field "isbn", extract_marc("020az", :separator => nil)
-  
-  # Same thing, but more explicit
-  to_field "isbn", extract_marc("020a:020z")
   
-  
-  # Make sure that you don't get any duplicates
-  # by passing in ":deduplicate => true"
-  to_field 'language008', extract_marc('008[35-37]', :deduplicate=>true)
+  # Can list tag twice with different field combinations
+  # to extract separately
+  to_field "isbn", extract_marc("245a:245abcde")
 ~~~
 
 The `extract_marc` function *by default* includes any linked
 MARC `880` fields with alternate-script versions. Another reason
 to use the `:first` option if you really only want one.
 
+By default, specifications with multiple subfields (like "240abc") will produce
+one single string of output for each matching field. Specifications
+with single subfields (like "020a") will split subfields and produce
+an output string for each matching subfield. 
+
 For MARC control (aka 'fixed') fields, you can use square
 brackets to take a slice by byte offset.
 
+~~~ruby
     to_field "langauge_code", extract_marc("008[35-37]")
+~~~
+
+For more information on extraction specifications, see
+the [MarcExtractor class](./lib/traject/marc_extractor.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/MarcExtractor)).
 
 `extract_marc` also supports `translation maps` similar
-to SolrMarc's. There will be some translation maps built in,
-and you can provide your own. translation maps can be supplied
+to SolrMarc's. There are some translation maps provided by traject,
+and you can also define your own. translation maps can be supplied
 in yaml or ruby.  Translation maps are especially useful
-for mapping form MARC codes to user-displayable strings. See Traject::TranslationMap for more info:
+for mapping form MARC codes to user-displayable strings:
 
+~~~ruby
     # "translation_map" will be passed to Traject::TranslationMap.new
     # and the created map used to translate all values
     to_field "language", extract_marc("008[35-37]:041a:041d", :translation_map => "marc_language_code")
+~~~
+
+See [Traject::TranslationMap](./lib/traject/translation_map.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/TranslationMap)) for more info on translation mapping. 
 
 #### Direct indexing logic vs. Macros
 
@@ -348,11 +351,11 @@ This will over-ride any settings set with `provide` in conf files.
 There are some built-in command-line option shortcuts for useful
 settings:
 
-Use `-j` to output as pretty-printed JSON
-hashes, instead of sending to solr. Useful for debugging or sanity
-checking.
+Use `--debug-mode` to output in a human-readable format, instead of sending to solr.
+Also turns on debug logging and restricts processing to single-threaded. Useful for
+debugging or sanity checking.
 
-    traject -j -c conf_file.rb marc_file
+    traject --debug-mode -c conf_file.rb marc_file
 
 Use `-u` as a shortcut for `s solr.url=X`
 

data/doc/settings.md

@@ -4,7 +4,8 @@ Traject settings are a flat list of key/value pairs -- a single
 Hash, not nested. Keys are always strings, and dots (".") can be
 used for grouping and namespacing.
 
-Values are usually strings, but occasionally something else.
+Values are usually strings, but occasionally something else. String values can be easily
+set via the command line. 
 
 Settings can be set in configuration files, usually like:
 
@@ -17,6 +18,11 @@ end
 or on the command line: `-s key=value`.  There are also some command line shortcuts
 for commonly used settings, see `traject -h`. 
 
+`provide` will only set the key if it was previously unset, so first time to set 'wins'. And command-line 
+settings are applied first of all. It's recommended you use `provide`. 
+
+`store` is also available, and forces setting of the new value overriding any previous value set. 
+
 ## Known settings
 
 * `debug_ascii_progress`: true/'true' to print ascii characters to STDERR indicating progress. Note,
@@ -101,4 +107,4 @@ for commonly used settings, see `traject -h`.
                                     Note that processing_thread_pool threads can end up submitting
                                     to solr too, if solrj_writer.thread_pool is full. 
 
-* `writer_class_name`: a Traject Writer class, used by indexer to send processed dictionaries off. Default Traject::SolrJWriter, also available Traject::JsonWriter. See Traject::Indexer for more info. Command line shortcut `-w`
+* `writer_class_name`: a Traject Writer class, used by indexer to send processed dictionaries off. Default Traject::SolrJWriter, also available Traject::JsonWriter. See Traject::Indexer for more info. Command line shortcut `-w`

data/lib/traject/command_line.rb

@@ -268,10 +268,6 @@ module Traject
       if options[:solr]
         settings["solr.url"] = options[:solr]
       end
-      if options[:j]
-        settings["writer_class_name"] = "JsonWriter"
-        settings["json_writer.pretty_print"] = "true"
-      end
       if options[:marc_type]
         settings["marc_source.type"] = options[:marc_type]
       end
@@ -296,12 +292,11 @@ module Traject
         on :o, "output_file", "output file for Writer classes that write to files", :argument => true
         on :w, :writer, "Set writer class, shortcut for -s writer_class_name=", :argument => true
         on :u, :solr, "Set solr url, shortcut for -s solr.url=", :argument => true
-        on :j, "output as pretty printed json, shortcut for -s writer_class_name=JsonWriter -s json_writer.pretty_print=true"
         on :t, :marc_type, "xml, json or binary. shortcut for -s marc_source.type=", :argument => true
         on :I, "load_path", "append paths to ruby $LOAD_PATH", :argument => true, :as => Array, :delimiter => ":"
         on :G, "Gemfile", "run with bundler and optionally specified Gemfile", :argument => :optional, :default => nil
 
-        on :x, "command", "alternate traject command: process (default); marcout", :argument => true, :default => "process"
+        on :x, "command", "alternate traject command: process (default); marcout; commit", :argument => true, :default => "process"
 
         on "stdin", "read input from stdin"
         on "debug-mode", "debug logging, single threaded, output human readable hashes"

data/lib/traject/macros/marc21_semantics.rb

@@ -144,7 +144,7 @@ module Traject::Macros
     # return the filing version (i.e., the string without the 
     # non-filing characters)
     
-    def self.filing_version(field, str, spechash)
+    def self.filing_version(field, str, spec)
       # Control fields don't have non-filing characters
       return str if field.kind_of? MARC::ControlField
       
@@ -155,7 +155,7 @@ module Traject::Macros
       # The spechash must either (a) have no subfields specified, or
       # (b) include the first subfield in the record
       
-      subs = spechash[:subfields]
+      subs = spec.subfields
       return str unless subs && subs.include?(field.subfields[0].code)
       
       # OK. If we got this far we actually need to strip characters off the string
@@ -183,7 +183,7 @@ module Traject::Macros
       lambda do |record, accumulator|
         codes = extractor.collect_matching_lines(record) do |field, spec, extractor|
           if extractor.control_field?(field)
-            (spec[:bytes] ? field.value.byteslice(spec[:bytes]) : field.value)
+            (spec.bytes ? field.value.byteslice(spec.bytes) : field.value)
           else
             extractor.collect_subfields(field, spec).collect do |value|
               # sometimes multiple language codes are jammed together in one subfield, and
@@ -212,9 +212,16 @@ module Traject::Macros
       extractor = MarcExtractor.new(spec)
 
       lambda do |record, accumulator|
-        accumulator.concat( extractor.collect_matching_lines(record) do |field, spec, extractor|
+        values = extractor.collect_matching_lines(record) do |field, spec, extractor|
           extractor.collect_subfields(field, spec) unless (field.tag == "490" && field.indicator1 == "1")
-        end.compact)
+        end.compact
+
+        # trim punctuation
+        values.collect! do |s|
+          Marc21.trim_punctuation(s)
+        end
+
+        accumulator.concat( values )
       end
     end
 

data/lib/traject/marc_extractor.rb

@@ -6,9 +6,79 @@ module Traject
   #
   # Examples:
   #
-  #    array_of_stuff = MarcExtractor.new("001:245abc:700a").extract(marc_record)
-  #    values         = MarcExtractor.new("040a", :separator => nil).extract(marc_record)
+  #    array_of_stuff   = MarcExtractor.new("001:245abc:700a").extract(marc_record)
+  #    values           = MarcExtractor.new("245a:245abc").extract_marc(marc_record)
+  #    seperated_values = MarcExtractor.new("020a:020z").extract(marc_record)
+  #    bytes            = MarcExtractor.new("008[35-37]")
   #
+  # == String extraction specifications
+  #
+  # Extraction directions are supplied in strings, usually as the first
+  # parameter to MarcExtractor.new or MarcExtractor.cached. These specifications
+  # are also the first parameter to the #marc_extract macro. 
+  #
+  # A String specification is a string (or array of strings) which consists
+  # of one or more Data and Control Field Specifications seperated by colons. 
+  #
+  # A Data Field Specification is of the form:
+  #  `{tag}{|indicators|}{subfields}`
+  # * {tag} is three chars (usually but not neccesarily numeric)
+  # * {indicators} are optional two chars enclosed in pipe ('|') characters,
+  # * {subfields} are optional list of chars (alphanumeric)
+  #
+  # indicator spec must be two chars, but one can be * meaning "don't care".
+  # space to mean 'blank'
+  #
+  # "245|01|abc65:345abc:700|*5|:800"
+  #
+  # A Control Field Specification is used with tags for control (fixed) fields (ordinarily fields 001-010)
+  # and includes a tag and a a byte slice specification. 
+  #
+  #  "008[35-37]:007[5]""
+  #  => bytes 35-37 inclusive of any field 008, and byte 5 of any field 007 (TODO: Should we support
+  #    "LDR" as a pseudo-tag to take byte slices of leader?)
+  #
+  # * subfields and indicators can only be provided for marc data/variable fields
+  # * byte slice can only be provided for marc control fields (generally tags less than 010)
+  #
+  # == Subfield concatenation
+  #
+  # Normally, for a spec including multiple subfield codes, multiple subfields
+  # from the same MARC field will be concatenated into one string separated by spaces:
+  #
+  #    600 a| Chomsky, Noam x| Philosophy.
+  #    600 a| Chomsky, Noam x| Political and social views.
+  #    MarcExtractor.new("600ax").extract(record)
+  #    # results in two values sent to Solr: 
+  #    "Chomsky, Noam Philosophy."
+  #    "Chomsky, Noam Political and social views."
+  #
+  # You can turn off this concatenation and leave individual subfields in seperate
+  # strings by setting the `separator` option to nil:
+  #
+  #    MarcExtractor.new("600ax", :separator => nil).extract(record)
+  #    # Results in four values being sent to Solr (or 3 if you de-dup):
+  #    "Chomksy, Noam"
+  #    "Philosophy."
+  #    "Chomsky, Noam"
+  #    "Political and social views."
+  #
+  # However, **the default is different for specifications with only a single
+  # subfield**, these are by default kept seperated:
+  #
+  #    020 a| 285197145X a| 9782851971456
+  #    MarcExtractor.new("020a:020z").extract(record)
+  #    # two seperate strings sent to Solr:
+  #    "285197145X"
+  #    "9782851971456"
+  #
+  # For single subfield specifications, you force concatenation by
+  # repeating the subfield specification:
+  #
+  #    MarcExtractor.new("020aa:020zz").extract(record)
+  #    # would result in a single string sent to solr for
+  #    # the single field, by default space-separated:
+  #    "285197145X 9782851971456"
   #
   # == Note on Performance and MarcExtractor creation and reuse
   #
@@ -37,14 +107,15 @@ module Traject
   class MarcExtractor
     attr_accessor :options, :spec_hash
 
-    # Take a hash that's the output of #parse_string_spec, return
-    # an array of strings extracted from a marc record accordingly
+    # First arg is a specification for extraction of data from a MARC record. 
+    # Specification can be given in two forms:
     #
-    # Second arg can either be a string specification that will be passed
-    # to MarcExtractor.parse_string_spec, or a Hash that's
-    # already been created by it.
+    #  * a string specification like "008[35]:020a:245abc", see top of class
+    #    for examples. A string specification is most typical argument. 
+    #  * The output of a previous call to MarcExtractor.parse_string_spec(string_spec),
+    #    a 'pre-parsed' specification. 
     #
-    # options:
+    # Second arg is options:
     #
     # [:separator]  default ' ' (space), what to use to separate
     #               subfield values when joining strings
@@ -108,57 +179,30 @@ module Traject
 
     # Check to see if a tag is interesting (meaning it may be covered by a spec
     # and the passed-in options about alternate scripts)
-
     def interesting_tag?(tag)
       return @interesting_tags_hash.include?(tag)
     end
 
 
-    # Converts from a string marc spec like "245abc:700a" to a nested hash used internally
-    # to represent the specification.
-    #
-    # a String specification is a string (or array of strings) of form:
-    #  {tag}{|indicators|}{subfields} separated by colons
-    # tag is three chars (usually but not neccesarily numeric),
-    # indicators are optional two chars enclosed in pipe ('|') characters,
-    # subfields are optional list of chars (alphanumeric)
-    #
-    # indicator spec must be two chars, but one can be * meaning "don't care".
-    # space to mean 'blank'
-    #
-    # "245|01|abc65:345abc:700|*5|:800"
-    #
-    # Or, for control (fixed) fields (ordinarily fields 001-010), you can include a byte slice specification,
-    # but can NOT include subfield or indicator specifications. Plus can use special tag "LDR" for
-    # the marc leader. (TODO)
-    #
-    #  "008[35-37]:LDR[5]"
-    #  => bytes 35-37 inclusive of field 008, and byte 5 of the marc leader.
+    # Converts from a string marc spec like "008[35]:245abc:700a" to a hash used internally
+    # to represent the specification. See comments at head of class for
+    # documentation of string specification format. 
     #
-    # Returns a nested hash whose keys are tags and whose value is an array
-    # of hash structures indicating what indicators and subfields (or
-    # byte-offsets for control fields) are needed, e.g.
     #
-    # '245|1*|a:245ab:110:008[15-17]:008[17]' would give us
+    # == Return value
     #
-    # {
-    #   '245' => [
-    #     {:indicators => ['1', nil], :subfields=>['a']},
-    #     {:subfields => ['a', 'b']}
-    #    ]
-    #    '110' => [{}] # all subfields, indicators don't matter
-    #    '008' => [
-    #      {:bytes => (15..17)}
-    #      {:bytes => 17}
-    #    ]
-    # }
+    # The hash returned is keyed by tag, and has as values an array of 0 or
+    # or more MarcExtractor::Spec objects representing the specified extraction
+    # operations for that tag. 
     #
-    # * subfields and indicators can only be provided for marc data/variable fields
-    # * byte slice can only be provided for marc control fields (generally tags less than 010)
+    # It's an array of possibly more than one, because you can specify
+    # multiple extractions on the same tag: for instance "245a:245abc"
     #
     # See tests for more examples.
     def self.parse_string_spec(spec_string)
-      hash = {}
+      # hash defaults to []
+      hash = Hash.new {|hash,key| hash[key] = []}
+
       spec_strings = spec_string.is_a?(Array) ? spec_string.map{|s| s.split(/\s*:\s*/)}.flatten : spec_string.split(/s*:\s*/)
 
       spec_strings.each do |part|
@@ -166,31 +210,32 @@ module Traject
           # variable field
           tag, indicators, subfields = $1, $3, $4
 
-          hash[tag] ||= []
-          spec = {}
+          spec = Spec.new(:tag => tag)
 
           if subfields and !subfields.empty?
-            spec[:subfields] = subfields.split('')
+            spec.subfields = subfields.split('')
           end
 
           if indicators
-           spec[:indicators] = [ (indicators[0] if indicators[0] != "*"), (indicators[1] if indicators[1] != "*") ]
+           # if specified as '*', leave nil
+           spec.indicator1 = indicators[0] if indicators[0] != "*"
+           spec.indicator2 = indicators[1] if indicators[1] != "*"
           end
+
+          hash[spec.tag] << spec
           
-          hash[tag] << spec
-          
-        elsif (part =~ /\A([a-zA-Z0-9]{3})(\[(\d+)(-(\d+))?\])\Z/) # "005[4-5]"
+        elsif (part =~ /\A([a-zA-Z0-9]{3})(\[(\d+)(-(\d+))?\])\Z/) # control field, "005[4-5]"
           tag, byte1, byte2 = $1, $3, $5
-          hash[tag] ||= []
-          spec = {}
+
+          spec = Spec.new(:tag => tag)
 
           if byte1 && byte2
-            spec[:bytes] = ((byte1.to_i)..(byte2.to_i))
+            spec.bytes = ((byte1.to_i)..(byte2.to_i))
           elsif byte1
-           spec[:bytes] = byte1.to_i
+           spec.bytes = byte1.to_i
           end
           
-          hash[tag] << spec
+          hash[spec.tag] << spec
         else
           raise ArgumentError.new("Unrecognized marc extract specification: #{part}")
         end
@@ -206,7 +251,7 @@ module Traject
 
       self.each_matching_line(marc_record) do |field, spec|
         if control_field?(field)
-          results << (spec[:bytes] ? field.value.byteslice(spec[:bytes]) : field.value)
+          results << (spec.bytes ? field.value.byteslice(spec.bytes) : field.value)
         else
           results.concat collect_subfields(field, spec)
         end
@@ -217,7 +262,7 @@ module Traject
 
     # Yields a block for every line in source record that matches
     # spec. First arg to block is MARC::DataField or ControlField, second
-    # is the hash specification that it matched on. May take account
+    # is the MarcExtractor::Spec that it matched on. May take account
     # of options such as :alternate_script
     #
     # Third (optional) arg to block is self, the MarcExtractor object, useful for custom
@@ -225,19 +270,14 @@ module Traject
     def each_matching_line(marc_record)
       marc_record.fields(@interesting_tags_hash.keys).each do |field|
 
-        specs = spec_covering_field(field)
-
-        # Don't have a spec that addresses this field? Move on.
-        next unless specs
-
-        # Make sure it matches indicators too, spec_covering_field
-        # doens't check that.
-        
-        specs.each do |spec|
-          if matches_indicators(field, spec)
+        # Make sure it matches indicators too, specs_covering_field
+        # doesn't check that.
+        specs_covering_field(field).each do |spec|
+          if spec.matches_indicators?(field)
             yield(field, spec, self)
           end
         end
+
       end
     end
 
@@ -245,6 +285,8 @@ module Traject
     # but collects results of block into an array -- flattens any subarrays for you!
     #
     # Useful for re-use of this class for custom processing
+    #
+    # yields the MARC Field, the MarcExtractor::Spec object, the MarcExtractor object. 
     def collect_matching_lines(marc_record)
       results = []
       self.each_matching_line(marc_record) do |field, spec, extractor|
@@ -254,31 +296,36 @@ module Traject
     end
 
 
-    # Pass in a marc data field and a hash spec, returns
-    # an ARRAY of one or more strings, subfields extracted
+    # Pass in a marc data field and a Spec object with extraction
+    # instructions, returns an ARRAY of one or more strings, subfields extracted
     # and processed per spec. Takes account of options such
     # as :separator
     #
     # Always returns array, sometimes empty array.
     def collect_subfields(field, spec)
       subfields = field.subfields.collect do |subfield|
-        subfield.value if spec[:subfields].nil? || spec[:subfields].include?(subfield.code)
+        subfield.value if spec.includes_subfield_code?(subfield.code)
       end.compact
 
       return subfields if subfields.empty? # empty array, just return it.
 
-      return options[:separator] ? [ subfields.join( options[:separator]) ] : subfields
+      if options[:separator] && spec.joinable?
+        subfields = [subfields.join(options[:separator])]
+      end
+      
+      return subfields
     end
 
 
-    # Find a spec, if any, covering extraction from this field
+
+    # Find Spec objects, if any, covering extraction from this field.
+    # Returns an array of 0 or more MarcExtractor::Spec objects
     #
     # When given an 880, will return the spec (if any) for the linked tag iff
     # we have a $6 and we want the alternate script.
     #
-    # Returns nil if no matching spec is found
-
-    def spec_covering_field(field)
+    # Returns an empty array in case of no matching extraction specs. 
+    def specs_covering_field(field)
       tag = field.tag
 
       # Short-circuit the unintersting stuff
@@ -301,13 +348,60 @@ module Traject
       # define #control_field? on both ControlField and DataField?
       return field.kind_of? MARC::ControlField
     end
+    
 
-    # a marc field, and an individual spec hash, {:subfields => array, :indicators => array}
-    def matches_indicators(field, spec)
-      return true if spec[:indicators].nil?
+    # Represents a single specification for extracting data
+    # from a marc field, like "600abc" or "600|1*|x". 
+    #
+    # Includes the tag for reference, although this is redundant and not actually used
+    # in logic, since the tag is also implicit in the overall spec_hash 
+    # with tag => [spec1, spec2]
+    class Spec
+      attr_accessor :tag, :subfields, :indicator1, :indicator2, :bytes
+
+      def initialize(hash = {})
+        hash.each_pair do |key, value|
+          self.send("#{key}=", value)
+        end
+      end
+
+    
+      #  Should subfields extracted by joined, if we have a seperator?
+      #  * '630' no subfields specified => join all subfields
+      #  * '630abc' multiple subfields specified = join all subfields
+      #  * '633a' one subfield => do not join, return one value for each $a in the field
+      #  * '633aa' one subfield, doubled => do join after all, will return a single string joining all the values of all the $a's.
+      #
+      # Last case is handled implicitly at the moment when subfields == ['a', 'a']
+      def joinable?
+        (self.subfields.nil? || self.subfields.size != 1)
+      end
+
+      # Pass in a MARC field, do it's indicators match indicators
+      # in this spec? nil indicators in spec mean we don't care, everything
+      # matches. 
+      def matches_indicators?(field)      
+        return (self.indicator1.nil? || self.indicator1 == field.indicator1) &&
+          (self.indicator2.nil? || self.indicator2 == field.indicator2)
+      end
 
-      return (spec[:indicators][0].nil? || spec[:indicators][0] == field.indicator1) &&
-        (spec[:indicators][1].nil? || spec[:indicators][1] == field.indicator2)
+      # Pass in a string subfield code like 'a'; does this
+      # spec include it?
+      def includes_subfield_code?(code)
+        # subfields nil means include them all
+        self.subfields.nil? || self.subfields.include?(code)
+      end
+
+      def ==(spec)
+        return false unless spec.kind_of?(Spec)
+
+        return (self.tag == spec.tag) &&
+          (self.subfields == spec.subfields) && 
+          (self.indicator1 == spec.indicator1) &&
+          (self.indicator1 == spec.indicator2) &&
+          (self.bytes == spec.bytes)
+      end
     end
+
   end
 end

data/lib/traject/translation_map.rb

@@ -7,7 +7,7 @@ module Traject
   # A TranslationMap is basically just something that has a hash-like #[]
   # method to map from input strings to output strings:
   #
-  #   translation_map["some_input"] #=> some_output
+  #    translation_map["some_input"] #=> some_output
   #
   # Input is assumed to always be string, output is either string
   # or array of strings.
@@ -17,10 +17,10 @@ module Traject
   # yaml, or java .properties.  (Limited basic .properties, don't try any fancy escaping please,
   # no = or : in key names, no split lines.)
   #
-  # TranslationMap.new("dir/some_file")
+  #     TranslationMap.new("dir/some_file")
   #
-  # Will look through the entire ruby $LOAD_PATH, for a translation_maps subdir
-  # that contains either some_file.rb OR  some_file.yaml OR some_file.properties. 
+  # Will look for a file named `some_file.rb` or `some_file.yaml` or `some_file.properties`, 
+  # somewhere in the ruby $LOAD_PATH in a `/translation_maps` subdir. 
   # * Looks for "/translation_maps" subdir in load paths, so
   #   for instance you can have a gem that keeps translation maps
   #   in ./lib/translation_maps, and it Just Works. 
@@ -47,12 +47,12 @@ module Traject
   # Or, when calling TranslationMap.new(), you can pass in options over-riding special
   # key too:
   #
-  #  TranslationMap.new("something", :default => "foo")
-  #  TranslationMap.new("something", :default => :passthrough)
+  #    TranslationMap.new("something", :default => "foo")
+  #    TranslationMap.new("something", :default => :passthrough)
   #
   # == Output: String or array of strings
   #
-  # The output can be a string or an array of strings, or nil.  It should not be anything
+  # The output can be a string or an array of strings, or nil.  It should not be anything else.
   # When used with the #translate_array! method, one string can be replaced by multiple values
   # (array of strings) or removed (nil)
   #

data/lib/traject/version.rb

@@ -1,3 +1,3 @@
 module Traject
-  VERSION = "0.15.0"
+  VERSION = "0.16.0"
 end

data/test/indexer/macros_marc21_semantics_test.rb

@@ -36,7 +36,8 @@ describe "Traject::Macros::Marc21Semantics" do
     end
     output = @indexer.map_record(@record)
 
-    assert_equal ["Big bands."], output["series_facet"]
+    # trims punctuation too
+    assert_equal ["Big bands"], output["series_facet"]
   end
 
   describe "marc_sortable_author" do

data/test/indexer/to_field_test.rb

@@ -51,6 +51,27 @@ describe "Traject::Indexer.to_field" do
       flunk("Should only fail with a NamingError")
     end
   end
-
   
-end
+  # Just verifying this is how it works
+  it "doesn't allow you to just wholesale assignment to the accumulator" do
+    @indexer.to_field('foo') do |rec, acc|
+      acc = ['hello']
+    end
+    output = @indexer.map_record('never looked at')
+    assert_equal nil, output['foo']
+  end
+  
+  it "allows use of accumulator.replace" do
+    @indexer.to_field('foo') do |rec, acc|
+      acc.replace ['hello']
+    end
+    output = @indexer.map_record('never looked at')
+    assert_equal ['hello'], output['foo']
+  end
+  
+  
+end
+
+
+
+

data/test/marc_extractor_test.rb

@@ -13,15 +13,12 @@ describe "Traject::MarcExtractor" do
       assert_kind_of Hash, parsed
       assert_equal 1, parsed.keys.length
       spec = parsed['245'].first
-      assert_kind_of Hash, spec
-
-      assert_kind_of Array, spec[:indicators]
-      assert_equal 2, spec[:indicators].length
-      assert_equal "1", spec[:indicators][0]
-      assert_nil spec[:indicators][1]
-
-      assert_kind_of Array, spec[:subfields]
+      assert_kind_of Traject::MarcExtractor::Spec, spec
 
+      assert_equal "1", spec.indicator1
+      assert_nil spec.indicator2
+      
+      assert_kind_of Array, spec.subfields
     end
 
     it "parses a mixed bag" do
@@ -34,25 +31,28 @@ describe "Traject::MarcExtractor" do
 
       #245abcde
       assert spec245
-      assert_nil spec245[:indicators]
-      assert_equal %w{a b c d e}, spec245[:subfields]
+      assert_nil spec245.indicator1
+      assert_nil spec245.indicator2
+      assert_equal %w{a b c d e}, spec245.subfields
 
       #810
       assert spec810
-      assert_nil spec810[:indicators]
-      assert_nil spec810[:subfields], "No subfields"
+      assert_nil spec810.indicator1
+      assert_nil spec810.indicator2
+      assert_nil spec810.subfields, "No subfields"
 
       #700-*4bcd
       assert spec700
-      assert_equal [nil, "4"], spec700[:indicators]
-      assert_equal %w{b c d}, spec700[:subfields]
+      assert_nil spec700.indicator1
+      assert_equal "4", spec700.indicator2      
+      assert_equal %w{b c d}, spec700.subfields
     end
 
     it "parses fixed field byte offsets" do
       parsed = Traject::MarcExtractor.parse_string_spec("005[5]:008[7-10]")
 
-      assert_equal 5, parsed["005"].first[:bytes]
-      assert_equal 7..10, parsed["008"].first[:bytes]
+      assert_equal 5, parsed["005"].first.bytes
+      assert_equal 7..10, parsed["008"].first.bytes
     end
     
     it "allows arrays of specs" do
@@ -79,7 +79,7 @@ describe "Traject::MarcExtractor" do
 
   # Mostly an internal method, not neccesarily API, but
   # an important one, so we unit test some parts of it.
-  describe "#spec_covering_field" do
+  describe "#specs_covering_field" do
     describe "for alternate script tags" do
       before do
         @record = MARC::Reader.new(support_file_path  "hebrew880s.marc").to_a.first
@@ -102,17 +102,17 @@ describe "Traject::MarcExtractor" do
         assert ! @a880_100.nil?, "Found an 880-100 to test"
       end
       it "finds spec for relevant 880" do
-        assert_equal( [{}], @extractor.spec_covering_field(@a880_245) )
-        assert_nil        @extractor.spec_covering_field(@a880_100)
+        assert_equal( [Traject::MarcExtractor::Spec.new(:tag => "245")], @extractor.specs_covering_field(@a880_245) )
+        assert_equal [],   @extractor.specs_covering_field(@a880_100)
       end
       it "does not find spec for 880 if disabled" do
         @extractor = Traject::MarcExtractor.new("245", :alternate_script => false)
-        assert_nil @extractor.spec_covering_field(@a880_245) 
+        assert_nil @extractor.specs_covering_field(@a880_245) 
       end
       it "finds only 880 if so configured" do
         @extractor = Traject::MarcExtractor.new("245", :alternate_script => :only)
-        assert_nil @extractor.spec_covering_field(@a245) 
-        assert_equal([{}],  @extractor.spec_covering_field(@a880_245))
+        assert_nil @extractor.specs_covering_field(@a245) 
+        assert_equal([Traject::MarcExtractor::Spec.new(:tag => "245")],  @extractor.specs_covering_field(@a880_245))
       end
     end
   end
@@ -260,7 +260,7 @@ describe "Traject::MarcExtractor" do
       @extractor.each_matching_line(@record) do |field, spec|
         called = true
         assert_kind_of MARC::DataField, field
-        assert_kind_of Hash, spec
+        assert_kind_of Traject::MarcExtractor::Spec, spec
       end
       assert called, "calls block"
     end
@@ -269,7 +269,7 @@ describe "Traject::MarcExtractor" do
       @extractor.each_matching_line(@record) do |field, spec, extractor|
         called = true
         assert_kind_of MARC::DataField, field
-        assert_kind_of Hash, spec
+        assert_kind_of Traject::MarcExtractor::Spec, spec
         assert_kind_of Traject::MarcExtractor, extractor
         assert_same @extractor, extractor
       end
@@ -292,9 +292,11 @@ describe "Traject::MarcExtractor" do
 
   describe "MarcExtractor.cached" do
     it "creates" do
-      ext = Traject::MarcExtractor.cached("245abc", :separator => nil)
-      assert_equal({"245"=>[{:subfields=>["a", "b", "c"]}]}, ext.spec_hash)
-      assert ext.options[:separator].nil?, "extractor options[:separator] is nil"
+      extractor = Traject::MarcExtractor.cached("245abc", :separator => nil)
+      spec_hash = extractor.spec_hash
+      
+      assert extractor.options[:separator].nil?, "extractor options[:separator] is nil"
+      assert_equal({"245"=>[Traject::MarcExtractor::Spec.new(:tag => "245", :subfields=>["a", "b", "c"])]}, spec_hash)
     end
     it "caches" do
       ext1 = Traject::MarcExtractor.cached("245abc", :separator => nil)
@@ -326,11 +328,45 @@ describe "Traject::MarcExtractor" do
       
       
     
-    it "works the same as ::separator=>nil" do
-      ex1 = Traject::MarcExtractor.new("245a:245b")
-      ex2 = Traject::MarcExtractor.new("245ab", :separator=>nil)
-      assert_equal ex1.extract(@record), ex2.extract(@record)
+    it "provides multiple values for repeated subfields with single specified subfield" do
+      ex = Traject::MarcExtractor.new("245a")
+      f = @record.fields('245').first
+      title_a = f['a']
+      f.append(MARC::Subfield.new('a', title_a))
+      results = ex.extract(@record)
+      assert_equal [title_a, title_a], results
+    end
+
+    it "concats single subfield spec when given as eg 245aa" do
+      ex = Traject::MarcExtractor.new("245aa")
+      f = @record.fields('245').first
+      title_a = f['a']
+      f.append(MARC::Subfield.new('a', title_a))
+      results = ex.extract(@record)
+      assert_equal ["#{title_a} #{title_a}"], results
+    end
+    
+    it "provides single value for repeated subfields with multiple specified subfields" do
+      ex = Traject::MarcExtractor.new("245ab")
+      f = @record.fields('245').first
+      title_a = f['a']
+      title_b = f['b']
+      f.append(MARC::Subfield.new('a', title_a))
+      results = ex.extract(@record)
+      assert_equal ["#{title_a} #{title_b} #{title_a}"], results
+      
+    end
+    
+    it "provides single value for repeated subfields with no specified subfield" do
+      ex = Traject::MarcExtractor.new("245")
+      f = @record.fields('245').first
+      title_a = f['a']
+      f.append(MARC::Subfield.new('a', title_a))
+      results = ex.extract(@record)
+      assert_equal 1, results.size
     end
+    
+    
       
   
     it "allows repeated tags for a control field" do
@@ -352,6 +388,17 @@ describe "Traject::MarcExtractor" do
     end
 
   end
+
+  describe "MarcExtractor::Spec" do
+    describe "==" do
+      it "equals when equal" do
+        assert_equal Traject::MarcExtractor::Spec.new(:subfields => %w{a b c}), Traject::MarcExtractor::Spec.new(:subfields => %w{a b c}) 
+      end
+      it "does not equal when not" do
+        refute_equal Traject::MarcExtractor::Spec.new(:subfields => %w{a b c}), Traject::MarcExtractor::Spec.new(:subfields => %w{a b c}, :indicator2 => '1') 
+      end
+    end
+  end
       
 
-end
+end

metadata

@@ -2,7 +2,7 @@
 name: traject
 version: !ruby/object:Gem::Version
   prerelease:
-  version: 0.15.0
+  version: 0.16.0
 platform: ruby
 authors:
 - Jonathan Rochkind
@@ -10,7 +10,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2013-09-25 00:00:00.000000000 Z
+date: 2013-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: marc