traject 2.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7ace7c33d88c92e40e7600984d0e037435c1ecf7
+  data.tar.gz: 9b052363bbabaed5a498e1e803fa9ec6d6c4cfa6
+SHA512:
+  metadata.gz: a8f3159d608efed85d37e703e5ea2965e1cd9e67c481e181790114dc4bf33e5003116f982800b36e53ad2c4ffb1350a16eb989778decb63c5e36a56e7f150622
+  data.tar.gz: ed74374f0593b3c1b17e8808a7b030d3d5d8fe0661a7cba93a16db04c28494d0d4e5917a08fb529e51e68871c3d046af02a16e519ac571573485505d13ae5138

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+.DS_Store
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+vendor/solrj/ivy

data/.travis.yml

@@ -0,0 +1,27 @@
+language: ruby
+rvm:
+  - jruby-19mode
+  - jruby-head
+  - 1.9
+  - 2.1
+  - 2.2
+  - rbx-2
+jdk:
+  - openjdk7
+  - openjdk6
+matrix:
+  exclude:
+    - rvm: 1.9
+      jdk: openjdk7
+    - rvm: 2.1
+      jdk: openjdk7
+    - rvm: rbx-2
+      jdk: openjdk7
+    - rvm: jruby-head
+      jdk: openjdk6
+    - rvm: 2.2
+      jdk: openjdk6
+  allow_failures:
+    - rvm: jruby-head
+
+bundler_args: --without debug

data/.yardopts

@@ -0,0 +1,3 @@
+--markup markdown
+-
+doc/*.md

data/Gemfile

@@ -0,0 +1,12 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in traject.gemspec
+gemspec
+
+group :development do
+  gem "nokogiri" # used only for rake tasks load_maps:
+end
+
+group :debug do
+  gem "ruby-debug", :platform => "jruby"
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,461 @@
+# Traject
+
+An easy to use, high-performance, flexible and extensible MARC to Solr indexer. 
+
+You might use traject to index MARC data for a Solr-based discovery product like [Blacklight](https://github.com/projectblacklight/blacklight) or [VUFind](http://vufind.org/).
+
+Traject can also be generalized to a set of tools for getting structured data from a source, and transforming it to a hash-like object to send to a destination. In addition to sending data
+to solr, Traject can produce json or yaml files, tab-delimited files, CSV files, and output suitable
+for debugging by a human.
+
+**Traject is stable, mature software, that is already being used in production by its authors.**
+
+[![Gem Version](https://badge.fury.io/rb/traject.png)](http://badge.fury.io/rb/traject)
+[![Build Status](https://travis-ci.org/traject-project/traject.png)](https://travis-ci.org/traject-project/traject)
+
+
+## Background/Goals
+
+Initially by Jonathan Rochkind (Johns Hopkins Libraries) and Bill Dueber (University of Michigan Libraries).
+
+* Basic configuration files can be easily written even by non-rubyists,  with a few simple directives traject provides. But config files are 'ruby all the way down', so we can provide a gradual slope to more complex needs, with the full power of ruby.
+* Easy to program, easy to read, easy to modify.
+* Fast. Traject by default indexes using multiple threads, on multiple cpu cores, when the underlying
+ruby implementation (i.e., JRuby) allows it, and can use a separate thread for communication with
+solr even under MRI.
+* Composed of decoupled components, for flexibility and extensibility. 
+* Designed to support local code and configuration that's maintainable and testable, and can be shared between projects as ruby gems.
+* Easy to split configuration between multiple files, for simple "pick-and-choose" command line options
+that can combine to deal with any of your local needs.
+
+
+## Installation
+
+Traject runs under jruby (1.7.x or higher), MRI ruby (1.9.3 or higher), or probably any other ruby platform.
+
+**Traject runs much faster on JRuby** where it can use multi-core parallelism, and the Java
+Marc4J marc reader. If performance is a concern, you should run traject on JRuby. 
+
+Some options for installing a ruby other than your system-provided one are [chruby](https://github.com/postmodern/chruby) and [ruby-install](https://github.com/postmodern/ruby-install#readme).
+
+Once you have ruby, just `$ gem install traject`.
+
+( **Note**: We might in the future provide an all-in-one .jar distribution, which does not require you to install jruby  on your system, for those who want the multi-threading of jruby without having to actually install it. Let us know if interested.).
+
+
+## Configuration files
+
+traject is configured using configuration files. To get a sense of what they look like, you can
+take a look at our sample basic configuration file,
+[demo_config.rb](./test/test_support/demo_config.rb). You could run traject with that configuration file
+as: `traject -c path/to/demo_config.rb marc_file.marc`.
+
+Configuration files are actually just ruby -- so by convention they end in `.rb`.
+
+We hope you can write basic useful configuration files without much ruby experience, since
+traject gives you some easy functions to use for common directives. But the full power
+of ruby is available to you if needed.
+
+**rubyist tip**: Technically, config files are executed with `instance_eval` in a Traject::Indexer instance, so the special commands you see are just methods on Traject::Indexer (or mixed into it). But you can
+call ordinary ruby `require` in config files, etc., too, to load
+external functionality. See more at Extending Logic below.
+
+You can keep your settings and indexing rules in one config file,
+or split them accross multiple config files however you like. (Connection details vs indexing? Common things vs environmental specific things?)
+
+There are two main categories of directives in your configuration files: _Settings_, and _Indexing Rules_.
+
+## Settings
+
+Settings are a flat list of key/value pairs, where the keys are always strings and the values usually are. They look like this
+in a config file:
+
+~~~ruby
+# configuration_file.rb
+# Note that "#" is a comment, cause it's just ruby
+
+settings do
+  # Where to find solr server to write to
+  provide "solr.url", "http://example.org/solr"
+
+  # solr.version doesn't currently do anything, but set it
+  # anyway, in the future it will warn you if you have settings
+  # that may not work with your version.
+  provide "solr.version", "4.3.0"
+
+  # default source type is binary, traject can't guess
+  # you have to tell it.
+  provide "marc_source.type", "xml"
+
+  # various others...
+  provide "solr_writer.commit_on_close", "true"
+
+  # The default writer is the Traject::SolrJsonWriter. The default
+  # reader is Marc4JReader (using Java Marc4J library) on Jruby,
+  # MarcReader (using ruby-marc) otherwise.
+end
+~~~
+
+`provide` will only set the key if it was previously unset, so first
+setting wins, and command-line comes first of all and overrides everything.
+You can also use `store` if you want to force-set, last set wins.
+
+See, docs page on [Settings](./doc/settings.md) for list
+of all standardized settings.
+
+
+## Indexing rules: Let's start with 'to_field' and 'extract_marc'
+
+There are a few methods that can be used to create indexing rules, but the
+one you'll most common is called `to_field`, and establishes a rule
+to extract content to a particular named output field.
+
+A `to_field` extraction rule can use built-in 'macros', or, as we'll see later,
+entirely custom logic.
+
+The built-in macro you'll use the most is `extract_marc`, to extract
+data out of a MARC record according to a tag/subfield specification.
+
+~~~ruby
+    # Take the value of the first 001 field, and put
+    # it in output field 'id', to be indexed in Solr
+    # field 'id'
+    to_field "id", extract_marc("001", :first => true)
+
+    # 245 subfields a, p, and s. 130, all subfields.
+    # built-in punctuation trimming routine.
+    to_field "title_t", extract_marc("245nps:130", :trim_punctuation => true)
+
+    # Can limit to certain indicators with || chars.
+    # "*" is a wildcard in indicator spec.  So this is
+    # 856 with first indicator '0', subfield u.
+    to_field "email_addresses", extract_marc("856|0*|u")
+
+    # Can list tag twice with different field combinations
+    # to extract separately
+    to_field "isbn", extract_marc("245a:245abcde")
+
+    # For MARC Control ('fixed') fields, you can optionally
+    # use square brackets to take a byte offset.
+    to_field "langauge_code", extract_marc("008[35-37]")
+~~~
+
+`extract_marc` by default includes all 'alternate script' linked fields correspoinding
+to matched specifications, but you can turn that off, or extract *only* corresponding
+880s.
+
+~~~ruby
+    to_field "title", extract_marc("245abc", :alternate_script => false)
+    to_field "title_vernacular", extract_marc("245abc", :alternate_script => :only)
+~~~
+
+By default, specifications with multiple subfields (like "240abc") will produce one single string of output per field (for each '240'), with the concatenation of each matched subfield. Specifications with single subfields (like "020a") will split subfields and produce an output string for each matching subfield.
+
+For the syntax and complete possibilities of the specification
+string argument to extract_marc, see docs at the [MarcExtractor class](./lib/traject/marc_extractor.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/MarcExtractor)).
+
+`extract_marc` also supports `translation maps` similar
+to SolrMarc's. There are some translation maps provided by traject,
+and you can also define your own, in yaml or ruby. Translation maps are especially useful
+for mapping form MARC codes to user-displayable strings:
+
+~~~ruby
+    # "translation_map" will be passed to Traject::TranslationMap.new
+    # and the created map used to translate all values
+    to_field "language", extract_marc("008[35-37]:041a:041d", :translation_map => "marc_language_code")
+~~~
+
+To see all options for `extract_marc`, see the [method documentation](http://rdoc.info/gems/traject/Traject/Macros/Marc21:extract_marc)
+
+## other built-in utility macros
+
+Other built-in methods that can be used with `to_field` include a hard-coded
+literal string:
+
+~~~ruby
+    to_field "source", literal("LIB_CATALOG")
+~~~
+
+The current record serialized back out as MARC, in binary, XML, or json:
+
+~~~ruby
+    # or :format => "json" for marc-in-json
+    # or :format => "binary", by default Base64-encoded for Solr
+    # 'binary' field, or, for more like what SolrMarc did, without
+    # escaping:
+    to_field "marc_record_raw", serialized_marc(:format => "binary", :binary_escape => false, :allow_oversized => true)
+~~~
+
+Text of all fields in a range:
+
+~~~ruby
+    to_field "text", extract_all_marc_values(:from => 100, :to => 899)
+~~~
+
+All of these methods are defined at [Traject::Macros::Marc21](./lib/traject/macros/marc21.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/Macros/Marc21))
+
+## more complex canned MARC semantic logic
+
+Some more complex (and opinionated/subjective) algorithms for deriving semantics
+from Marc are also packaged with Traject, but not available by default. To make
+them available to your indexing, you just need to use ruby `require` and `extend`.
+
+A number of methods are in [Traject::Macros::Marc21Semantics](./lib/traject/macros/marc21_semantics.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/Macros/Marc21Semantics))
+
+~~~ruby
+    require 'traject/macros/marc21_semantics'
+    extend Traject::Macros::Marc21Semantics
+
+    to_field 'title_sort',        marc_sortable_title
+    to_field 'broad_subject',     marc_lcc_to_broad_category
+    to_field "geographic_facet",  marc_geo_facet
+    # And several more
+~~~
+
+And, there's a routine for classifying MARC to an internal
+format/genre/type vocabulary:
+
+~~~ruby
+    require 'traject/macros/marc_format_classifier'
+    extend Traject::Macros::MarcFormats
+
+    to_field 'format_facet',    marc_formats
+~~~
+
+(Alternately, see the [traject_umich_format](https://github.com/billdueber/traject_umich_format) gem for the often-ridiculously-complex
+logic used at the University of Michigan.)
+
+## Custom logic
+
+The built-in routines are there for your convenience, but if you need
+something local or custom, you can write ruby logic directly
+in a configuration file, using a ruby block, which looks like this:
+
+~~~ruby
+    to_field "id" do |record, accumulator|
+       # take the record's 001, prefix it with "bib_",
+       # and then add it to the 'accumulator' argument,
+       # to send it to the specified output field
+       value = record['001']
+       value = "bib_#{value}"
+       accumulator << value
+    end
+~~~
+
+`do |record, accumulator| ... ` is the definition of a ruby block taking
+two arguments.  The first one passed in will be a MARC record. The
+second is an array, you add values to the array to send them to
+output.
+
+Here's another example that shows how you'd get the
+record type byte 06 out of a MARC leader, then translate it
+to a human-readable string with a TranslationMap
+
+~~~ruby
+    to_field "marc_type" do |record, accumulator|
+      leader06 = record.leader.byteslice(6)
+      # this translation map doesn't actually exist, but could
+      accumulator << TranslationMap.new("marc_leader")[ leader06 ]
+    end
+~~~
+
+You can also add a block onto the end of a built-in 'macro', to
+further customize the output. The `accumulator` passed to your block
+will already have values in it from the first step, and you can
+use ruby methods like `map!` to modify it:
+
+~~~ruby
+    to_field "big_title", extract_marc("245abcdefg") do |record, accumulator|
+      # put it all in all uppercase, I don't know why.
+      accumulator.map! {|v| v.upcase}
+    end
+~~~
+
+If you find yourself repeating boilerplate code in your custom logic, you can
+even create your own 'macros' (like `extract_marc`). `extract_marc` and other
+macros are nothing more than methods that return ruby lambda objects of
+the same format as the blocks you write for custom logic.
+
+For tips, gotchas, and a more complete explanation of how this works, see
+additional documentation page on [Indexing Rules: Macros and Custom Logic](./doc/indexing_rules.md)
+
+## each_record and after_processing
+
+In addition to `to_field`, an `each_record` method is available, which,
+like `to_field`, is executed for every record, but without being tied
+to a specific field.
+
+`each_record` can be used for logging or notifiying; computing intermediate
+results; or writing to more than one field at once.
+
+~~~ruby
+  each_record do |record|
+    some_custom_logging(record)
+  end
+~~~
+
+For more on `each_record`, see documentation page on [Indexing Rules: Macros and Custom Logic](./doc/indexing_rules.md).
+
+There is also an `after_processing` method that can be used to register
+logic that will be called after the entire has been processed. You can use it for whatever custom
+ruby code you might want for your app (send an email? Clean up a log file? Trigger
+a Solr replication?)
+
+~~~ruby
+after_processing do
+  whatever_ruby_code
+end
+~~~
+
+
+## Readers and Writers
+
+Traject uses modular 'Writer' classes to take the output hashes from transformation, and
+send them somewhere or do something useful with them.
+
+By default traject uses the [Traject::SolrJsonWriter](lib/traject/solr_json_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/SolrJsonWriter)) to send to Solr for indexing.
+Several other writers are also built-in:
+* [Traject::DebugWriter](lib/traject/debug_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/DebugWriter))
+* [Traject::JsonWriter](lib/traject/json_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/JsonWriter))
+* [Traject::YamlWriter](lib/traject/yaml_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/YamlWriter))
+* [Traject::DelimitedWriter](lib/traject/delimited_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/DelimitedWriter))
+* [Traject::CSVWriter](lib/traject/csv_writer.rb) ([rdoc](http://rdoc.info/gems/traject/Traject/CSVWriter))
+
+You set which writer is being used in settings (`provide "writer_class_name", "Traject::DebugWriter"`),
+or with the shortcut command line argument  `-w Traject::DebugWriter`.
+
+The [SolrJWriter](https://github.com/traject-project/traject-solrj_writer) is packaged separately,
+and will be useful if you need to index to Solr's older than version 3.2. It requires Jruby.  
+
+You can easily write your own Readers and Writers if you'd like, see comments at top
+
+of [Traject::Indexer](lib/traject/indexer.rb).
+
+## The traject command Line
+
+The simplest invocation is:
+
+    traject -c conf_file.rb marc_file.mrc
+
+Traject assumes marc files are in ISO 2709 MARC 'binary' format; it is not
+currently able to guess other marc format types like XML from filenames or content. If you are reading
+marc files in another format, you need to tell traject either with the `marc_source.type` or the command-line shortcut:
+
+    traject -c conf.rb -t xml marc_file.xml
+
+You can supply more than one conf file to traject with repeated `-c` arguments.
+
+    traject -c connection_conf.rb -c indexing_conf.rb marc_file.mrc
+
+If you supply a `--stdin` argument, traject will try to read from stdin.
+You can only supply one marc file at a time, but we can take advantage of stdin to get around this:
+
+    cat some/dir/*.marc | traject -c conf_file.rb --stdin
+
+You can set any setting on the command line with `-s key=value`.
+This will over-ride any settings set with `provide` in conf files.
+
+    traject -c conf_file.rb marc_file -s solr.url=http://somehere/solr -s solrj_writer.commit_on_close=true
+
+There are some built-in command-line option shortcuts for useful
+settings:
+
+Use `--debug-mode` to output in a human-readable format, instead of sending to solr.
+Also turns on debug logging and restricts processing to single-threaded. Useful for
+debugging or sanity checking.
+
+    traject --debug-mode -c conf_file.rb marc_file
+
+Use `-u` as a shortcut for `s solr.url=X`
+
+    traject -c conf_file.rb -u http://example.com/solr marc_file.mrc
+
+Run `traject -h` to see the command line help screen listing all available options.
+
+Also see `-I load_path` option and suggestions for Bundler use under Extending With Your Own Code.
+
+See also [Hints for batch and cronjob use](./doc/batch_execution.md) of traject.
+
+
+## Extending With Your Own Code
+
+Traject config files are full live ruby files, where you can do anything,
+including declaring new classes, etc.
+
+However, beyond limited trivial logic, you'll want to organize your
+code reasonably into separate files, not jam everything into config
+files.
+
+Traject wants to make sure it makes it convenient for you to do so,
+whether project-specific logic in files local to the traject project,
+or in ruby gems that can be shared between projects.
+
+There are standard ruby mechanisms you can use to do this, and
+traject provides a couple features to make sure this remains
+convenient with the traject command line.
+
+For more information, see documentation page on [Extending With Your
+Own Code](./doc/extending.md)
+
+**Expert summary** :
+* Traject `-I` argument command line can be used to list directories to
+  add to the load path, similar to the `ruby -I` argument. You
+  can then 'require' local project files from the load path.
+  * translation map files found on the load path or in a
+    "./translation_maps" subdir on the load path will be found
+    for Traject translation maps.
+* Use [Bundler](http://bundler.io/) with traject simply by creating a Gemfile with `bundler init`,
+  and then running command line with `bundle exec traject` or
+  even `BUNDLE_GEMFILE=path/to/Gemfile bundle exec traject`
+
+## More
+
+* [Other traject commands](./doc/other_commands.md) including `marcout`, and `commit`
+* [Hints for batch and cronjob use](./doc/batch_execution.md) of  traject.
+* Plugin extensions: Gems that add functionality to traject
+  * [traject_alephsequential_reader](https://github.com/traject-project/traject_alephsequential_reader/): read MARC files serialized in the AlephSequential format, as output by Ex Libris's Alpeh ILS.
+  * [traject_horizon](https://github.com/jrochkind/traject_horizon): Export MARC records directly from a Horizon ILS rdbms, as serialized MARC or to  index into Solr.
+  * [traject_umich_format](https://github.com/billdueber/traject_umich_format/): opinionated code and associated macros to extract format (book, audio file, etc.) and types (bibliography, conference report, etc.) from a MARC record. Code mirrors that used by the University of Michigan, and is an alternate approach to that taken by the `marc_formats` macro in `Traject::Macros::MarcFormatClassifier`.
+  * [traject-solrj_writer](https://github.com/traject-project/traject-solrj_writer): a jruby-only writer that uses the solrj .jar to talk directly to solr. Your only option for speaking to a solr version < 3.2, which is when the json handler was added to solr.
+  * [traject_marc4j_reader](https://github.com/billdueber/traject_marc4j_reader): Packaged with traject automatically on jruby. A JRuby-only reader for
+  reading marc records using the Marc4J library, fastest MARC reading on JRuby. 
+
+# Development
+
+Run tests with `rake test` or just `rake`.  Tests are written using Minitest (please, no rspec).  We use the spec-style describe/it to
+list the tests -- but generally prefer unit-style "assert_*" methods
+to make actual assertions, for clarity.
+
+To make a pull request, please make a feature branch *created from the master branch*, not from an existing feature branch. (If you need to do a feature branch dependent on an existing not-yet merged feature branch... discuss
+this with other developers first!)
+
+Pull requests should come with tests, as well as docs where applicable. Docs can be inline rdoc-style, edits to this README,
+and/or extra files in ./docs -- as appropriate for what needs to be docs.
+
+**Inline api docs** Note that our [`.yardopts` file](./.yardopts) used by rdoc.info to generate
+online api docs has a `--markup markdown` specified -- inline class/method docs are in markdown, not rdoc.
+
+Bundler rake tasks included for gem releases: `rake release`
+* Every traject release needs to be done once when running MRI, and switch to JRuby
+and do the same release again. The JRuby release is identical but for including
+a gemspec dependency on the Marc4JReader gem. 
+
+## TODO
+
+* Readers and index rules helpers for reading XML files as input? Maybe. 
+
+* Writers for writing to stores other than Solr? ElasticSearch? Maybe. 
+
+* Unicode normalization. Has to normalize to NFKC on way out to index. Except for serialized marc field and other exceptions? Except maybe don't have to, rely on solr analyzer to do it?
+
+  * Should it normalize to NFC on the way in, to make sure translation maps and other string comparisons match properly?
+
+  * Either way, all optional/configurable of course. based
+    on Settings.
+
+* CommandLine class isn't covered by tests -- it's written using functionality
+from Indexer and other classes taht are well-covered, but the CommandLine itself
+probably needs some tests -- especially covering error handling, which probably
+needs a bit more attention and using exceptions instead of exits, etc.
+
+* Optional built-in jetty stop/start to allow indexing to Solr that wasn't running before. maybe https://github.com/projecthydra/jettywrapper ?