high_level_browse 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 286d0ce64d0d9e8dffa58b716f111d086310654d
+  data.tar.gz: 2a13aad07ee29e47b0bcc00f4ba16740491e9bfd
+SHA512:
+  metadata.gz: 9960852abc0686da303da11c8ead326df0ea7e7df89432962f7d1353e62350afbc7a3ad556d1beecfe6cce816c1bf654ce4bdee78bb195caefdb08caeb67b7cf
+  data.tar.gz: 3d29b51feb0bd70c37eea28248eff4f3dccd8a38cdb23617be8998fdfa821e392743d0f87c25e9f233ee326b4097072ca63d3d04bcdf5c21216ec43a96ecae04

data/.gitignore

@@ -0,0 +1,22 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.travis.yml

@@ -0,0 +1,3 @@
+language: ruby
+rvm:
+  - 1.9.3

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in high_level_browse.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Bill Dueber
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,198 @@
+# HighLevelBrowse
+
+Given an LC Call Number, try to get a set of academic disciplines associated with it
+
+## Usage
+
+```ruby
+
+use 'high_level_browse'
+
+# Pull a new version of the raw data from the UM website,
+# transform it into something that can be quickly searched,
+# and serialize it to `hlb.json.gz` in the specified directory
+hlb = HighLevelBrowse.fetch_and_save(dir: '/tmp')
+
+# ...or just grab an already fetch_and_saved copy
+hlb = HighLevelBrowse.load(dir: '/tmp')
+
+# What HLB categories is an LC Call Number in?
+hlb.topics 'hc 9112.2'
+# => [["Social Sciences", "Economics"],
+#     ["Social Sciences", "Social Sciences (General)"]]
+
+# ... or use the #[] shortcut syntax
+
+hlb['NC1766 .U52 D733 2014']
+# => [["Arts", "Art History"],
+#    ["Arts", "Art and Design"],
+#    ["Arts", "Film and Video Studies"]]
+
+# You can also send more than one call number at a time
+
+hlb.topics('E 99 .S2 Y67 1993', 'PS 3565 .R5734 F67 2015')
+# => [["Humanities", "American Culture"],
+#     ["Humanities", "United States History"],
+#     ["Social Sciences", "Native American Studies"],
+#     ["Social Sciences", "Archaeology"],
+#     ["Humanities", "English Language and Literature"]]
+
+```
+
+
+## Overview
+
+While we in the library world sometimes use LC Call Numbers (or at least
+the initial letters) as a proxy for subject matter, the mapping is iffy
+in many cases and is, in any case, one-dimensional. Many works simply
+cover multiple subjects or are relevant to sometimes quite different
+types of academics.
+
+Take, for example, the chemistry of the brain as it applies to mental
+illness. We have a book, _Endorphins : new waves in brain chemistry_
+cataloged as **QP552.E53 D381 1984**. The QP's map to "Phsiology", which
+is correct but not complete.
+
+The University of Michigan Library has for years maintained 
+the [High Level Browse](https://www.lib.umich.edu/browse/categories/) (HLB),
+a mapping of call-number ranges to academic subjects. The entire 
+data set is available as [1.8MB XML file](https://www.lib.umich.edu/browse/categories/xml.php)
+for download.
+
+In the HLB, the call number for _Endorphins : new waves in brain chemistry_ maps
+to the following categories:
+
+* Science | Physiology
+* Health Sciences | Physiology
+* Health Sciences | Public Health (General)
+* Science | Chemical Engineering
+* Engineering | Chemical Engineering
+* Health Sciences | Biological Chemistry
+* Science | Chemistry | Biological Chemistry
+
+This opens up potentially more accurate categorization of works for, say, 
+faceting in a library catalog.
+
+This gem gives a relatively time-efficient way to get the set of disciplines associated
+with the given callnumber or callnumbers as part of indexing MARC records into Solr. 
+This mapping is used in many places in the University Library at the University of 
+Michigan, including the 
+[Mirlyn Catalog](https://mirlyn.lib.umich.edu/)
+(exposed as "Academic Discipline" in the facets) and ejournals/databases (and even 
+Librarians!) via the [Browse page](https://www.lib.umich.edu/browse). 
+ 
+This categorization may be useful for clustering/faceting
+in similar applications at other institutions. Note that the actual creation and 
+maintenance of the call number ranges is done by subject specialist librarians and 
+is out of scope for this gem.
+
+## Command line utilities: `fetch_new_hlb` and `hlb`
+
+There are also a couple command line applications for managing and querying the
+data.
+
+* **fetch_new_hlb** tries to grab a new copy of the data from the umich website
+  and serialize it to a ~500k file called `hlb.json.gz` in the given directory. 
+  Useful for putting in a cron job to periodically update with fresh data
+  
+```bash
+
+$> fetch_new_hlb
+
+fetch_new_hlb -- get a new copy of the HLB ready for use by high_level_browse
+and stick it in the given directory
+
+   Usage: fetch_new_hlb <dir>
+```
+
+* **hlb** takes one or more callnumbers and returns a text display of the categories
+  associated with them. It will stash a copy of the database in `Dir.tmpdir`if there 
+  isn't one there already, and use it on subsequent calls so things aren't so 
+  desperately slow. (To find your tmpdir, in your shell
+  run `ruby -e 'require "tmpdir"; puts Dir.tmpdir'`)
+
+
+```bash
+$> hlb
+
+hlb -- get high level browse data for an LC call number
+
+Example:
+   hlb "qa 11.33 .C4 .H3"
+    or do several at once
+   hlb "PN 33.4" "AC 1122.3 .C22" ...
+ 
+# Let's try it
+$> hlb "qa 11.33 .C4"
+   
+   Science | Mathematics
+   Social Sciences | Education   
+   
+```
+
+
+## A warning about (lack of) coverage
+
+Note that not every possible valid callnumber will be necessarily be contained in any 
+dicipline at all. Many books aren't academic in nature, and even then
+coverage is known to have some holes. Some of the ranges cover essentially a 
+single book in the umich collection. And, of course, not every record is going 
+to have a LC Call Number, so there's that.
+
+This is all to say: this may or may not be useful at your insitution. You'll 
+have to experiment.
+
+To help with this, there's a little script in the `bin/` directory called 
+`test_marc_file_for_hlb` which will, when given a MARC-XML file (ending in `.xml`)
+or a MARC-binary file (ending in anything else), output some statistics on
+what kind of coverage you would get. It might be useful to send a test file 
+through there to see what comes up. It looks in the `050` and the `852[h]` to
+see if anything pops, but you can make it looks elsewhere pretty easily.
+
+It produces something like this:
+
+```
+050 fields
+     9790 total
+      209 not recognized as LC call numbers
+     9337 with at least one HLB category
+      244 with NO category
+
+Of 17642 records,
+  9677 (54.85%) had a field that often contains an LC Call Number
+  9262 (95.71%) of *those* had at least one HLB category
+
+```
+
+## Performance
+
+On my laptop under normal load (e.g., not very scientific at all)
+I get the following running in a single thread
+
+```
+  ruby 2.3  this gem       ~8500 lookups/second
+  ruby 2.4  this gem       ~9100 lookups/second
+  jruby 9   this gem     ~20,000 lookups/second
+  jruby 9,  old HLB.jar    ~6500 lookups/second
+  jruby 1.7 this gem             error, can't do named arguments since it's 1.9 mode
+  jruby 1.7 old HLB.jar    ~6700 lookups/second
+```
+
+The [old HLB.jar](https://github.com/billdueber/HLB-Java) refers to a pure java version that I call from within
+Jruby as part of my catalog indexing process now. Ithas a different (worse) algorithm, but is of
+interest because it's what I'm writing this to replace.
+
+## Installation
+
+```bash
+    gem 'high_level_browse'
+```
+
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/high_level_browse/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+end
+
+task :default => :test
+

data/bench/bench.rb

@@ -0,0 +1,57 @@
+require 'benchmark/ips'
+$:.unshift '../lib'
+$:.unshift '.'
+
+
+# On my laptop under normal load (e.g., not very scientific at all)
+# I get the following running in a single thread
+#   ruby 2.3               ~8500 lookups/second
+#   ruby 2.4               ~9100 lookups/second
+#   jruby 9                ~20k  lookups/second
+#   jruby 9, old HLB.jar   ~6500 lookups/second
+#   jruby 1.7                    error, can't do named arguments
+#   jruby 1.7, old HLB.jar ~6700 lookups/second
+#
+# The old HLB.jar has a different (worse) algorithm, but is of
+# interest because it's what I'm writing this to replace.
+
+# umich_traject holds .jar files with the old java implementation; see
+# https://github.com/hathitrust/ht_traject/tree/9e8d414fd9bb2c79e243d289c4d39c05d2de27e5/lib/umich_traject
+#
+
+TEST_OLD_STUFF = defined? JRUBY_VERSION and Dir.exist?('./umich_traject')
+if TEST_OLD_STUFF
+  puts "Loading old HLB3.jar stuff"
+  require 'umich_traject/jackson-core-asl-1.4.3.jar'
+  require 'umich_traject/jackson-mapper-asl-1.4.3.jar'
+  require 'umich_traject/apache-solr-umichnormalizers.jar'
+  require 'umich_traject/HLB3.jar'
+  java_import Java::edu.umich.lib.hlb::HLB
+  puts "Initializing HLB"
+  HLB.initialize()
+end
+
+require 'high_level_browse'
+
+h = HighLevelBrowse.load(dir: '.')
+
+cns = File.read('call_numbers.txt').split(/\n/).cycle
+
+puts RUBY_DESCRIPTION
+
+total = 0
+Benchmark.ips do |x|
+  x.config(:time => 25, :warmup => 25)
+
+  x.report("HLB lookups") do
+    total += h[cns.next].count
+  end
+
+  if TEST_OLD_STUFF
+    total = 0
+    x.report("Old java lookups") do
+      total += HLB.categories(cns.next).to_a.count
+    end
+    x.compare!
+  end
+end

data/bin/fetch_new_hlb

@@ -0,0 +1,62 @@
+#!/usr/bin/env ruby
+
+# If we're loading from source instead of a gem, rubygems
+# isn't setting load paths for us, so we need to set it ourselves
+self_load_path = File.expand_path("../lib", File.dirname(__FILE__))
+unless $LOAD_PATH.include? self_load_path
+  $LOAD_PATH << self_load_path
+end
+
+def silence_warnings(&block)
+  warn_level = $VERBOSE
+  $VERBOSE = nil
+  result = block.call
+  $VERBOSE = warn_level
+  result
+end
+
+# minitest has a circular require warning, which
+# drives me crazy. Suppress it.
+silence_warnings do
+  require 'high_level_browse'
+end
+
+require 'fileutils'
+
+def putsmsg(msg)
+  puts "---------------------------------------------------"
+  puts " ERROR: #{msg}"
+  puts "---------------------------------------------------"
+  puts
+end
+
+def usage(msg = nil)
+  puts
+  putsmsg(msg) if msg
+  puts "fetch_new_hlb -- get a new copy of the HLB ready for use by high_level_browse"
+  puts "and stick it in the given directory"
+  puts
+  puts "   Usage: fetch_new_hlb <dir>"
+  puts
+  exit
+end
+
+unless ARGV.size == 1
+  usage
+end
+
+dir = ARGV.shift
+
+File.exist? dir or usage "#{dir} does not exist"
+Dir.exist? dir or usage "#{dir} is not a directory"
+File.writable? dir or usage "#{dir} is not writable"
+
+begin
+  db = HighLevelBrowse.fetch
+  db.save(dir: dir)
+rescue => e
+  puts "============================="
+  puts "ERROR FETCHING HLB SOURCE"
+  puts "  #{e}"
+  puts "============================="
+end

data/bin/hlb

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+
+# Hmmm. How to pass along the location? Stick one in /tmp
+# and see if it exists?
+
+
+def usage
+  puts "hlb -- get high level browse data for an LC call number"
+  puts
+  puts %Q{Example:\n   hlb "qa 11.33 .C4 .H3"}
+  puts "   or do several at once"
+  puts %Q{   hlb "PN 33.4" "AC 1122.3 .C22" ... }
+  puts
+  exit(1)
+end
+
+usage if ARGV.empty?
+
+self_load_path = File.expand_path("../lib", File.dirname(__FILE__))
+unless $LOAD_PATH.include? self_load_path
+  $LOAD_PATH << self_load_path
+end
+
+require 'high_level_browse'
+require 'fileutils'
+require 'tmpdir'
+
+filename = HighLevelBrowse::DB::FILENAME
+dir      = Dir.tmpdir()
+fullpath = File.join(dir, filename)
+
+hlb = if File.exist?(fullpath)
+        HighLevelBrowse.load(dir: dir)
+      else
+        STDERR.puts "Fetching raw data from UMich; wait a sec"
+        HighLevelBrowse.fetch_and_save(dir: dir)
+      end
+
+
+topics = hlb[*ARGV]
+
+if topics.empty?
+  puts "\nNo categories found for #{ARGV}\n\n"
+else
+  puts "\n" + topics.map { |x| x.join(' | ') }.join("\n") + "\n\n"
+end

data/bin/test_marc_file_for_hlb

@@ -0,0 +1,122 @@
+#!/usr/bin/env ruby
+
+self_load_path = File.expand_path("../lib", File.dirname(__FILE__))
+unless $LOAD_PATH.include? self_load_path
+  $LOAD_PATH << self_load_path
+end
+
+require 'marc'
+require 'high_level_browse'
+require 'lcsort'
+require 'tmpdir'
+
+
+filename = ARGV[0]
+
+reader = if filename =~ /xml\Z/i
+           MARC::XMLReader.new(filename)
+         else
+           MARC::Reader.new(filename)
+         end
+
+Counter = Struct.new(:count, :invalid, :found, :notfound, :hlb) do
+
+  def update(cn)
+    self.count += 1
+    case check_cn(cn)
+    when :invalid
+      self.invalid += 1
+      0
+    when :found
+      self.found += 1
+      1
+    when :notfound
+      self.notfound += 1
+      0
+    end
+  end
+
+
+  def check_cn(cn)
+    normalized = Lcsort.normalize(cn)
+    return :invalid if normalized.nil?
+    cats = hlb[cn]
+    if cats.empty?
+      :notfound
+    else
+      :found
+    end
+  end
+
+  def puts_pretty_output
+    puts '%9d total' % count
+    puts '%9d not recognized as LC call numbers' % invalid
+    puts '%9d with at least one HLB category' % found
+    puts '%9d with NO category' % notfound
+  end
+end
+
+
+def puts_output(f050, f852)
+  puts "050 fields"
+  f050.puts_pretty_output
+  puts "\n852h fields"
+  f852.puts_pretty_output
+end
+
+puts "Fetching/parsing HLB XML file"
+filename = HighLevelBrowse::DB::FILENAME
+dir      = Dir.tmpdir()
+fullpath = File.join(dir, filename)
+
+hlb = if File.exist?(fullpath)
+        puts "Using file at #{fullpath}"
+        HighLevelBrowse.load(dir: dir)
+      else
+        HighLevelBrowse.fetch_and_save(dir: dir)
+      end
+
+f050    = Counter.new(0, 0, 0, 0, hlb)
+f852    = Counter.new(0, 0, 0, 0, hlb)
+records = 0
+matched_records = 0
+possible_records = 0
+puts "Beginning analysis of marc records with 2k record progress reports"
+reader.each do |r|
+  records += 1
+  found = 0
+  possible = false
+  puts '%8d records processed so far' % records if records % 2_000 == 0
+  if r['050']
+    cns = r.fields('050').map { |x| x.map(&:value).join('') }
+    cns.each do |cn|
+      found += f050.update(cn)
+      possible = true
+    end
+    cns = r.fields('852').keep_if { |x| x['h'] }.map { |x| x['h'] }
+    cns.each do |cn|
+      found += f852.update(cn)
+      possible = true
+    end
+  end
+  matched_records += 1 if found > 0
+  possible_records += 1 if possible
+end
+
+puts "\n\n"
+puts_output(f050, f852)
+puts format(
+       %Q[\nOf %d records,
+  %d (%4.2f%%) had a field that often contains an LC Call Number
+  %d (%4.2f%%) of *those* had at least one HLB category],
+        records,
+        possible_records,
+        possible_records.to_f / records * 100,
+        matched_records,
+        matched_records.to_f / possible_records * 100)
+
+
+
+
+
+

data/high_level_browse.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'high_level_browse/version'
+
+Gem::Specification.new do |spec|
+  spec.name     = "high_level_browse"
+  spec.version  = HighLevelBrowse::VERSION
+  spec.authors  = ["Bill Dueber"]
+  spec.email    = ["bill@dueber.com"]
+  spec.summary  = %q{Map LC call numbers to academic categories.}
+  spec.homepage = ""
+  spec.license  = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'oga', '~> 2.1'
+  spec.add_dependency 'lcsort'
+
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "minitest"
+end

data/lib/high_level_browse.rb

@@ -0,0 +1,41 @@
+require "high_level_browse/version"
+require 'high_level_browse/db'
+require 'uri'
+require 'open-uri'
+
+module HighLevelBrowse
+
+  SOURCE_URL = ENV['HLB_XML_ENDPOINT'] || 'https://www.lib.umich.edu/browse/categories/xml.php'
+
+  # Fetch a new version of the raw file and turn it into a db
+  # @return [DB] The loaded database
+  def self.fetch
+    uri = URI.parse(SOURCE_URL)
+    # Why on earth OpenURI::OpenRead is mixed into http but not https, I don't know
+    uri.extend OpenURI::OpenRead
+
+    xml = uri.read
+    return DB.new_from_xml(xml)
+  rescue => e
+    raise "Could not fetch xml from '#{SOURCE_URL}': #{e}"
+  end
+
+
+  # Fetch and save to the specified directory
+  # @param [String] dir The directory where the hlb.json.gz file will end up
+  # @return [DB] The fetched and saved database
+  def self.fetch_and_save(dir:)
+    db = self.fetch
+    db.save(dir: dir)
+    db
+  end
+
+
+  # Load from disk
+  # @param [String] dir The directory where the hlb.json.gz file is located
+  # @return [DB] The loaded database
+  def self.load(dir:)
+    DB.load(dir: dir)
+  end
+
+end

data/lib/high_level_browse/call_number_range.rb

@@ -0,0 +1,154 @@
+require 'lcsort'
+require 'high_level_browse/range_tree'
+
+
+# An efficient set of CallNumberRanges from which to get topics
+class HighLevelBrowse::CallNumberRangeSet < HighLevelBrowse::RangeTree
+
+
+  # Returns the array of topic arrays for the given LC string
+  # @param [String] raw_lc A raw LC string (eg., 'qa 112.3 .A4 1990')
+  # @return [Array<Array<String>>] Arrays of topic labels
+  def topics_for(raw_lc)
+    normalized = Lcsort.normalize(HighLevelBrowse::CallNumberRange.preprocess(raw_lc))
+    self.search(normalized).map(&:topic_array).uniq
+  end
+end
+
+
+# A callnumber-range keeps track of the original begin/end
+# strings as well as the normalized versions, and can be
+# serialized to JSON
+
+class HighLevelBrowse::CallNumberRange
+  include Comparable
+
+  attr_reader :min, :max, :min_raw, :max_raw, :firstletter
+
+
+  attr_accessor :topic_array, :redundant
+
+  SPACE_OR_PUNCT = /\A[\s\p{Punct}]*(.*?)[\s\p{Punct}]*\Z/
+  DIGIT_TO_LETTER = /(\d)([A-Z])/i
+
+  # @nodoc
+  # Remove spaces/punctuation from the ends of the string
+  def self.strip_spaces_and_punct(str)
+    str.gsub(SPACE_OR_PUNCT, '\1')
+  end
+
+  # @nodoc
+  # Force a space between any digit->letter transition
+  def self.force_break_between_digit_and_letter(str)
+    str.gsub(DIGIT_TO_LETTER, '\1 \2')
+  end
+  # @nodoc
+  # Preprocess the string, removing spaces/punctuation off the end
+  # and forcing a space where there's a digit->letter transition
+  def self.preprocess(str)
+    str ||= ''
+    force_break_between_digit_and_letter(
+        strip_spaces_and_punct(str)
+    )
+  end
+
+
+  def initialize(min:, max:, topic_array:)
+    @illegal     = false
+    @redundant   = false
+    self.min     = self.class.preprocess(min)
+    self.max     = self.class.preprocess(max)
+    @topic_array = topic_array
+    @firstletter = self.min[0] unless @illegal
+  end
+
+
+  # Compare based on @min, then end
+  # @param [CallNumberRange] o the range to compare to
+  def <=>(o)
+    [self.min, self.max] <=> [o.min, o.max]
+  end
+
+  def to_s
+    "[#{self.min_raw} - #{self.max_raw}]"
+  end
+
+  def reconstitute(min, max, min_raw, max_raw, firstletter, topic_array)
+    @min         = min
+    @max         = max
+    @min_raw     = min_raw
+    @max_raw     = max_raw
+    @firstletter = firstletter
+    @topic_array = topic_array
+  end
+
+
+  # Two ranges are equal if their @min, @max, and topic array
+  # are all the same
+  # @param [CallNumberRange] o the range to compare to
+  def ==(other)
+    @min == other.min and
+        @max == other.max and
+        @topic_array == other.topic_array
+  end
+
+
+  # @nodoc
+  # JSON roundtrip
+  def to_json(*a)
+    {
+        'json_class' => self.class.name,
+        'data'       => [@min, @max, @min_raw, @max_raw, @firstletter, @topic_array]
+    }.to_json(*a)
+  end
+
+  # @nodoc
+  def self.json_create(h)
+    cnr = self.allocate
+    cnr.reconstitute(*(h['data']))
+    cnr
+  end
+
+
+  # In both @min= and end=, we also rescue any parsing errors
+  # and simply set the @illegal flag so we can use it later on.
+  def min=(x)
+    @min_raw     = x
+    possible_min = Lcsort.normalize(x)
+    if possible_min.nil? # didn't normalize
+      @illegal = true
+      nil
+    else
+      @min = possible_min
+    end
+  end
+
+  # Same as start. Set the illegal flag if we get an error
+  def max=(x)
+    @max_raw     = x
+    possible_max = Lcsort.normalize(x)
+    if possible_max.nil? # didn't normalize
+      @illegal = true
+      nil
+    else
+      @max = possible_max + '~' # add a tilde to make it a true endpoint
+    end
+  end
+
+  def illegal?
+    @illegal
+  end
+
+
+  def surrounds(other)
+    @min <= other.min and @max >= other.max
+  end
+
+  def contains(x)
+    @min <= x and @max >= x
+  end
+
+  alias_method :cover?, :contains
+  alias_method :member?, :contains
+
+end

data/lib/high_level_browse/db.rb

@@ -0,0 +1,150 @@
+require 'oga'
+require 'high_level_browse/call_number_range'
+require 'zlib'
+require 'json'
+
+class HighLevelBrowse::DB
+
+  # Hard-code filename. If you need more than one, put them
+  # in different directories
+  FILENAME = 'hlb.json.gz'
+
+  # Given a bunch of CallNumberRange objects, create a new
+  # database with an efficient structure for querying
+  # @param [Array<HighLevelBrowse::CallNumberRange>] array_of_ranges
+  def initialize(array_of_ranges)
+    @all    = array_of_ranges
+    @ranges = self.create_letter_indexed_ranges(@all)
+  end
+
+  # Given an array of ranges, create efficient
+  # search structures
+  # @private
+  def create_letter_indexed_ranges(all)
+    bins = {}
+    ('A'..'Z').each do |letter|
+      cnrs         = all.find_all {|x| x.firstletter == letter}
+      bins[letter] = HighLevelBrowse::CallNumberRangeSet.new(cnrs)
+    end
+    bins
+  end
+
+  # Get the topic arrays associated with this callnumber
+  # of the form:
+  #   [
+  #     [toplevel, secondlevel],
+  #     [toplevel, secondlevel, thirdlevel],
+  #      ...
+  #   ]
+  # @param [String] raw_callnumber_string
+  # @return [Array<Array>] A (possibly empty) array of arrays of topics
+  def topics(*raw_callnumber_strings)
+    raw_callnumber_strings.reduce([]) do |acc, raw_callnumber_string|
+      firstletter = raw_callnumber_string.strip.upcase[0]
+      if @ranges.has_key? firstletter
+        acc + @ranges[firstletter].topics_for(raw_callnumber_string)
+      else
+        acc
+      end
+    end.uniq
+  end
+
+
+  alias_method :[], :topics
+
+  # Create a new object from a string with the XML
+  # in it.
+  # @param [String] xml The contents of the HLB XML dump
+  #    (e.g., from 'https://www.lib.umich.edu/browse/categories/xml.php')
+  # @return [DB]
+  def self.new_from_xml(xml)
+    oga_doc_root         = Oga.parse_xml(xml)
+    simple_array_of_cnrs = cnrs_within_oga_node(node: oga_doc_root)
+    self.new(simple_array_of_cnrs).freeze
+  end
+
+
+  # Save to disk
+  # @param [String] dir The directory where the hlb.json.gz file will be saved
+  # @return [DB] The loaded database
+  def save(dir:)
+    Zlib::GzipWriter.open(File.join(dir, FILENAME)) do |out|
+      out.puts JSON.fast_generate(@all)
+    end
+  end
+
+
+  # Load from disk
+  # @param [String] dir The directory where the hlb.json.gz file is located
+  # @return [DB] The loaded database
+  def self.load(dir:)
+    simple_array_of_cnrs = Zlib::GzipReader.open(File.join(dir, FILENAME)) do |infile|
+      JSON.load(infile.read).to_a
+    end
+    db                   = self.new(simple_array_of_cnrs)
+    db.freeze
+    db
+  end
+
+
+  # Freeze everything
+  # @return [DB] the frozen db
+  def freeze
+    @ranges.freeze
+    @all.freeze
+    self
+  end
+
+  private
+
+  # Recurse through the parsed XML document, at each stage keeping track of
+  #  * where we are (what are the xpath children?)
+  #  * what the current topics are ([level1, level2])
+  # Get all the call numbers assocaited with the topic represented by the given node,
+  # as well as all the children of the given node, and send it back as a big ol' array
+  # @param [Oga::Node] node A node of the parsed HLB XML file
+  # @param [Array<String>] decendent_xpaths A list of xpaths to the decendents of this node
+  # @param [Array<String>] topic_array An array with all levels of the topics associated with this node
+  # @return [Array<HighLevelBrowse::CallNumberRange>]
+  def self.cnrs_within_oga_node(node:, decendent_xpaths: ['/hlb/subject', 'topic', 'sub-topic'], topic_array: [])
+    if decendent_xpaths.empty?
+      [] # base case -- we're as low as we're going to go
+    else
+      current_xpath_component = decendent_xpaths[0]
+      new_xpath               = decendent_xpaths[1..-1]
+      new_topic               = topic_array.dup
+      new_topic.push node.get(:name) unless node == node.root_node # skip the root
+      cnrs = []
+      # For each sub-component, get both the call-number-ranges (cnrs) assocaited
+      # with this level, as well as recusively getting from all the children
+      node.xpath(current_xpath_component).each do |c|
+        cnrs += call_numbers_list_from_leaves(node: c, topic_array: new_topic)
+        cnrs += cnrs_within_oga_node(node: c, decendent_xpaths: new_xpath, topic_array: new_topic)
+      end
+      cnrs
+    end
+  end
+
+
+  # Given a second-to-lowest-level node, get its topic and
+  # extract call number ranges from its children
+  def self.call_numbers_list_from_leaves(node:, topic_array:)
+    cnrs      = []
+    new_topic = topic_array.dup.push node.get(:name)
+    node.xpath('call-numbers').each do |cn_node|
+      min = cn_node.get(:start)
+      max = cn_node.get(:end)
+
+      new_cnr = HighLevelBrowse::CallNumberRange.new(min: min, max: max, topic_array: new_topic)
+      if new_cnr.illegal?
+        # do some sort of logging
+      else
+        cnrs.push new_cnr
+      end
+    end
+    cnrs
+
+  end
+
+
+end

data/lib/high_level_browse/range_tree.rb

@@ -0,0 +1,90 @@
+# Never released as a gem, as near as I can tell.
+# Taken from https://github.com/clearhaus/range-tree,
+# which was released under the MIT license
+# by ClearHaus (https://www.clearhaus.com/)
+
+# Namespaced to avoid conflicts with other range_tree
+# gems
+
+module HighLevelBrowse
+  class RangeTree
+    class Node
+      def initialize(left, range, right, min, max)
+        @left  = left
+        @range = range
+        @right = right
+        @min   = min || range.min
+        @max   = max || range.max
+      end
+
+      attr_reader :left, :range, :right, :min, :max
+    end
+
+    def initialize(ranges, sorted: false)
+      # ranges.sort_by! {|r| [r.min, r.max]} unless sorted
+      # It's only required to be sorted by `r.min`, but if many ranges has the
+      # same left endpoint, then it's more efficient if also secondarily sorted by
+      # the right endpoint (or equivalently by the length).
+
+      @root = RangeTree.split(ranges.sort{|a,b| (a.min <=> b.min) || (a.max <=> b.max)})
+    end
+
+    attr_reader :root
+
+    def self.split(ranges)
+      return nil if ranges.empty?
+
+      middle = ranges.length/2
+
+      left  = split(ranges.slice(0, middle)) # Handle middle == 0 correctly.
+      range = ranges[middle] # Current range.
+      right = split(ranges[(middle+1)..-1]) # Handle middle == ranges.length correctly.
+
+      ary = [left, range, right].compact
+
+      Node.new(left, range, right,
+               ary.map(&:min).min, # Subtree's min.
+               ary.map(&:max).max) # Subtree's max.
+    end
+
+    def search(range, limit: Float::INFINITY)
+      range = range.is_a?(Range) ? range : (range..range)
+
+      result = []
+      RangeTree.search_helper(range, @root, result, limit)
+
+      result
+    end
+
+    def self.search_helper(q, root, result, limit)
+      return if root.nil?
+
+      # Visit left child?
+      if (l = root.left) and l.max and q.min and \
+        not l.max < q.min # The interesting part.
+        search_helper(q, root.left, result, limit)
+      end
+
+      return if result.length >= limit
+      # Yes, it needs to be checked here rather than in the top. Otherwise, at the
+      # point of checking, there wasn't added too many, but after left child has
+      # been checked, we might hit the limit and then, "this" will add one as
+      # well.
+
+      # Add root?
+      result << root.range if RangeTree.ranges_intersect?(q, root.range)
+
+      # Visit right child?
+      if (r = root.right) and q.max and r.min and \
+        not q.max < r.min # The interesting part.
+        search_helper(q, root.right, result, limit)
+      end
+    end
+
+    def self.ranges_intersect?(a, b)
+      return false unless a.min && a.max && b.min && b.max
+
+      a.min <= b.max && a.max >= b.min
+    end
+  end
+end

data/lib/high_level_browse/version.rb

@@ -0,0 +1,3 @@
+  module HighLevelBrowse
+    VERSION = "0.1.0"
+  end

data/test/minitest_helper.rb

@@ -0,0 +1,14 @@
+$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
+
+# Both oga and minitest have stupid warnings that I don't want to
+# hear about
+
+verbose = $VERBOSE
+$VERBOSE = nil
+require 'oga'
+require 'minitest'
+require 'minitest/spec'
+require 'minitest/autorun'
+$VERBOSE = verbose
+
+require 'high_level_browse'

data/test/test_high_level_browse.rb

@@ -0,0 +1,27 @@
+require 'minitest_helper'
+
+require 'json'
+TESTDIR = File.expand_path(File.dirname(__FILE__))
+
+describe "loads" do
+  it "loads" do
+    assert true
+  end
+
+  it "has a version" do
+    HighLevelBrowse::VERSION.wont_be_nil
+  end
+end
+
+describe "Works the same as before" do
+  it "gets the same output for 30k randomly chosen call numbers" do
+    h = HighLevelBrowse.fetch_and_save(dir: TESTDIR)
+    JSON.load(File.open(File.join(TESTDIR, '30k_random_old_mappings.json'))).each do |rec|
+      cn = rec['cn'].strip
+      newcats = h[cn]
+      next if rec['jar'].empty?
+      assert_equal [cn, rec['jar'].sort], [rec['cn'], newcats.sort]
+    end
+
+  end
+end

metadata

@@ -0,0 +1,138 @@
+--- !ruby/object:Gem::Specification
+name: high_level_browse
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Bill Dueber
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-06-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: oga
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: lcsort
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- bill@dueber.com
+executables:
+- fetch_new_hlb
+- hlb
+- test_marc_file_for_hlb
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bench/bench.rb
+- bench/hlb.json.gz
+- bin/fetch_new_hlb
+- bin/hlb
+- bin/test_marc_file_for_hlb
+- high_level_browse.gemspec
+- lib/high_level_browse.rb
+- lib/high_level_browse/call_number_range.rb
+- lib/high_level_browse/db.rb
+- lib/high_level_browse/range_tree.rb
+- lib/high_level_browse/version.rb
+- test/minitest_helper.rb
+- test/test_high_level_browse.rb
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.8
+signing_key: 
+specification_version: 4
+summary: Map LC call numbers to academic categories.
+test_files:
+- test/minitest_helper.rb
+- test/test_high_level_browse.rb