oai 0.0.1 → 1.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f07e6aaa014da7ec0e3fc389e2dd8dfe05c91e3583cea3123852558ba90e9acf
+  data.tar.gz: 88f2c92e7697899526e6df57582b4b4ae054c8e7b9ad5dc0b27774bd51198544
+SHA512:
+  metadata.gz: 85064e01e23a76b7dfec02dac0f6cdd76389683401052d20b8021f16b8b499f33b344ef850c1a1b5b38a14ac8a70bbb6e0013429c0ad4e0d9ad2ab7d6750052c
+  data.tar.gz: 2b294bf969081f3fd310156a7ad3b3dd7aff977c49473f2892a625998dff3b01ae4c0b74a2ce04215c3832ccb1c8a0a5bec468f54dae46b1e5ce111f438c4607

data/README.md

@@ -0,0 +1,128 @@
+ruby-oai
+========
+[![Build Status](https://github.com/code4lib/ruby-oai/workflows/CI/badge.svg)](https://github.com/code4lib/ruby-oai/actions)
+
+[![Gem Version](https://badge.fury.io/rb/oai.svg)](https://badge.fury.io/rb/oai)
+
+ruby-oai is a Open Archives Protocol for Metadata Harvesting (OAI-PMH)
+library for Ruby. [OAI-PMH](http://openarchives.org) is a somewhat
+archaic protocol for sharing metadata between digital library repositories.
+If you are looking to share metadata on the web you are probably better off
+using a feed format like [RSS](http://www.rssboard.org/rss-specification) or
+[Atom](http://www.atomenabled.org/). If have to work with a backwards
+digital repository that only offers OAI-PMH access then ruby-oai is your
+friend.
+
+The [OAI-PMH](http://openarchives.org) spec defines six verbs
+(`Identify`, `ListIdentifiers`, `ListRecords`,
+`GetRecords`, `ListSets`, `ListMetadataFormat`) used for discovery and sharing of
+metadata.
+
+The ruby-oai gem includes a client library, a server/provider library and
+a interactive harvesting shell.
+
+Client
+------
+
+The OAI client library is used for harvesting metadata from repositories.
+For example to initiate a ListRecords request to pubmed you can:
+
+```ruby
+  require 'oai'
+  client = OAI::Client.new 'http://www.pubmedcentral.gov/oai/oai.cgi', :headers => { "From" => "oai@example.com" }
+  response = client.list_records
+  # Get the first page of records
+  response.each do |record|
+    puts record.metadata
+  end
+  # Get the second page of records
+  response = client.list_records(:resumption_token => response.resumption_token)
+  response.each do |record|
+    puts record.metadata
+  end
+  # Get all pages together (may take a *very* long time to complete)
+  client.list_records.full.each do |record|
+    puts record.metadata
+  end
+```
+
+### Retry-After
+This library depends on faraday, but allows a wide range of versions. Depending on the client application's installed version of faraday, there may be different middleware libraries required to support automatically retrying requests that are rate limited/denied with a `Retry-After` header. The OAI client can, however, accept an externally configured faraday http client for handling this. For example, to retry on `429 Too Many Requests`:
+
+```ruby
+require 'oai'
+require 'faraday_middleware' # if using faraday version < 2
+http_client = Faraday.new do |conn|
+    conn.request(:retry, max: 5, retry_statuses: 429)
+    conn.response(:follow_redirects, limit: 5)
+    conn.adapter :net_http
+end
+client = OAI::Client.new(base_url, http: http_client)
+opts = {from:'2012-03-01', until:'2012-04-01', metadata_prefix:'oai_dc'}
+puts client.list_records(opts).full.count
+```
+
+See {OAI::Client} for more details
+
+Server
+------
+
+The OAI provider library handles serving local content to other clients. Here's how to set up a simple provider:
+
+```ruby
+  class MyProvider < Oai::Provider
+    repository_name 'My little OAI provider'
+    repository_url  'http://localhost/provider'
+    record_prefix 'oai:localhost'
+    admin_email 'root@localhost'   # String or Array
+    source_model MyModel.new       # Subclass of OAI::Provider::Model
+  end
+```
+
+See comment docs at top of [OAI::Provider](./lib/oai/provider.rb) for more details, including discussion of the `OAI::Provider::ActiveRecordWrapper` class for quich setup of an OAI provider for an ActiveRecord model class (single database table)
+
+Interactive Harvester
+---------------------
+
+The OAI-PMH client shell allows OAI Harvesting to be configured in an interactive manner.  Typing `oai` on the command line starts the shell. After initial configuration, the shell can be used to manage harvesting operations.
+
+See {OAI::Harvester::Shell} for more details
+
+Installation
+------------
+
+Normally the best way to install oai is as part of your `Gemfile`:
+
+    source :rubygems
+    gem 'oai'
+
+Alternately it can be installed globally using RubyGems:
+
+    $ gem install oai
+
+Running tests
+-------------
+
+Tests are with Test::Unit, in a somewhat archaic/legacy style. Test setup especially is not how we would do things today. Run all tests with:
+
+    $ bundle exec rake test
+
+There are also convenience tasks to run subsets of tests.
+
+We use [appraisal](https://github.com/thoughtbot/appraisal) to test ActiveRecord-related functionality under multiple versions of ActiveRecord. While the above commands will test with latest ActiveRecord (allowed in our .gemspec development dependency), you can test under a particular version defined in the [Appraisals](./Appraisals) file like so:
+
+    $ bundle exec appraisal rails-52 rake test
+    $ bundle exec appraisal rails-70 rake test
+
+If you run into trouble with appraisal's gemfiles getting out of date and bundler complaining,
+try:
+
+   $ bundle exec appraisal clean
+   $ appraisal generate
+
+That may make changes to appraisal gemfiles that you should commit to repo.
+
+License
+-------
+
+[MIT](./LICENSE)

data/Rakefile

@@ -0,0 +1,64 @@
+
+require 'rubygems'
+require 'rake'
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+require 'yard'
+
+task :default => ["test", "yard"]
+
+Rake::TestTask.new('test') do |t|
+  t.description = "Run all Test::Unit tests"
+
+  t.libs << ['lib', 'test/client', 'test/provider', 'test/activerecord_provider']
+
+  t.pattern = 'test/{client,provider,activerecord_provider}/tc_*.rb'
+  #t.verbose = true
+  t.warning = false
+end
+
+
+# To run just subsets of tests
+namespace :test do
+  Rake::TestTask.new('client') do |t|
+    t.libs << ['lib', 'test/client']
+    t.pattern = 'test/client/tc_*.rb'
+    #t.verbose = true
+    t.warning = false
+  end
+
+  Rake::TestTask.new('harvester') do |t|
+    t.libs << ['lib', 'test/harvester']
+    t.pattern = 'test/harvester/tc_*.rb'
+    #t.verbose = true
+    t.warning = false
+  end
+
+  Rake::TestTask.new('provider') do |t|
+    t.libs << ['lib', 'test/provider']
+    t.pattern = 'test/provider/tc_*.rb'
+    #t.verbose = true
+    t.warning = false
+  end
+
+  Rake::TestTask.new('activerecord_provider') do |t|
+    t.description = "Active Record base Provider Tests"
+
+    t.libs << ['lib', 'test/activerecord_provider']
+    t.pattern = 'test/activerecord_provider/tc_*.rb'
+    #t.verbose = true
+    t.warning = false
+  end
+end
+
+YARD::Rake::YardocTask.new do |t|
+  t.files = ["lib/**/*.rb"]
+  t.options = ['--output-dir', 'doc']
+end

data/bin/oai

@@ -0,0 +1,66 @@
+#!/usr/bin/env ruby -rubygems
+#
+#  Created by William Groppe on 2006-11-05.
+#  Copyright (c) 2006. All rights reserved.
+
+require 'optparse'
+
+require 'oai/harvester'
+
+include OAI::Harvester
+
+conf = OAI::Harvester::Config.load
+
+startup = :interactive
+
+rexml = false
+
+opts = OptionParser.new do |opts|
+  opts.banner = "Usage: oai ..."
+  opts.define_head "#{File.basename($0)}, a OAI harvester shell."
+  opts.separator ""
+  opts.separator "Options:"
+  
+  opts.on("-D", "--daemon", "Non-interactive mode, to be called via scheduler") { startup = :daemon }
+  opts.on("-R", "--rexml", "Use rexml even if libxml is available") { rexml = true }
+  opts.on("-?", "--help", "Show this message") do
+    puts opts
+    exit
+  end
+  
+  # Another typical switch to print the version.
+  opts.on_tail("-v", "--version", "Show version") do
+      class << Gem; attr_accessor :loaded_specs; end
+      puts Gem.loaded_specs['oai'].version
+      exit
+  end
+end
+
+begin
+  opts.parse! ARGV
+rescue
+  puts opts
+  exit
+end
+
+unless rexml
+  begin # Try to load libxml to speed up harvesting
+    require 'xml/libxml'
+  rescue LoadError
+  end
+end
+
+case startup
+when :interactive
+  shell = Shell.new(conf)
+  shell.start
+when :daemon
+  if conf.storage
+    harvest = Harvest.new(conf)
+    harvest.start(harvestable_sites(conf))
+  else
+    puts "Missing or corrupt configuration file, cannot continue."
+    exit(-1)
+  end
+end
+  

data/examples/models/file_model.rb

@@ -0,0 +1,63 @@
+#!/usr/bin/env ruby
+#
+# Created by William Groppe on 2007-02-01.
+#
+# Simple file based Model.  Basically just serves a directory of xml files to the
+# Provider.
+#
+class File
+  def id
+    File.basename(self.path)
+  end
+  
+  def to_oai_dc
+    self.read
+  end
+end
+    
+class FileModel < OAI::Provider::Model
+  include OAI::Provider
+  
+  def initialize(directory = 'data')
+    # nil specifies no partial results aka resumption tokens, and 'mtime' is the
+    # method that the provider will call for determining the timestamp
+    super(nil, 'mtime')
+    @directory = directory
+  end
+  
+  def earliest
+    e = Dir["#{@directory}/*.xml"].min { |a,b| File.stat(a).mtime <=> File.stat(b).mtime }
+    File.stat(e).mtime.utc.xmlschema
+  end
+  
+  def latest
+    e = Dir["#{@directory}/*.xml"].max { |a,b| File.stat(a).mtime <=> File.stat(b).mtime }
+    File.stat(e).mtime.utc.xmlschema
+  end
+
+  def sets
+    nil
+  end
+  
+  def find(selector, opts={})
+    return nil unless selector
+
+    case selector
+    when :all
+      records = Dir["#{@directory}/*.xml"].sort.collect do |file|
+        File.new(file) unless File.stat(file).mtime.utc < opts[:from].to_time or
+          File.stat(file).mtime.utc > opts[:until].to_time
+      end
+      records
+    else
+      Find.find("#{@directory}/#{selector}") rescue nil
+    end
+  end
+  
+end
+
+# == Example Usage:
+# class FileProvider < OAI::Provider::Base
+#   repository_name 'XML File Provider'
+#   source_model FileModel.new('/tmp')
+# end