jstor-dl 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: cd0503a0a9da1a8298c012fab0dd19f866e907a87a58537f6f1b26174b3bb1ab
+  data.tar.gz: 20779888e79ac989cc10b14ac96877b9200641f7e4cc3fe679f252b4602b1570
+SHA512:
+  metadata.gz: 6839ae6451e76bfe92b282b4829a7dcea3b117c103bf68d345ad8e91d7a922d56d30bd27007a3be8c0920ac219c67c0b04ef98afa0e0206e0e9e7e8b3ebdf758
+  data.tar.gz: d0a458946ce663a24638a25b4fbc23d8579352c101105cb371e764b086524fef22723080e93314512cdfcdd0a8fd061cfb929edaac067f1c06087e24cef0d392

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+## [0.1.0]
+
+First version. Per-article offline archive of JSTOR's public-domain Early Journal Content, fetched only from the Internet Archive's copy, never from jstor.org.
+
+- Accepts JSTOR stable IDs, jstor.org URLs, `10.2307/…` DOIs and doi.org URLs, and archive.org `jstor-…` items and URLs.
+- Saves the scanned PDF, the OCR plaintext, JSTOR's article metadata XML verbatim, and four sidecar metadata files (`metadata.md`, `metadata.yaml`, `metadata.json`, `metadata.bib`).
+- Layout: `YYYY/MM/DD/<journal>/<jstor-id>-<slug>/`.
+- Articles not in the Early Journal Content on archive.org raise `Jstor::Downloader::ItemNotFound`.
+- Rate-limited HTTP client (3s default) with timeouts and retries on 429/503, staged downloads that skip already-archived articles, and a CLI with `--input FILE|-`, per-target error reporting, and exit status 1 on any failure. All copied from arxiv-dl.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,83 @@
+# Contributor Covenant 3.0 Code of Conduct
+
+## Our Pledge
+
+We pledge to make our community welcoming, safe, and equitable for all.
+
+We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant.
+
+## Encouraged Behaviors
+
+While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language.
+
+With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including:
+
+1. Respecting the **purpose of our community**, our activities, and our ways of gathering.
+2. Engaging **kindly and honestly** with others.
+3. Respecting **different viewpoints** and experiences.
+4. **Taking responsibility** for our actions and contributions.
+5. Gracefully giving and accepting **constructive feedback**.
+6. Committing to **repairing harm** when it occurs.
+7. Behaving in other ways that promote and sustain the **well-being of our community**.
+
+## Restricted Behaviors
+
+We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct.
+
+1. **Harassment.** Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop.
+2. **Character attacks.** Making insulting, demeaning, or pejorative comments directed at a community member or group of people.
+3. **Stereotyping or discrimination.** Characterizing anyone’s personality or behavior on the basis of immutable identities or traits.
+4. **Sexualization.** Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community.
+5. **Violating confidentiality**. Sharing or acting on someone's personal or private information without their permission.
+6. **Endangerment.** Causing, encouraging, or threatening violence or other harm toward any person or group.
+7. Behaving in other ways that **threaten the well-being** of our community.
+
+### Other Restrictions
+
+1. **Misleading identity.** Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions.
+2. **Failing to credit sources.** Not properly crediting the sources of content you contribute.
+3. **Promotional materials**. Sharing marketing or other commercial content in a way that is outside the norms of the community.
+4. **Irresponsible communication.** Failing to responsibly present content which includes, links or describes any other restricted behaviors.
+
+## Reporting an Issue
+
+Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm.
+
+When an incident does occur, it is important to report it promptly. To report a possible violation, **email the maintainer at [veganstraightedge@gmail.com](mailto:veganstraightedge@gmail.com).**
+
+Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution.
+
+## Addressing and Repairing Harm
+
+If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped.
+
+1. Warning
+   1. Event: A violation involving a single incident or series of incidents.
+   2. Consequence: A private, written warning from the Community Moderators.
+   3. Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations.
+2. Temporarily Limited Activities
+   1. Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation.
+   2. Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members.
+   3. Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over.
+3. Temporary Suspension
+   1. Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation.
+   2. Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions.
+   3. Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted.
+4. Permanent Ban
+   1. Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member.
+   2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior.
+   3. Repair: There is no possible repair in cases of this severity.
+
+This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at [https://www.contributor-covenant.org/version/3/0/](https://www.contributor-covenant.org/version/3/0/).
+
+Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/)
+
+For answers to common questions about Contributor Covenant, see the FAQ at [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq). Translations are provided at [https://www.contributor-covenant.org/translations](https://www.contributor-covenant.org/translations). Additional enforcement and community guideline resources can be found at [https://www.contributor-covenant.org/resources](https://www.contributor-covenant.org/resources). The enforcement ladder was inspired by the work of [Mozilla’s code of conduct team](https://github.com/mozilla/inclusion).

data/LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Shane Becker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,121 @@
+# jstor-dl
+
+Download articles from JSTOR's public-domain Early Journal Content for offline archives.
+
+For each article, `jstor-dl` saves:
+
+- The scanned PDF
+- The OCR plaintext
+- JSTOR's own article metadata (XML), verbatim
+- Four sidecar metadata files: `metadata.md`, `metadata.yaml`, `metadata.json`, `metadata.bib`
+
+## Where the content comes from
+
+`jstor-dl` fetches only from the [Internet Archive's copy](https://archive.org/details/jstor_ejc) of JSTOR's Early Journal Content, never from jstor.org. [JSTOR's terms](https://about.jstor.org/terms) forbid any tool from downloading from jstor.org, even a single article; they allow only manual downloading.
+
+The Early Journal Content is nearly 500,000 public-domain articles from 200+ journals (published before 1923 in the US, before 1870 elsewhere), released by JSTOR for free non-commercial use with acknowledgement, and uploaded to the Internet Archive in 2013 for bulk harvesting. Articles JSTOR added to its Early Journal Content after 2013 may be missing from the Internet Archive's copy.
+
+## Installation
+
+```sh
+gem install jstor-dl
+```
+
+## CLI usage
+
+```sh
+jstor-dl <JSTOR_ID_OR_URL> [<JSTOR_ID_OR_URL>...]
+```
+
+Accepted input forms:
+
+| Form                    | Example                                                  |
+| ----------------------- | -------------------------------------------------------- |
+| Stable ID               | `4385670`                                                |
+| JSTOR URL               | `https://www.jstor.org/stable/4385670`                   |
+| JSTOR URL, DOI form     | `https://www.jstor.org/stable/10.2307/4385670`           |
+| JSTOR PDF URL           | `https://www.jstor.org/stable/pdf/4385670.pdf`           |
+| DOI                     | `10.2307/4385670`, `doi:10.2307/4385670`                 |
+| DOI URL                 | `https://doi.org/10.2307/4385670`                        |
+| Internet Archive item   | `jstor-4385670`                                          |
+| Internet Archive URL    | `https://archive.org/details/jstor-4385670`              |
+
+JSTOR URLs are only read for the article's ID; nothing is requested from jstor.org.
+
+### Flags
+
+| Flag                      | Description                                                                   |
+| ------------------------- | ----------------------------------------------------------------------------- |
+| `-i FILE`, `--input FILE` | Read IDs/URLs from FILE, one per line (`-` for stdin; blanks and `#` skipped) |
+| `-p PATH`, `--path PATH`  | Root download directory                                                       |
+| `--rate-limit SECONDS`    | Seconds between HTTP requests; `0` disables throttling                        |
+| `-v`, `--verbose`         | Print step lines and per-request URL/byte logs to stdout                      |
+| `-q`, `--quiet`           | Print nothing to stdout; errors still go to stderr                            |
+| `--version`               | Print the gem version and exit                                                |
+| `-h`, `--help`            | Print help and exit                                                           |
+
+`-v` and `-q` are mutually exclusive.
+
+### Environment variables
+
+| Variable              | Effect                                                            |
+| --------------------- | ----------------------------------------------------------------- |
+| `JSTOR_DOWNLOAD_PATH` | Root download directory (default: `$HOME/Downloads/JSTOR_Papers`) |
+| `JSTOR_RATE_LIMIT`    | Seconds between HTTP requests (default: `3`; `0` disables)        |
+
+Precedence: CLI flag > ENV var > default.
+
+### Errors and exit status
+
+A target that fails (unrecognized ID, not in the Early Journal Content on archive.org, HTTP error, network failure) is reported on stderr as `<target>: <message>`, and the remaining targets still download. Exit status is `0` when every target succeeds and `1` when any fails.
+
+## Output layout
+
+```txt
+$JSTOR_DOWNLOAD_PATH/                   # default: $HOME/Downloads/JSTOR_Papers
+  YYYY/MM/DD/<journal>/<jstor-id>-<slug>/
+    <jstor-id>.pdf                      # scanned article
+    <jstor-id>.txt                      # OCR plaintext
+    jstor.xml                           # JSTOR's article metadata, verbatim
+    metadata.md                         # YAML frontmatter + Markdown body
+    metadata.yaml
+    metadata.json
+    metadata.bib                        # synthesized from the metadata
+```
+
+`YYYY/MM/DD` is the publication date (shorter when only the year or month is known). `<journal>` is JSTOR's journal abbreviation (`clasweek` for The Classical Weekly). `<slug>` is derived from the article title.
+
+Each article downloads into a sibling `.partial` folder and is renamed into place only when every file succeeded. Re-running skips articles already archived.
+
+## Library usage
+
+```ruby
+require 'jstor/downloader'
+
+identifier = Jstor::Downloader::Identifier.new 'https://www.jstor.org/stable/4385670'
+client     = Jstor::Downloader::Client.new                # 3-second rate limit by default
+path       = Jstor::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run
+# => "/tmp/papers/1907/10/05/clasweek/4385670-the-elements-of-the-translation-of-latin"
+```
+
+## Development
+
+```sh
+script/setup    # install dependencies
+script/test     # run specs and rubocop
+script/console  # interactive prompt
+```
+
+Specs run offline against recorded fixtures in `spec/fixtures/http/`. To check those fixtures against the live archive.org API, run:
+
+```sh
+ARCHIVE_LIVE=1 script/test
+```
+
+## License
+
+MIT — see [LICENSE.md](LICENSE.md).
+
+## Code of Conduct
+
+This project follows the [Contributor Covenant](https://www.contributor-covenant.org/version/3/0/) 3.0 — see [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

data/exe/jstor-dl

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'jstor/downloader'
+
+exit Jstor::Downloader::CLI.new(ARGV).run

data/lib/jstor/downloader/archive.rb

@@ -0,0 +1,78 @@
+require 'fileutils'
+
+module Jstor
+  module Downloader
+    class Archive
+      def initialize identifier, root:, client: Client.new
+        @identifier = identifier
+        @root       = root
+        @client     = client
+      end
+
+      # Downloads into <article>.partial/ and
+      # renames it into place only once everything succeeded.
+      # An existing article folder is always complete and is skipped.
+      def run
+        return article_dir if Dir.exist? article_dir
+
+        FileUtils.rm_rf staging_dir
+        FileUtils.mkdir_p staging_dir
+
+        download_pdf
+        download_text
+        download_jstor_xml
+        write_sidecars
+
+        File.rename staging_dir, article_dir
+        article_dir
+      end
+
+      private
+
+      def metadata
+        @metadata ||= MetadataParser.new(@client.get(metadata_url).to_s).metadata
+      end
+
+      def metadata_url
+        "https://archive.org/metadata/#{@identifier.item}"
+      end
+
+      def article_dir
+        @article_dir ||= File.join @root, Path.new(metadata).to_s
+      end
+
+      # a sibling of article_dir, so the rename is a single atomic step
+      def staging_dir
+        "#{article_dir}.partial"
+      end
+
+      def download_pdf
+        ItemFile.new(@identifier, "#{@identifier}.pdf", client: @client)
+                .download to: File.join(staging_dir, "#{@identifier}.pdf")
+      end
+
+      # the OCR plaintext, when the item has it
+      def download_text
+        download_optional "#{@identifier}_djvu.txt", to: File.join(staging_dir, "#{@identifier}.txt")
+      end
+
+      # JSTOR's own article metadata, verbatim, when the item has it
+      def download_jstor_xml
+        download_optional "10.2307_#{@identifier}.xml", to: File.join(staging_dir, 'jstor.xml')
+      end
+
+      def download_optional name, to:
+        ItemFile.new(@identifier, name, client: @client).download to: to
+      rescue HTTPError => e
+        raise unless e.status == 404
+      end
+
+      def write_sidecars
+        Metadata::Markdown.new(metadata).write to: staging_dir
+        Metadata::YAML.new(metadata).write     to: staging_dir
+        Metadata::JSON.new(metadata).write     to: staging_dir
+        Metadata::Bibtex.new(metadata).write   to: staging_dir
+      end
+    end
+  end
+end

data/lib/jstor/downloader/author.rb

@@ -0,0 +1,11 @@
+module Jstor
+  module Downloader
+    Author = Data.define :name, :affiliations do
+      def initialize name:, affiliations: []
+        super
+      end
+
+      def to_s = name
+    end
+  end
+end

data/lib/jstor/downloader/bibtex.rb

@@ -0,0 +1,57 @@
+module Jstor
+  module Downloader
+    # Synthesized from the metadata only: fetching JSTOR's own citation
+    # export would be an automated download from jstor.org.
+    class Bibtex
+      def initialize metadata
+        @metadata = metadata
+      end
+
+      def to_s
+        <<~BIBTEX
+          @article{#{key},
+            title={#{@metadata.title}},
+            author={#{authors}},
+            journal={#{@metadata.journal[:name]}},
+            year={#{year}},
+            volume={#{@metadata.volume}},
+            pages={#{pages}},
+            doi={#{@metadata.doi}},
+            url={#{@metadata.jstor_url}},
+          }
+        BIBTEX
+      end
+
+      def key
+        title_word = Slug.new(@metadata.title).to_s.split('-').first
+
+        "#{surname}#{year}#{title_word}"
+      end
+
+      private
+
+      def authors
+        @metadata.authors.map(&:name).join ' and '
+      end
+
+      # "Stebbins, Joel" and "Ella Catherine Greene" both give a lowercase surname
+      def surname
+        first_author = @metadata.authors.first
+        return 'anonymous' if first_author.nil?
+
+        name = first_author.name
+        last = name.include?(',') ? name.split(',').first : name.split.last
+        last.downcase.gsub(/[^a-z]/, '')
+      end
+
+      def year
+        @metadata.published.to_s[0, 4]
+      end
+
+      # BibTeX page ranges use an en dash: 2--5
+      def pages
+        @metadata.pages.to_s.sub '-', '--'
+      end
+    end
+  end
+end

data/lib/jstor/downloader/cli.rb

@@ -0,0 +1,106 @@
+require 'optparse'
+
+module Jstor
+  module Downloader
+    class CLI
+      DEFAULT_DOWNLOAD_PATH = File.join Dir.home, 'Downloads', 'JSTOR_Papers'
+      USAGE                 = 'Usage: jstor-dl [options] <JSTOR_ID_OR_URL> [<JSTOR_ID_OR_URL>...]'.freeze
+
+      def initialize argv, stderr: $stderr, stdin: $stdin, stdout: $stdout
+        @argv   = argv
+        @stderr = stderr
+        @stdin  = stdin
+        @stdout = stdout
+      end
+
+      def run
+        options = parse
+        return options.fetch(:exit_status) if options.key? :exit_status
+
+        return error_with USAGE       if options[:targets].empty?
+        return error_with conflict if options[:verbose] && options[:quiet]
+
+        failures = download_each options
+        failures.zero? ? 0 : 1
+      end
+
+      private
+
+      def conflict
+        '-v and -q are mutually exclusive'
+      end
+
+      def parse
+        options = { targets: [], verbose: false, quiet: false }
+        parser  = build_parser options
+
+        begin
+          parser.parse! @argv
+          options[:targets] = @argv + input_targets(options[:input])
+        rescue OptionParser::ParseError, SystemCallError => e
+          @stderr.puts e.message
+          return { exit_status: 1 }
+        end
+
+        options[:path]       ||= ENV['JSTOR_DOWNLOAD_PATH'] || DEFAULT_DOWNLOAD_PATH
+        options[:rate_limit] ||= (ENV['JSTOR_RATE_LIMIT'] || Client::DEFAULT_RATE_LIMIT).to_i
+        options
+      end
+
+      def build_parser options
+        OptionParser.new do |parser|
+          parser.banner = USAGE
+          parser.on('-i FILE', '--input FILE')       { |value| options[:input] = value }
+          parser.on('-p PATH', '--path PATH')        { |value| options[:path] = value }
+          parser.on('--rate-limit SECONDS', Integer) { |value| options[:rate_limit] = value }
+          parser.on('-v', '--verbose')               { options[:verbose] = true }
+          parser.on('-q', '--quiet')                 { options[:quiet]   = true }
+          parser.on('--version') do
+            @stdout.puts VERSION
+            options[:exit_status] = 0
+          end
+          parser.on('-h', '--help') do
+            @stdout.puts parser.help
+            options[:exit_status] = 0
+          end
+        end
+      end
+
+      # one target per line from FILE, or stdin for "-"; blank lines and # comments skipped
+      def input_targets input
+        return [] if input.nil?
+
+        text = input == '-' ? @stdin.read : File.read(input)
+        text.lines.map(&:strip).reject { it.empty? || it.start_with?('#') }
+      end
+
+      def error_with message
+        @stderr.puts message
+        1
+      end
+
+      def download_each options
+        client = Client.new rate_limit: options[:rate_limit], log: (options[:verbose] ? @stdout : nil)
+
+        failures = 0
+        options[:targets].each do |target|
+          failures += 1 unless download_one(target, client:, options:)
+        end
+        failures
+      end
+
+      # true on success; reports the failure and returns false otherwise
+      def download_one target, client:, options:
+        identifier = Identifier.new target
+        @stdout.puts "==> Downloading #{identifier.id}" if options[:verbose]
+
+        path = Archive.new(identifier, root: options[:path], client: client).run
+        @stdout.puts path unless options[:quiet]
+        true
+      rescue Error, HTTP::Error => e
+        @stderr.puts "#{target}: #{e.message}"
+        false
+      end
+    end
+  end
+end

data/lib/jstor/downloader/client.rb

@@ -0,0 +1,86 @@
+require 'http'
+
+module Jstor
+  module Downloader
+    class Client
+      SOURCE_URL         = 'https://github.com/xoengineering/jstor-dl'.freeze
+      DEFAULT_RATE_LIMIT = 3
+      TIMEOUTS           = { connect: 10, read: 60, write: 10 }.freeze # seconds, per operation
+      MAX_RETRIES        = 3
+      RETRY_BACKOFF      = 10 # seconds before the first retry. doubles on each retry.
+      RETRYABLE_STATUSES = [429, 503].freeze
+
+      attr_reader :rate_limit
+
+      def initialize rate_limit: DEFAULT_RATE_LIMIT, log: nil
+        @rate_limit = rate_limit
+        @log        = log
+      end
+
+      def user_agent
+        "jstor-dl/#{VERSION} (+#{SOURCE_URL})"
+      end
+
+      def get url
+        retries = 0
+
+        loop do
+          response = request url
+          return response if response.status.success?
+          raise http_error(url, response) unless retryable? response, retries
+
+          retries += 1
+          wait_before_retry response, retries
+        end
+      end
+
+      private
+
+      def request url
+        throttle
+        response = HTTP.timeout(TIMEOUTS).headers('User-Agent' => user_agent).follow.get(url)
+        @last_request_at = Time.now
+        log_request url, response
+        response
+      end
+
+      def http_error url, response
+        HTTPError.new status: response.status.code, url: url, reason: response.status.reason
+      end
+
+      def retryable? response, retries
+        RETRYABLE_STATUSES.include?(response.status.code) && retries < MAX_RETRIES
+      end
+
+      def wait_before_retry response, retries
+        seconds = retry_after(response) || (RETRY_BACKOFF * (2**(retries - 1)))
+        @log&.puts "==> #{response.status}. Retrying in #{seconds}s"
+        sleep seconds
+      end
+
+      # Retry-After in delay-seconds form. The HTTP-date form falls back to backoff.
+      def retry_after response
+        value = response.headers['Retry-After']
+        return if value.nil?
+
+        Integer(value, exception: false)
+      end
+
+      def log_request url, response
+        return if @log.nil?
+
+        @log.puts "==> GET #{url} (#{response.body.to_s.bytesize} bytes)"
+      end
+
+      def throttle
+        return if @rate_limit.zero?
+        return if @last_request_at.nil?
+
+        elapsed = Time.now - @last_request_at
+        return if elapsed >= @rate_limit
+
+        sleep(@rate_limit - elapsed)
+      end
+    end
+  end
+end

data/lib/jstor/downloader/error.rb

@@ -0,0 +1,5 @@
+module Jstor
+  module Downloader
+    class Error < StandardError; end
+  end
+end

data/lib/jstor/downloader/http_error.rb

@@ -0,0 +1,14 @@
+module Jstor
+  module Downloader
+    class HTTPError < Error
+      attr_reader :status, :url
+
+      def initialize status:, url:, reason: nil
+        @status = status
+        @url    = url
+
+        super("GET #{url} failed: #{[status, reason].compact.join ' '}")
+      end
+    end
+  end
+end

data/lib/jstor/downloader/identifier.rb

@@ -0,0 +1,66 @@
+require 'uri'
+
+module Jstor
+  module Downloader
+    # A JSTOR stable ID for an Early Journal Content article. EJC stable IDs are
+    # numeric; JSTOR's newer IDs (j.ctt…, resrep…) are never in the EJC.
+    class Identifier
+      class Invalid < Error; end
+
+      STABLE_ID = /\A\d+\z/
+      DOI       = %r{\A(?:doi:)?10\.2307/(\d+)\z}i
+      ITEM      = /\Ajstor-(\d+)\z/
+
+      JSTOR_PATHS   = [%r{\A/stable/(?:10\.2307/)?(\d+)\z}, %r{\A/stable/pdf/(\d+)\.pdf\z}].freeze
+      DOI_PATH      = %r{\A/10\.2307/(\d+)\z}
+      ARCHIVE_PATH  = %r{\A/(?:details|download)/jstor-(\d+)(?:/|\z)}
+      JSTOR_HOSTS   = %w[jstor.org www.jstor.org].freeze
+      DOI_HOSTS     = %w[doi.org dx.doi.org].freeze
+      ARCHIVE_HOSTS = %w[archive.org www.archive.org].freeze
+
+      attr_reader :id, :input
+
+      def initialize input
+        @input = input
+        @id    = parse input.to_s.strip
+        raise Invalid, "not a JSTOR Early Journal Content identifier: #{input}" if @id.nil?
+      end
+
+      # the archive.org item holding this article
+      def item = "jstor-#{id}"
+
+      def doi = "10.2307/#{id}"
+
+      def to_s = id
+
+      private
+
+      def parse text
+        return text if STABLE_ID.match? text
+        return Regexp.last_match(1) if DOI.match(text) || ITEM.match(text)
+
+        parse_url text
+      end
+
+      def parse_url text
+        uri  = URI.parse(text.include?('://') ? text : "https://#{text}")
+        path = uri.path.to_s
+
+        return first_capture(JSTOR_PATHS, path) if JSTOR_HOSTS.include? uri.host
+        return first_capture([DOI_PATH], path)  if DOI_HOSTS.include? uri.host
+
+        first_capture([ARCHIVE_PATH], path) if ARCHIVE_HOSTS.include? uri.host
+      rescue URI::InvalidURIError
+        nil
+      end
+
+      def first_capture patterns, path
+        patterns.each do |pattern|
+          match = pattern.match path
+          return match[1] if match
+        end
+        nil
+      end
+    end
+  end
+end

data/lib/jstor/downloader/item_file.rb

@@ -0,0 +1,23 @@
+module Jstor
+  module Downloader
+    # One file from the article's archive.org item. /download/ redirects to a
+    # storage node; Client follows redirects across hosts.
+    class ItemFile
+      def initialize identifier, name, client:
+        @identifier = identifier
+        @name       = name
+        @client     = client
+      end
+
+      def download to:
+        File.binwrite to, @client.get(url).to_s
+      end
+
+      private
+
+      def url
+        "https://archive.org/download/#{@identifier.item}/#{@name}"
+      end
+    end
+  end
+end

data/lib/jstor/downloader/item_not_found.rb

@@ -0,0 +1,13 @@
+module Jstor
+  module Downloader
+    class ItemNotFound < Error
+      MESSAGE = <<~MESSAGE.chomp
+        not in the Early Journal Content on archive.org (JSTOR's terms allow only manual download from jstor.org)
+      MESSAGE
+
+      def initialize message = MESSAGE
+        super
+      end
+    end
+  end
+end

data/lib/jstor/downloader/metadata/bibtex.rb

@@ -0,0 +1,20 @@
+require 'fileutils'
+
+module Jstor
+  module Downloader
+    class Metadata
+      class Bibtex
+        FILENAME = 'metadata.bib'.freeze
+
+        def initialize metadata
+          @metadata = metadata
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), Downloader::Bibtex.new(@metadata).to_s
+        end
+      end
+    end
+  end
+end

data/lib/jstor/downloader/metadata/json.rb

@@ -0,0 +1,33 @@
+require 'fileutils'
+require 'json'
+
+module Jstor
+  module Downloader
+    class Metadata
+      class JSON
+        FILENAME = 'metadata.json'.freeze
+
+        def initialize metadata
+          @metadata = metadata
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), "#{::JSON.pretty_generate(serialize(@metadata.to_h))}\n"
+        end
+
+        private
+
+        def serialize object
+          case object
+          when Hash       then object.to_h { |key, value| [key.to_s, serialize(value)] }
+          when Array      then object.map { |item| serialize item }
+          when Data       then serialize object.to_h
+          when Date, Time then object.iso8601
+          else object
+          end
+        end
+      end
+    end
+  end
+end

data/lib/jstor/downloader/metadata/markdown.rb

@@ -0,0 +1,69 @@
+require 'fileutils'
+require 'yaml'
+
+module Jstor
+  module Downloader
+    class Metadata
+      class Markdown
+        FILENAME = 'metadata.md'.freeze
+
+        def initialize metadata
+          @metadata = metadata
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), "#{frontmatter}\n#{body}"
+        end
+
+        private
+
+        def frontmatter
+          "---\n#{::YAML.dump(stringify(frontmatter_hash)).delete_prefix("---\n")}---"
+        end
+
+        def frontmatter_hash
+          @metadata.to_h.merge bibtex_key: bibtex_key
+        end
+
+        def bibtex_key
+          Downloader::Bibtex.new(@metadata).key
+        end
+
+        def stringify object
+          case object
+          when Hash  then object.to_h { |key, value| [key.to_s, stringify(value)] }
+          when Array then object.map { |item| stringify item }
+          when Data  then stringify object.to_h
+          else object
+          end
+        end
+
+        # JSTOR asks for acknowledgement as the source of Early Journal Content
+        def body
+          <<~MARKDOWN
+
+            # #{@metadata.title}
+
+            #{authors_list}
+
+            - Published: #{@metadata.published}
+            - #{citation}
+            - JSTOR: [#{@metadata.doi}](#{@metadata.jstor_url})
+            - Internet Archive: [jstor-#{@metadata.jstor_id}](#{@metadata.archive_url})
+
+            From JSTOR Early Journal Content, via the Internet Archive.
+          MARKDOWN
+        end
+
+        def authors_list
+          @metadata.authors.map { |author| "- #{author.name}" }.join "\n"
+        end
+
+        def citation
+          "#{@metadata.journal[:name]}, volume #{@metadata.volume}, pages #{@metadata.pages}"
+        end
+      end
+    end
+  end
+end

data/lib/jstor/downloader/metadata/yaml.rb

@@ -0,0 +1,32 @@
+require 'fileutils'
+require 'yaml'
+
+module Jstor
+  module Downloader
+    class Metadata
+      class YAML
+        FILENAME = 'metadata.yaml'.freeze
+
+        def initialize metadata
+          @metadata = metadata
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), ::YAML.dump(stringify(@metadata.to_h))
+        end
+
+        private
+
+        def stringify object
+          case object
+          when Hash  then object.to_h { |key, value| [key.to_s, stringify(value)] }
+          when Array then object.map { |item| stringify item }
+          when Data  then stringify object.to_h
+          else object
+          end
+        end
+      end
+    end
+  end
+end

data/lib/jstor/downloader/metadata.rb

@@ -0,0 +1,21 @@
+module Jstor
+  module Downloader
+    # Field order is the reading order of the metadata sidecars.
+    Metadata = Data.define(
+      :jstor_id,
+      :doi,
+      :jstor_url,
+      :archive_url,
+      :title,
+      :authors,
+      :published,
+      :journal,
+      :volume,
+      :pages,
+      :issn,
+      :language,
+      :publisher,
+      :article_type
+    )
+  end
+end

data/lib/jstor/downloader/metadata_parser.rb

@@ -0,0 +1,43 @@
+require 'json'
+
+module Jstor
+  module Downloader
+    # Parses an archive.org item metadata response (https://archive.org/metadata/jstor-<id>)
+    class MetadataParser
+      def initialize json
+        @json = JSON.parse json
+      end
+
+      def metadata
+        fields = @json['metadata']
+        raise ItemNotFound if fields.nil?
+
+        identifier = Identifier.new fields.fetch('identifier')
+
+        Metadata.new(
+          jstor_id:     identifier.id,
+          doi:          identifier.doi,
+          jstor_url:    "https://www.jstor.org/stable/#{identifier}",
+          archive_url:  "https://archive.org/details/#{identifier.item}",
+          title:        fields['title'],
+          authors:      authors_from(fields['creator']),
+          published:    fields['date'],
+          journal:      { id: fields['journalabbrv'], name: fields['journaltitle'] },
+          volume:       fields['volume'],
+          pages:        fields['pagerange'],
+          issn:         fields['issn'],
+          language:     fields['language'],
+          publisher:    fields['publisher'],
+          article_type: fields['article-type']
+        )
+      end
+
+      private
+
+      # archive.org gives a string for one creator and a list for several
+      def authors_from creator
+        Array(creator).map { Author.new name: it }
+      end
+    end
+  end
+end

data/lib/jstor/downloader/path.rb

@@ -0,0 +1,28 @@
+module Jstor
+  module Downloader
+    class Path
+      def initialize metadata
+        @metadata = metadata
+      end
+
+      def to_s
+        [date_dir, journal, "#{@metadata.jstor_id}-#{slug}"].join '/'
+      end
+
+      private
+
+      # 1907-10-05 becomes 1907/10/05; a year-only or year-month date gives a shorter path
+      def date_dir
+        @metadata.published.to_s.split('-').join '/'
+      end
+
+      def journal
+        @metadata.journal[:id]
+      end
+
+      def slug
+        Slug.new(@metadata.title).to_s
+      end
+    end
+  end
+end

data/lib/jstor/downloader/slug.rb

@@ -0,0 +1,44 @@
+require 'stringex'
+
+module Jstor
+  module Downloader
+    class Slug
+      MAX_LENGTH = 80
+
+      TEX_INLINE_MATH  = /\$[^$]*\$/                # $...$
+      TEX_DISPLAY_MATH = /\\\(.*?\\\)|\\\[.*?\\\]/m # \(...\) or \[...\]
+      TEX_COMMAND      = /\\[a-zA-Z]+\*?/           # \emph, \alpha, etc.
+
+      def initialize title
+        @title = title
+      end
+
+      def to_s
+        truncate strip_tex(@title).to_url
+      end
+
+      private
+
+      def strip_tex string
+        string
+          .gsub(TEX_INLINE_MATH, ' ')
+          .gsub(TEX_DISPLAY_MATH, ' ')
+          .gsub(TEX_COMMAND, ' ')
+      end
+
+      def truncate slug
+        return slug if slug.length <= MAX_LENGTH
+
+        words = slug.split '-'
+        result = +''
+        words.each do |word|
+          break if result.length + 1 + word.length > MAX_LENGTH
+
+          result << '-' unless result.empty?
+          result << word
+        end
+        result
+      end
+    end
+  end
+end

data/lib/jstor/downloader/version.rb

@@ -0,0 +1,5 @@
+module Jstor
+  module Downloader
+    VERSION = '0.1.0'.freeze
+  end
+end

data/lib/jstor/downloader.rb

@@ -0,0 +1,24 @@
+require_relative 'downloader/archive'
+require_relative 'downloader/author'
+require_relative 'downloader/bibtex'
+require_relative 'downloader/cli'
+require_relative 'downloader/client'
+require_relative 'downloader/error'             # before errors below that subclass Error
+require_relative 'downloader/http_error'        # after error
+require_relative 'downloader/identifier'        # after error
+require_relative 'downloader/item_file'
+require_relative 'downloader/item_not_found'    # after error
+require_relative 'downloader/metadata'          # before metadata/*: they reopen class Metadata
+require_relative 'downloader/metadata/bibtex'   # after metadata
+require_relative 'downloader/metadata/json'     # after metadata
+require_relative 'downloader/metadata/markdown' # after metadata
+require_relative 'downloader/metadata/yaml'     # after metadata
+require_relative 'downloader/metadata_parser'
+require_relative 'downloader/path'
+require_relative 'downloader/slug'
+require_relative 'downloader/version'
+
+module Jstor
+  module Downloader
+  end
+end

data/sig/jstor/downloader.rbs

@@ -0,0 +1,5 @@
+module Jstor
+  module Downloader
+    VERSION: String
+  end
+end

metadata

@@ -0,0 +1,119 @@
+--- !ruby/object:Gem::Specification
+name: jstor-dl
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Shane Becker
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: http
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+- !ruby/object:Gem::Dependency
+  name: stringex
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.8'
+description: |
+  Command line tool and Ruby library for archiving JSTOR Early Journal Content articles
+  as PDFs and OCR plaintext with sidecar metadata. Fetches only from the Internet Archive's copy,
+  never from jstor.org.
+email:
+- veganstraightedge@gmail.com
+executables:
+- jstor-dl
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.md
+- README.md
+- exe/jstor-dl
+- lib/jstor/downloader.rb
+- lib/jstor/downloader/archive.rb
+- lib/jstor/downloader/author.rb
+- lib/jstor/downloader/bibtex.rb
+- lib/jstor/downloader/cli.rb
+- lib/jstor/downloader/client.rb
+- lib/jstor/downloader/error.rb
+- lib/jstor/downloader/http_error.rb
+- lib/jstor/downloader/identifier.rb
+- lib/jstor/downloader/item_file.rb
+- lib/jstor/downloader/item_not_found.rb
+- lib/jstor/downloader/metadata.rb
+- lib/jstor/downloader/metadata/bibtex.rb
+- lib/jstor/downloader/metadata/json.rb
+- lib/jstor/downloader/metadata/markdown.rb
+- lib/jstor/downloader/metadata/yaml.rb
+- lib/jstor/downloader/metadata_parser.rb
+- lib/jstor/downloader/path.rb
+- lib/jstor/downloader/slug.rb
+- lib/jstor/downloader/version.rb
+- sig/jstor/downloader.rbs
+homepage: https://github.com/xoengineering/jstor-dl
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/xoengineering/jstor-dl
+  source_code_uri: https://github.com/xoengineering/jstor-dl
+  bug_tracker_uri: https://github.com/xoengineering/jstor-dl/issues
+  changelog_uri: https://github.com/xoengineering/jstor-dl/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 4.0.7
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.21
+specification_version: 4
+summary: Download JSTOR's public-domain Early Journal Content from archive.org for
+  offline archives.
+test_files: []