arxiv-dl 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5b80906292c193199cba698da4d7e85ff927e41af088c2452b1b3c8ee3234997
+  data.tar.gz: bd732114a37495687cd23d62bbdc67cf645db682c8b3302cc1eb283c64ebd356
+SHA512:
+  metadata.gz: 76153d238bfc2dddf1d9343e130e10beac24f508193c7429c8bbbe42a603a4451996bfb88dc1b592b2d7e34e31055e313d69c2f09235f28c714f03fa25288189
+  data.tar.gz: 7c508e83063cbf8d6832d34b9d55a89c7fa8ba55c60f97cbfb1aaab08a5c0d4489a2af359bf55d0a483416ec390ed8402ee7f42114cf15365c9ed647bb53028f

data/CHANGELOG.md

@@ -0,0 +1,38 @@
+## [0.1.0]
+
+First release. Per-paper offline archive of arxiv.org papers, with PDF, abstract HTML, full HTML version (with relative-path images and a deduplicated cross-paper shared assets cache), TeX source, and four metadata sidecar files.
+
+### CLI
+
+- `arxiv-dl <ARXIV_ID_OR_URL>...` accepts bare IDs, prefixed IDs, legacy IDs (`cs/0002001`), versioned IDs (`2508.16190v1`), and `/abs`, `/pdf`, `/html` URLs.
+- Flags: `-p/--path PATH`, `--rate-limit SECONDS`, `-v/--verbose`, `-q/--quiet`, `--version`, `-h/--help`. `-v` and `-q` are mutually exclusive.
+- ENV fallbacks: `ARXIV_DOWNLOAD_PATH`, `ARXIV_RATE_LIMIT`. Precedence: CLI flag > ENV > default.
+- Default download path: `$HOME/Downloads/ArXiv_Papers`. Default rate limit: 3 seconds (arxiv etiquette).
+- Verbose mode prints `==> Downloading <id>` step lines and `==> GET <url> (<bytes> bytes)` per HTTP request.
+
+### Per-paper output layout
+
+```txt
+$ARXIV_DOWNLOAD_PATH/
+  _shared/<host>/<path>           # deduplicated cross-paper static assets (CSS, JS, fonts)
+  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
+    <arxiv-id>.pdf
+    <arxiv-id>-abstract.html
+    metadata.{md,yaml,json,bib}
+    html/<arxiv-id>.html + x1.png, x2.png, ...
+    src/*.tex, *.bbl, ...
+```
+
+### Library API
+
+- `Arxiv::Downloader::Identifier.new(input).id` / `.version` parses every supported input form.
+- `Arxiv::Downloader::Client.new(rate_limit:, log:).get(url)` is the rate-limited HTTP wrapper.
+- `Arxiv::Downloader::FeedParser.new(xml).metadata` parses the arxiv Atom feed into a `Metadata` value object.
+- `Arxiv::Downloader::Bibtex.new(metadata, client:)` produces synthesized or fetched BibTeX.
+- `Arxiv::Downloader::PDF`, `AbstractPage`, `HTMLArchive`, `SourceArchive`, `AssetsCache` download individual artifacts.
+- `Arxiv::Downloader::Metadata::{Markdown,YAML,JSON,Bibtex}.new(metadata).write(to: dir)` produce the four sidecar files.
+- `Arxiv::Downloader::Archive.new(identifier, root:, client:).run` orchestrates the full per-paper pipeline and returns the paper directory path.
+
+### Dependencies
+
+Runtime: `feedjira`, `http`, `nokogiri`, `ostruct`, `stringex`. Stdlib only for the rest (`optparse`, `fileutils`, `pathname`, `yaml`, `json`, `rubygems/package`, `zlib`).

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,10 @@
+Code of Conduct
+
+"arxiv-dl" follows [The Ruby Community Conduct Guideline](https://www.ruby-lang.org/en/conduct) in all "collaborative space", which is defined as community communications channels (such as mailing lists, submitted patches, commit comments, etc.):
+
+- Participants will be tolerant of opposing views.
+- Participants must ensure that their language and actions are free of personal attacks and disparaging personal remarks.
+- When interpreting the words and actions of others, participants should always assume good intentions.
+- Behaviour which can be reasonably considered harassment will not be tolerated.
+
+If you have any concerns about behaviour within this project, please contact us at ["veganstraightedge@gmail.com"](mailto:"veganstraightedge@gmail.com").

data/LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Shane Becker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,141 @@
+# arxiv-dl
+
+Ruby gem to download papers and metadata from [arxiv.org](https://arxiv.org) for offline archives.
+
+For each paper it produces a self-contained per-paper folder with:
+
+- The PDF
+- The single-page abstract HTML
+- The full HTML version (with images and a deduplicated, cross-paper shared assets cache)
+- The TeX/LaTeX source tarball, extracted
+- Four sidecar metadata files: `metadata.md`, `metadata.yaml`, `metadata.json`, `metadata.bib`
+
+## Installation
+
+```sh
+gem install arxiv-dl
+```
+
+Or in a `Gemfile`:
+
+```ruby
+gem 'arxiv-dl'
+```
+
+## CLI usage
+
+```sh
+arxiv-dl <ARXIV_ID_OR_URL> [<ARXIV_ID_OR_URL>...]
+```
+
+Accepted input forms:
+
+| Form         | Example                                |
+| ------------ | -------------------------------------- |
+| Bare ID      | `2508.16190`                           |
+| Prefixed ID  | `arXiv:2508.16190`                     |
+| Legacy ID    | `cs/0002001`, `alg-geom/9708001`       |
+| Versioned ID | `2508.16190v1`                         |
+| Abstract URL | `https://arxiv.org/abs/2508.16190`     |
+| PDF URL      | `https://arxiv.org/pdf/2508.16190.pdf` |
+| HTML URL     | `https://arxiv.org/html/2508.16190`    |
+
+### Flags
+
+| Flag                     | Description                                              |
+| ------------------------ | -------------------------------------------------------- |
+| `-p PATH`, `--path PATH` | Root download directory                                  |
+| `--rate-limit SECONDS`   | Seconds between HTTP requests; `0` disables throttling   |
+| `-v`, `--verbose`        | Print step lines and per-request URL/byte logs to stdout |
+| `-q`, `--quiet`          | Print nothing; success/failure conveyed via exit code    |
+| `--version`              | Print the gem version and exit                           |
+| `-h`, `--help`           | Print help and exit                                      |
+
+`-v` and `-q` are mutually exclusive.
+
+### Environment variables
+
+| Variable              | Effect                                                                          |
+| --------------------- | ------------------------------------------------------------------------------- |
+| `ARXIV_DOWNLOAD_PATH` | Root download directory (default: `$HOME/Downloads/ArXiv_Papers`)               |
+| `ARXIV_RATE_LIMIT`    | Seconds between HTTP requests (default: `3`, per arxiv etiquette; `0` disables) |
+
+Precedence: CLI flag > ENV var > default.
+
+### Examples
+
+Download a single paper to the default location:
+
+```sh
+arxiv-dl 2508.16190
+```
+
+Download to a custom directory:
+
+```sh
+arxiv-dl --path ~/Archives/arxiv 2508.16190
+```
+
+Download multiple papers, verbose:
+
+```sh
+arxiv-dl -v 2508.16190 1207.7214 cs/0002001
+```
+
+Disable rate limiting (when running against a local mirror, etc):
+
+```sh
+arxiv-dl --rate-limit 0 2508.16190
+```
+
+## Output layout
+
+```txt
+$ARXIV_DOWNLOAD_PATH/                   # default: $HOME/Downloads/ArXiv_Papers
+  _shared/                              # cross-paper deduplicated static assets
+    arxiv.org/static/...
+    cdn.jsdelivr.net/...
+  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
+    <arxiv-id>.pdf
+    <arxiv-id>-abstract.html
+    metadata.md                         # YAML frontmatter + Markdown body
+    metadata.yaml
+    metadata.json
+    metadata.bib                        # upstream BibTeX, falls back to synthesized
+    html/
+      <arxiv-id>.html                   # path-rewritten to local assets
+      x1.png, x2.png, ...               # paper-specific images
+    src/
+      *.tex, *.bbl, ...                 # extracted from /src/<id> tarball
+```
+
+`YYYY/MM/DD` is the original submission date. `<primary_category>` is from the paper's metadata (`cs.CL`, `math.NT`, etc). `<slug>` is derived from the paper title (Unicode → ASCII, hyphenated, truncated to 80 chars at a word boundary).
+
+For legacy IDs containing `/` (e.g. `cs/0002001`), the slash is replaced with `-` in the directory name (`cs-0002001-...`).
+
+## Library usage
+
+```ruby
+require 'arxiv/downloader'
+
+identifier = Arxiv::Downloader::Identifier.new '2508.16190'
+client     = Arxiv::Downloader::Client.new                # 3-second rate limit by default
+path       = Arxiv::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run
+# => "/tmp/papers/2025/08/22/cs.CL/2508.16190-comicscene154-a-scene-dataset-for-comic-analysis"
+```
+
+## Development
+
+```sh
+script/setup    # install dependencies
+script/test     # run specs and rubocop
+script/console  # interactive prompt
+```
+
+## License
+
+MIT — see [LICENSE.md](LICENSE.md).
+
+## Code of Conduct
+
+See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

data/exe/arxiv-dl

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'arxiv/downloader'
+
+exit Arxiv::Downloader::CLI.new(ARGV).run

data/lib/arxiv/downloader/abstract_page.rb

@@ -0,0 +1,20 @@
+module Arxiv
+  module Downloader
+    class AbstractPage
+      def initialize identifier, client:
+        @identifier = identifier
+        @client     = client
+      end
+
+      def download to:
+        File.write to, @client.get(url).to_s
+      end
+
+      private
+
+      def url
+        "https://arxiv.org/abs/#{@identifier.id}"
+      end
+    end
+  end
+end

data/lib/arxiv/downloader/archive.rb

@@ -0,0 +1,68 @@
+require 'fileutils'
+
+module Arxiv
+  module Downloader
+    class Archive
+      def initialize identifier, root:, client: Client.new
+        @identifier = identifier
+        @root       = root
+        @client     = client
+      end
+
+      def run
+        FileUtils.mkdir_p paper_dir
+
+        download_pdf
+        download_abstract
+        download_html_archive
+        download_source_archive
+        write_sidecars
+
+        paper_dir
+      end
+
+      private
+
+      def metadata
+        @metadata ||= FeedParser.new(@client.get(atom_url).to_s).metadata
+      end
+
+      def atom_url
+        "https://export.arxiv.org/api/query?id_list=#{@identifier.id}"
+      end
+
+      def paper_dir
+        @paper_dir ||= File.join @root, Path.new(metadata).to_s
+      end
+
+      def download_pdf
+        PDF.new(@identifier, client: @client).download to: File.join(paper_dir, "#{@identifier.id}.pdf")
+      end
+
+      def download_abstract
+        AbstractPage.new(@identifier, client: @client)
+                    .download to: File.join(paper_dir, "#{@identifier.id}-abstract.html")
+      end
+
+      def download_html_archive
+        HTMLArchive.new(@identifier, client: @client, assets_cache: assets_cache)
+                   .download to: File.join(paper_dir, 'html')
+      end
+
+      def download_source_archive
+        SourceArchive.new(@identifier, client: @client).download to: File.join(paper_dir, 'src')
+      end
+
+      def assets_cache
+        @assets_cache ||= AssetsCache.new root: @root, client: @client
+      end
+
+      def write_sidecars
+        Metadata::Markdown.new(metadata).write to: paper_dir
+        Metadata::YAML.new(metadata).write     to: paper_dir
+        Metadata::JSON.new(metadata).write     to: paper_dir
+        Metadata::Bibtex.new(metadata, client: @client).write to: paper_dir
+      end
+    end
+  end
+end

data/lib/arxiv/downloader/assets_cache.rb

@@ -0,0 +1,32 @@
+require 'fileutils'
+require 'uri'
+
+module Arxiv
+  module Downloader
+    class AssetsCache
+      SHARED_DIR = '_shared'.freeze
+
+      def initialize root:, client:
+        @root   = root
+        @client = client
+      end
+
+      def fetch url
+        path = local_path_for url
+        return path if File.exist? path
+
+        FileUtils.mkdir_p File.dirname(path)
+        File.binwrite path, @client.get(url).to_s
+
+        path
+      end
+
+      private
+
+      def local_path_for url
+        uri = URI.parse url
+        File.join @root, SHARED_DIR, uri.host, uri.path.delete_prefix('/')
+      end
+    end
+  end
+end

data/lib/arxiv/downloader/bibtex.rb

@@ -0,0 +1,55 @@
+module Arxiv
+  module Downloader
+    class Bibtex
+      def initialize metadata, client: nil
+        @metadata = metadata
+        @client   = client
+      end
+
+      def synthesize
+        <<~BIBTEX
+          @misc{#{key},
+            title={#{@metadata.title}},
+            author={#{authors}},
+            year={#{@metadata.published.year}},
+            eprint={#{@metadata.arxiv_id}},
+            archivePrefix={arXiv},
+            primaryClass={#{@metadata.primary_category[:id]}},
+            url={#{@metadata.arxiv_url}},
+          }
+        BIBTEX
+      end
+
+      def fetch
+        return nil if @client.nil?
+
+        response = @client.get url
+        return nil unless response.status.success?
+
+        response.to_s
+      end
+
+      def to_s
+        fetch || synthesize
+      end
+
+      def key
+        last_name  = @metadata.authors.first.split.last.downcase.gsub(/[^a-z]/, '')
+        year       = @metadata.published.year
+        title_word = Slug.new(@metadata.title).to_s.split('-').first
+
+        "#{last_name}#{year}#{title_word}"
+      end
+
+      private
+
+      def url
+        "https://arxiv.org/bibtex/#{@metadata.arxiv_id}"
+      end
+
+      def authors
+        @metadata.authors.join ' and '
+      end
+    end
+  end
+end

data/lib/arxiv/downloader/categories.rb

@@ -0,0 +1,20 @@
+require 'yaml'
+
+module Arxiv
+  module Downloader
+    class Categories
+      DATA_PATH = File.expand_path('categories.yaml', __dir__).freeze
+
+      def initialize
+        @data = YAML.load_file(DATA_PATH)
+      end
+
+      def lookup id
+        entry = @data[id]
+        return nil if entry.nil?
+
+        { id: id, name: entry['name'], group: entry['group'], description: entry['description'] }
+      end
+    end
+  end
+end