dl-core 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: cf4c6083775de6f5cfb392c154831eab236fca2b9981dd0e43acd72c96c52d0f
+  data.tar.gz: 87ed15f963c4e8e6c571ec7ad9f2a10f9f4c79b1badb166911e4c106b7da22b2
+SHA512:
+  metadata.gz: f080be0fd4f22e3c96bc3d7c2581bf4cd4bdf6e95bc85f5cc501c0c5be0b379513e4a0ac96f509152a96bb074c556dc7956f2a902becc98220d14ba99c9cee31
+  data.tar.gz: af8f7747b957236d540661d1440e3b7eddd78f83f04a4017ec62da54498602a55e1f947bb43e60b8e9283ef96e2b8c1558ed7c790398fe4c0e66aa19c15d2cb5

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+## [0.1.0]
+
+First version, extracted from arxiv-dl 0.3.0 and jstor-dl 0.1.0.
+
+- `DL::Core::Client`: rate-limited HTTP client with timeouts and retries on 429/503. The User-Agent is now an argument, so each gem names itself.
+- `DL::Core::Error` (base for every gem's errors) and `DL::Core::HTTPError`.
+- `DL::Core::Author` and `DL::Core::Slug`.
+- `DL::Core::Sidecar::YAML`, `JSON`, and `Markdown` metadata writers for any metadata `Data` object.
+- `DL::Core::CLI`: the shared command line; each gem subclasses it with its name, defaults, identifier, and archive.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,83 @@
+# Contributor Covenant 3.0 Code of Conduct
+
+## Our Pledge
+
+We pledge to make our community welcoming, safe, and equitable for all.
+
+We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant.
+
+## Encouraged Behaviors
+
+While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language.
+
+With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including:
+
+1. Respecting the **purpose of our community**, our activities, and our ways of gathering.
+2. Engaging **kindly and honestly** with others.
+3. Respecting **different viewpoints** and experiences.
+4. **Taking responsibility** for our actions and contributions.
+5. Gracefully giving and accepting **constructive feedback**.
+6. Committing to **repairing harm** when it occurs.
+7. Behaving in other ways that promote and sustain the **well-being of our community**.
+
+## Restricted Behaviors
+
+We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct.
+
+1. **Harassment.** Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop.
+2. **Character attacks.** Making insulting, demeaning, or pejorative comments directed at a community member or group of people.
+3. **Stereotyping or discrimination.** Characterizing anyone’s personality or behavior on the basis of immutable identities or traits.
+4. **Sexualization.** Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community.
+5. **Violating confidentiality**. Sharing or acting on someone's personal or private information without their permission.
+6. **Endangerment.** Causing, encouraging, or threatening violence or other harm toward any person or group.
+7. Behaving in other ways that **threaten the well-being** of our community.
+
+### Other Restrictions
+
+1. **Misleading identity.** Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions.
+2. **Failing to credit sources.** Not properly crediting the sources of content you contribute.
+3. **Promotional materials**. Sharing marketing or other commercial content in a way that is outside the norms of the community.
+4. **Irresponsible communication.** Failing to responsibly present content which includes, links or describes any other restricted behaviors.
+
+## Reporting an Issue
+
+Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm.
+
+When an incident does occur, it is important to report it promptly. To report a possible violation, **email the maintainer at [veganstraightedge@gmail.com](mailto:veganstraightedge@gmail.com).**
+
+Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution.
+
+## Addressing and Repairing Harm
+
+If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped.
+
+1. Warning
+   1. Event: A violation involving a single incident or series of incidents.
+   2. Consequence: A private, written warning from the Community Moderators.
+   3. Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations.
+2. Temporarily Limited Activities
+   1. Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation.
+   2. Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members.
+   3. Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over.
+3. Temporary Suspension
+   1. Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation.
+   2. Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions.
+   3. Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted.
+4. Permanent Ban
+   1. Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member.
+   2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior.
+   3. Repair: There is no possible repair in cases of this severity.
+
+This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at [https://www.contributor-covenant.org/version/3/0/](https://www.contributor-covenant.org/version/3/0/).
+
+Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/)
+
+For answers to common questions about Contributor Covenant, see the FAQ at [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq). Translations are provided at [https://www.contributor-covenant.org/translations](https://www.contributor-covenant.org/translations). Additional enforcement and community guideline resources can be found at [https://www.contributor-covenant.org/resources](https://www.contributor-covenant.org/resources). The enforcement ladder was inspired by the work of [Mozilla’s code of conduct team](https://github.com/mozilla/inclusion).

data/LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Shane Becker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,64 @@
+# dl-core
+
+Shared foundation for the `<site>-dl` gems that build offline archives: [arxiv-dl](https://github.com/xoengineering/arxiv-dl), [jstor-dl](https://github.com/xoengineering/jstor-dl), and more.
+
+## What's in it
+
+| Class                                  | Purpose                                                                                           |
+| -------------------------------------- | ------------------------------------------------------------------------------------------------- |
+| `DL::Core::Client`                     | Polite HTTP: User-Agent naming the gem, minimum interval between requests, timeouts, retries on 429/503 (honoring `Retry-After`) |
+| `DL::Core::Error`, `DL::Core::HTTPError` | Base error for every gem; `HTTPError` for non-success responses (`status`, `url`)               |
+| `DL::Core::Author`                     | `name` and `affiliations`                                                                         |
+| `DL::Core::Slug`                       | Title → URL-safe slug, truncated at a word boundary                                               |
+| `DL::Core::Sidecar::YAML`, `JSON`      | `metadata.yaml` / `metadata.json` from any metadata `Data` object                                 |
+| `DL::Core::Sidecar::Markdown`          | `metadata.md`: YAML frontmatter from the metadata, plus a body each gem writes                    |
+| `DL::Core::CLI`                        | The shared command line: targets as args or `--input FILE\|-`, `-p`, `--rate-limit`, `-v`, `-q`, per-target errors, exit 1 on any failure |
+
+## Building a `<site>-dl` gem on it
+
+Each gem's errors subclass `DL::Core::Error`, and its CLI subclasses `DL::Core::CLI`:
+
+```ruby
+module Example
+  module Downloader
+    class Error < DL::Core::Error; end
+
+    class CLI < DL::Core::CLI
+      def program = 'example-dl'
+
+      def target_name = 'EXAMPLE_ID_OR_URL'
+
+      # EXAMPLE_DOWNLOAD_PATH, EXAMPLE_RATE_LIMIT
+      def env_prefix = 'EXAMPLE'
+
+      def default_path = File.join(Dir.home, 'Downloads', 'Example_Papers')
+
+      def version = VERSION
+
+      def user_agent = "example-dl/#{VERSION} (+https://github.com/xoengineering/example-dl)"
+
+      def identifier_for(target) = Identifier.new(target)
+
+      def archive_for(identifier, root:, client:) = Archive.new(identifier, root:, client:)
+    end
+  end
+end
+```
+
+`identifier_for` raises a `DL::Core::Error` subclass for input the gem doesn't recognize; the object it returns responds to `id`. `archive_for` returns an object whose `run` archives the paper and returns its folder.
+
+## Development
+
+```sh
+script/setup    # install dependencies
+script/test     # run specs and rubocop
+script/console  # interactive prompt
+```
+
+## License
+
+MIT — see [LICENSE.md](LICENSE.md).
+
+## Code of Conduct
+
+This project follows the [Contributor Covenant](https://www.contributor-covenant.org/version/3/0/) 3.0 — see [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

data/lib/dl/core/author.rb

@@ -0,0 +1,11 @@
+module DL
+  module Core
+    Author = Data.define :name, :affiliations do
+      def initialize name:, affiliations: []
+        super
+      end
+
+      def to_s = name
+    end
+  end
+end

data/lib/dl/core/cli.rb

@@ -0,0 +1,135 @@
+require 'optparse'
+
+module DL
+  module Core
+    # The shared <site>-dl command line. Each gem subclasses it and defines:
+    #
+    #   program                                  "jstor-dl"
+    #   target_name                              "JSTOR_ID_OR_URL", for the usage line
+    #   env_prefix                               "JSTOR", for JSTOR_DOWNLOAD_PATH and JSTOR_RATE_LIMIT
+    #   default_path                             where downloads go without -p or the ENV var
+    #   version                                  the gem's version, for --version
+    #   user_agent                               for requests, naming the gem
+    #   identifier_for(target)                   parses one target; raises an Error subclass if invalid
+    #   archive_for(identifier, root:, client:)  an object whose #run archives it and returns its folder
+    class CLI
+      def initialize argv, stderr: $stderr, stdin: $stdin, stdout: $stdout
+        @argv   = argv
+        @stderr = stderr
+        @stdin  = stdin
+        @stdout = stdout
+      end
+
+      def run
+        options = parse
+        return options.fetch(:exit_status) if options.key? :exit_status
+
+        return error_with usage    if options[:targets].empty?
+        return error_with conflict if options[:verbose] && options[:quiet]
+
+        failures = download_each options
+        failures.zero? ? 0 : 1
+      end
+
+      def program = raise(NotImplementedError, "#{self.class} must define #program")
+
+      def target_name = raise(NotImplementedError, "#{self.class} must define #target_name")
+
+      def env_prefix = raise(NotImplementedError, "#{self.class} must define #env_prefix")
+
+      def default_path = raise(NotImplementedError, "#{self.class} must define #default_path")
+
+      def version = raise(NotImplementedError, "#{self.class} must define #version")
+
+      def user_agent = raise(NotImplementedError, "#{self.class} must define #user_agent")
+
+      def identifier_for(_target) = raise(NotImplementedError, "#{self.class} must define #identifier_for")
+
+      def archive_for(_identifier, root:, client:) = raise(NotImplementedError, "#{self.class} must define #archive_for")
+
+      def client_for(rate_limit:, log:) = Client.new(user_agent:, rate_limit:, log:)
+
+      private
+
+      def usage
+        "Usage: #{program} [options] <#{target_name}> [<#{target_name}>...]"
+      end
+
+      def conflict
+        '-v and -q are mutually exclusive'
+      end
+
+      def parse
+        options = { targets: [], verbose: false, quiet: false }
+        parser  = build_parser options
+
+        begin
+          parser.parse! @argv
+          options[:targets] = @argv + input_targets(options[:input])
+        rescue OptionParser::ParseError, SystemCallError => e
+          @stderr.puts e.message
+          return { exit_status: 1 }
+        end
+
+        options[:path]       ||= ENV["#{env_prefix}_DOWNLOAD_PATH"] || default_path
+        options[:rate_limit] ||= (ENV["#{env_prefix}_RATE_LIMIT"] || Client::DEFAULT_RATE_LIMIT).to_i
+        options
+      end
+
+      def build_parser options
+        OptionParser.new do |parser|
+          parser.banner = usage
+          parser.on('-i FILE', '--input FILE')       { |value| options[:input] = value }
+          parser.on('-p PATH', '--path PATH')        { |value| options[:path] = value }
+          parser.on('--rate-limit SECONDS', Integer) { |value| options[:rate_limit] = value }
+          parser.on('-v', '--verbose')               { options[:verbose] = true }
+          parser.on('-q', '--quiet')                 { options[:quiet]   = true }
+          parser.on('--version') do
+            @stdout.puts version
+            options[:exit_status] = 0
+          end
+          parser.on('-h', '--help') do
+            @stdout.puts parser.help
+            options[:exit_status] = 0
+          end
+        end
+      end
+
+      # one target per line from FILE, or stdin for "-"; blank lines and # comments skipped
+      def input_targets input
+        return [] if input.nil?
+
+        text = input == '-' ? @stdin.read : File.read(input)
+        text.lines.map(&:strip).reject { it.empty? || it.start_with?('#') }
+      end
+
+      def error_with message
+        @stderr.puts message
+        1
+      end
+
+      def download_each options
+        client = client_for rate_limit: options[:rate_limit], log: (options[:verbose] ? @stdout : nil)
+
+        failures = 0
+        options[:targets].each do |target|
+          failures += 1 unless download_one(target, client:, options:)
+        end
+        failures
+      end
+
+      # true on success; reports the failure and returns false otherwise
+      def download_one target, client:, options:
+        identifier = identifier_for target
+        @stdout.puts "==> Downloading #{identifier.id}" if options[:verbose]
+
+        path = archive_for(identifier, root: options[:path], client: client).run
+        @stdout.puts path unless options[:quiet]
+        true
+      rescue Error, HTTP::Error => e
+        @stderr.puts "#{target}: #{e.message}"
+        false
+      end
+    end
+  end
+end

data/lib/dl/core/client.rb

@@ -0,0 +1,85 @@
+require 'http'
+
+module DL
+  module Core
+    # Polite HTTP: a User-Agent naming the calling gem, a minimum interval between
+    # requests, timeouts, retries on 429/503, and HTTPError on other failures.
+    class Client
+      DEFAULT_RATE_LIMIT = 3
+      TIMEOUTS           = { connect: 10, read: 60, write: 10 }.freeze # seconds, per operation
+      MAX_RETRIES        = 3
+      RETRY_BACKOFF      = 10 # seconds before the first retry. doubles on each retry.
+      RETRYABLE_STATUSES = [429, 503].freeze
+
+      attr_reader :rate_limit, :user_agent
+
+      # user_agent names the calling gem, e.g. "jstor-dl/0.1.0 (+https://github.com/xoengineering/jstor-dl)"
+      def initialize user_agent:, log: nil, rate_limit: DEFAULT_RATE_LIMIT
+        @log        = log
+        @rate_limit = rate_limit
+        @user_agent = user_agent
+      end
+
+      def get url
+        retries = 0
+
+        loop do
+          response = request url
+          return response if response.status.success?
+          raise http_error(url, response) unless retryable? response, retries
+
+          retries += 1
+          wait_before_retry response, retries
+        end
+      end
+
+      private
+
+      def request url
+        throttle
+        response = HTTP.timeout(TIMEOUTS).headers('User-Agent' => user_agent).follow.get(url)
+        @last_request_at = Time.now
+        log_request url, response
+        response
+      end
+
+      def http_error url, response
+        HTTPError.new status: response.status.code, url: url, reason: response.status.reason
+      end
+
+      def retryable? response, retries
+        RETRYABLE_STATUSES.include?(response.status.code) && retries < MAX_RETRIES
+      end
+
+      def wait_before_retry response, retries
+        seconds = retry_after(response) || (RETRY_BACKOFF * (2**(retries - 1)))
+        @log&.puts "==> #{response.status}. Retrying in #{seconds}s"
+        sleep seconds
+      end
+
+      # Retry-After in delay-seconds form. The HTTP-date form falls back to backoff.
+      def retry_after response
+        value = response.headers['Retry-After']
+        return if value.nil?
+
+        Integer(value, exception: false)
+      end
+
+      def log_request url, response
+        return if @log.nil?
+
+        @log.puts "==> GET #{url} (#{response.body.to_s.bytesize} bytes)"
+      end
+
+      def throttle
+        return if @rate_limit.zero?
+        return if @last_request_at.nil?
+
+        elapsed = Time.now - @last_request_at
+        return if elapsed >= @rate_limit
+
+        sleep(@rate_limit - elapsed)
+      end
+    end
+  end
+end

data/lib/dl/core/error.rb

@@ -0,0 +1,6 @@
+module DL
+  module Core
+    # each gem's own Error subclasses this, so the shared CLI can report any gem's failures
+    class Error < StandardError; end
+  end
+end

data/lib/dl/core/http_error.rb

@@ -0,0 +1,14 @@
+module DL
+  module Core
+    class HTTPError < Error
+      attr_reader :status, :url
+
+      def initialize status:, url:, reason: nil
+        @status = status
+        @url    = url
+
+        super("GET #{url} failed: #{[status, reason].compact.join ' '}")
+      end
+    end
+  end
+end

data/lib/dl/core/sidecar/json.rb

@@ -0,0 +1,39 @@
+require 'date'
+require 'fileutils'
+require 'json'
+
+module DL
+  module Core
+    module Sidecar
+      # metadata.json: every field of a metadata Data object, dates as ISO 8601 strings
+      class JSON
+        FILENAME = 'metadata.json'.freeze
+
+        def initialize metadata
+          @metadata = metadata
+        end
+
+        def to_s
+          "#{::JSON.pretty_generate(serialize(@metadata.to_h))}\n"
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), to_s
+        end
+
+        private
+
+        def serialize object
+          case object
+          when Hash       then object.to_h { |key, value| [key.to_s, serialize(value)] }
+          when Array      then object.map { |item| serialize item }
+          when Data       then serialize object.to_h
+          when Date, Time then object.iso8601
+          else object
+          end
+        end
+      end
+    end
+  end
+end

data/lib/dl/core/sidecar/markdown.rb

@@ -0,0 +1,36 @@
+require 'fileutils'
+
+module DL
+  module Core
+    module Sidecar
+      # metadata.md: YAML frontmatter from the metadata (plus any extras), then a
+      # Markdown body that each gem writes for its own source
+      class Markdown
+        FILENAME = 'metadata.md'.freeze
+
+        def initialize metadata, body:, extras: {}
+          @metadata = metadata
+          @body     = body
+          @extras   = extras
+        end
+
+        def to_s
+          "#{frontmatter}\n#{@body}"
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), to_s
+        end
+
+        private
+
+        def frontmatter
+          yaml = YAML.new(@metadata, extras: @extras).to_s.delete_prefix "---\n"
+
+          "---\n#{yaml}---"
+        end
+      end
+    end
+  end
+end

data/lib/dl/core/sidecar/yaml.rb

@@ -0,0 +1,38 @@
+require 'fileutils'
+require 'yaml'
+
+module DL
+  module Core
+    module Sidecar
+      # metadata.yaml: every field of a metadata Data object (plus any extras), in field order
+      class YAML
+        FILENAME = 'metadata.yaml'.freeze
+
+        def initialize metadata, extras: {}
+          @metadata = metadata
+          @extras   = extras
+        end
+
+        def to_s
+          ::YAML.dump stringify(@metadata.to_h.merge(@extras))
+        end
+
+        def write to:
+          FileUtils.mkdir_p to
+          File.write File.join(to, FILENAME), to_s
+        end
+
+        private
+
+        def stringify object
+          case object
+          when Hash  then object.to_h { |key, value| [key.to_s, stringify(value)] }
+          when Array then object.map { |item| stringify item }
+          when Data  then stringify object.to_h
+          else object
+          end
+        end
+      end
+    end
+  end
+end

data/lib/dl/core/slug.rb

@@ -0,0 +1,44 @@
+require 'stringex'
+
+module DL
+  module Core
+    class Slug
+      MAX_LENGTH = 80
+
+      TEX_INLINE_MATH  = /\$[^$]*\$/                # $...$
+      TEX_DISPLAY_MATH = /\\\(.*?\\\)|\\\[.*?\\\]/m # \(...\) or \[...\]
+      TEX_COMMAND      = /\\[a-zA-Z]+\*?/           # \emph, \alpha, etc.
+
+      def initialize title
+        @title = title
+      end
+
+      def to_s
+        truncate strip_tex(@title).to_url
+      end
+
+      private
+
+      def strip_tex string
+        string
+          .gsub(TEX_INLINE_MATH, ' ')
+          .gsub(TEX_DISPLAY_MATH, ' ')
+          .gsub(TEX_COMMAND, ' ')
+      end
+
+      def truncate slug
+        return slug if slug.length <= MAX_LENGTH
+
+        words = slug.split '-'
+        result = +''
+        words.each do |word|
+          break if result.length + 1 + word.length > MAX_LENGTH
+
+          result << '-' unless result.empty?
+          result << word
+        end
+        result
+      end
+    end
+  end
+end

data/lib/dl/core/version.rb

@@ -0,0 +1,5 @@
+module DL
+  module Core
+    VERSION = '0.1.0'.freeze
+  end
+end

data/lib/dl/core.rb

@@ -0,0 +1,15 @@
+require_relative 'core/author'
+require_relative 'core/cli'
+require_relative 'core/client'
+require_relative 'core/error'      # before errors below that subclass Error
+require_relative 'core/http_error' # after error
+require_relative 'core/sidecar/json'
+require_relative 'core/sidecar/markdown'
+require_relative 'core/sidecar/yaml'
+require_relative 'core/slug'
+require_relative 'core/version'
+
+module DL
+  module Core
+  end
+end

data/sig/dl/core.rbs

@@ -0,0 +1,5 @@
+module DL
+  module Core
+    VERSION: String
+  end
+end

metadata

@@ -0,0 +1,107 @@
+--- !ruby/object:Gem::Specification
+name: dl-core
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Shane Becker
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: http
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+- !ruby/object:Gem::Dependency
+  name: stringex
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.8'
+description: |
+  Polite rate-limited HTTP client with retries and timeouts, sidecar metadata writers,
+  title slugs, and a shared CLI for the <site>-dl family of gems that build offline archives.
+email:
+- veganstraightedge@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.md
+- README.md
+- lib/dl/core.rb
+- lib/dl/core/author.rb
+- lib/dl/core/cli.rb
+- lib/dl/core/client.rb
+- lib/dl/core/error.rb
+- lib/dl/core/http_error.rb
+- lib/dl/core/sidecar/json.rb
+- lib/dl/core/sidecar/markdown.rb
+- lib/dl/core/sidecar/yaml.rb
+- lib/dl/core/slug.rb
+- lib/dl/core/version.rb
+- sig/dl/core.rbs
+homepage: https://github.com/xoengineering/dl-core
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/xoengineering/dl-core
+  source_code_uri: https://github.com/xoengineering/dl-core
+  bug_tracker_uri: https://github.com/xoengineering/dl-core/issues
+  changelog_uri: https://github.com/xoengineering/dl-core/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 4.0.7
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.21
+specification_version: 4
+summary: Shared foundation for the <site>-dl offline-archive gems (arxiv-dl, jstor-dl,
+  and more).
+test_files: []