gem_mirror 0.1.0.pre

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9dc1e0fb7c8d5bee7140b693cb630b95b69f5e9ffb23fe31478759a311748c11
+  data.tar.gz: 909b09e6a4dab1184d83dde3004376491c5066ef54d9c8f55b036a90a7148c4c
+SHA512:
+  metadata.gz: 359b87727794e927bb4c9332dbae112bfe01e7bd578a99f574d4a0ad58d7a5749897d8975c7f8fff0637b8084806acd3390266191b0392d95c589bb12c4486a2
+  data.tar.gz: 376d68180e4fa30da16ae889952c43254be87e7d34ddefca526cad22085c6bdd5d72c8490072f528c23c5a85d8ec439068a0d5a97ea6d43f4ed2e19ad625916b

data/CHANGELOG.md

@@ -0,0 +1,7 @@
+## 0.1.0.pre
+
+### Changes
+
+  - Refactoring and bringing code up to date with modern Ruby
+  - Addressed over 250 rubocop recommendations
+  - Dropped legacy gem index option

data/LICENSE.txt

@@ -0,0 +1,19 @@
+Copyright (c) 2013, Yorick Peterse
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,85 @@
+# Gem Mirror
+
+gem_mirror is a Ruby application that makes it easy to create your own RubyGems
+mirror without having to stitch together ugly Bash scripts or deal with more
+complex tools such as [Geminabox][geminabox]. Unlike tools such as Geminabox
+and others gem_mirror does mirroring only, it has no authentication and you
+can't upload gems to it. I created this project to improve my knowledge of Ruby
+and Ruby infrastructure.
+
+It was forked from the now archived github project which has also moved
+to gitlab [gem-mirror].
+
+## Differences with Geminabox
+
+Geminabox is basically a tiny DIY RubyGems that focuses more on uploading your
+own gems (e.g. company specific ones) instead of mirroring an external source.
+gem_mirror however purely focuses on generating a mirror that can be used by
+the RubyGems CLI utility.
+
+## Differences with gem-server
+
+RubyGems comes with the command `gem-server` which can be used to serve the
+Gems you have installed on that machine and user. Similar to Geminabox it does
+not provide the means to actually mirror an external source. It's also Rack
+based and uses WEBRick, which isn't exactly the best server around.
+
+## How do I use it?
+
+The process of setting up a mirror is fairly easy and can be done in a matter
+of minutes depending on the number of gems you are mirroring and your internet
+speed.
+
+The first step is to set up a new, empty mirror. This is done by running the
+`gem_mirror init` command similar to how you initialize a new Git repository:
+
+    $ gem_mirror init /srv/http/mirror.com/
+
+Once created, you should edit the main configuration file named `config.rb`.
+This configuration file specifies what sources to mirror (you can mirror
+multiple ones!), which gems, etc.
+
+Once configured you can pull all the gems by running the following command:
+
+    $ gem_mirror update
+
+If your configuration file is not located in the current directory you can
+specify the path to it using the `-c` or `--config` option. The process of
+downloading all the Gems takes a while and can use quite a bit of memory so
+make sure you have plenty of time and RAM available.
+
+Once all the Gems have been downloaded you'll need to generate an index of all
+the installed files. This can be done as following:
+
+    $ gem_mirror index
+
+Last but not least you may want to generate a list of SHA512 checksums. Although
+RubyGems doesn't use these they might come in handy when verifying Gems (e.g.
+when RubyGems gets hacked again). This can be done using the following command:
+
+    $ gem_mirror checksum
+
+If you want to do all this automatically (which is recommended) you can use a
+simple Cronjob like the following one:
+
+    @hourly cd /srv/http/mirror.com && gem_mirror update && gem_mirror index
+
+## Requirements
+
+* Ruby 2.5.0 or newer
+* Enough space to store Gems
+
+## Installation
+
+Assuming RubyGems isn't down you can install the Gem as following:
+
+    $ gem install gem_mirror
+
+## License
+
+All source code in this repository is licensed under the MIT license unless
+specified otherwise. A copy of this license can be found in the file "LICENSE"
+in the root directory of this repository.
+
+[geminabox]: https://github.com/geminabox/geminabox
+[gem-mirror]: https://gitlab.com/yorickpeterse/gem-mirror

data/exe/gem_mirror

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+require File.expand_path("../../lib/gem_mirror", __FILE__)
+
+GemMirror::CLI.options.parse

data/gem_mirror.gemspec

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require_relative "lib/gem_mirror/version"
+
+Gem::Specification.new do |s|
+  s.name                  = "gem_mirror"
+  s.version               = GemMirror::VERSION
+  s.platform              = Gem::Platform::RUBY
+  s.required_ruby_version = Gem::Requirement.new(">= 2.5.0")
+  s.authors               = ["Yorick Peterse", "Patrick Callahan"]
+  s.description           = <<~DESCRIPTION
+    gem-mirror is a tool for creating and managing a private
+    mirror of RubyGems. It is also suitable for hosting private
+    gems.
+  DESCRIPTION
+
+  s.email                 = ["pmc@patrickcallahan.com"]
+  s.files                 = Dir["{assets,config,lib,template}/**/*"] + ["gem_mirror.gemspec"]
+  s.files                += %w[CHANGELOG.md LICENSE.txt README.md]
+  s.require_paths         = ["lib"]
+  s.bindir                = "exe"
+  s.executables           = %w[gem_mirror]
+  s.extra_rdoc_files      = %w[CHANGELOG.md LICENSE.txt README.md]
+  s.homepage              = "https://github.com/dirtyharrycallahan/gem_mirror"
+  s.licenses              = ["MIT"]
+  s.summary               = %q{A tool for creating and managing a private rubygems mirror.}
+
+  s.rdoc_options += [
+    "--title", "gem_mirror - A Tool for Managing a Private Gem Repository",
+    "--main", "README.md",
+    "--line-numbers",
+    "--inline-source",
+    "--quiet"
+  ]
+  s.metadata["allowed_push_host"] = "https://rubygems.org"
+  s.metadata["changelog_uri"]     = "https://github.com/dirtyharrycallahan/gem_mirror/blob/master/CHANGELOG.md"
+  s.metadata["homepage_uri"]      = s.homepage
+  s.metadata["source_code_uri"]   = s.homepage
+  s.metadata["bug_tracker_uri"]   = "https://github.com/dirtyharrycallahan/gem_mirror/issues"
+
+  s.add_runtime_dependency("confstruct", "~> 1.0", "< 2")
+  s.add_runtime_dependency("httpclient", "~> 2.8", "< 3")
+  s.add_runtime_dependency("slop",       "= 3.6")
+
+  s.add_development_dependency("bundler",             ">= 1.17.3", "< 3.0")
+  s.add_development_dependency("rake",                ">= 13.0.0", "< 14")
+  s.add_development_dependency("rake-manifest",       "~> 0.2.0")
+  s.add_development_dependency("redcarpet",           "~> 3.5.0")
+  s.add_development_dependency("rspec",               "~> 3.2")
+  s.add_development_dependency("rubocop",             "= 1.16.0")
+  s.add_development_dependency("rubocop-performance", "= 1.11.3")
+  s.add_development_dependency("rubocop-rake",        "= 0.5.1")
+  s.add_development_dependency("rubocop-rspec",       "= 2.3.0")
+  s.add_development_dependency("yard",                "~> 0.9.0")
+end

data/lib/gem_mirror.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "rubygems"
+require "rubygems/user_interaction"
+require "rubygems/indexer"
+require "fileutils"
+require "digest/sha2"
+require "confstruct"
+require "slop"
+require "zlib"
+require "httpclient"
+require "logger"
+require "stringio"
+
+$LOAD_PATH.unshift(File.expand_path(__dir__)) unless $LOAD_PATH.include?(File.expand_path(__dir__))
+
+require_relative "gem_mirror/version"
+require_relative "gem_mirror/configuration"
+require_relative "gem_mirror/gem"
+require_relative "gem_mirror/source"
+require_relative "gem_mirror/mirror_directory"
+require_relative "gem_mirror/mirror_file"
+require_relative "gem_mirror/versions_file"
+require_relative "gem_mirror/versions_fetcher"
+require_relative "gem_mirror/gems_fetcher"
+
+require_relative "gem_mirror/cli"
+require_relative "gem_mirror/cli/checksum"
+require_relative "gem_mirror/cli/index"
+require_relative "gem_mirror/cli/init"
+require_relative "gem_mirror/cli/update"

data/lib/gem_mirror/cli.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+# Namespace for GemMirror
+module GemMirror
+  # Namespace for classes and modules that handle the command line interface
+  module CLI
+    ##
+    # Hash containing the default Slop options.
+    #
+    # @return [Hash]
+    #
+    SLOP_OPTIONS = {
+      strict: true,
+      banner: "Usage: gem_mirror [COMMAND] [OPTIONS]"
+    }.freeze
+
+    ##
+    # @return [Slop]
+    #
+    def self.options
+      @options ||= default_options
+    end
+
+    ##
+    # Loads the specified configuration file or displays an error if it doesn't
+    # exist.
+    #
+    # @param [String] config_file
+    # @return [GemMirror::Configuration]
+    #
+    def self.load_configuration(config_file)
+      config_file ||= Configuration.default_configuration_file
+      config_file   = File.expand_path(config_file, Dir.pwd)
+
+      abort "The configuration file #{config_file} does not exist" unless File.file?(config_file)
+
+      require(config_file)
+    end
+
+    ##
+    # @return [Slop]
+    #
+    def self.default_options
+      Slop.new(SLOP_OPTIONS.dup) do
+        separator "\nOptions:\n"
+
+        on :h, :help, "Shows this help message" do
+          puts self
+          exit
+        end
+
+        on :v, :version, "Shows the current version" do
+          puts CLI.version_information
+        end
+      end
+    end
+
+    ##
+    # Returns a String containing some platform/version related information.
+    #
+    # @return [String]
+    #
+    def self.version_information
+      "gem_mirror v#{GemMirror::VERSION} on #{RUBY_DESCRIPTION}"
+    end
+  end
+end

data/lib/gem_mirror/cli/checksum.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+GemMirror::CLI.options.command "checksum" do
+  banner      "Usage: gem_mirror checksum [OPTIONS]"
+  description "Generates SHA512 checksums of all gems"
+  separator   "\nOptions:\n"
+
+  on :h, :help, "Shows this help message" do
+    puts self
+    exit
+  end
+
+  on :c=, :config=, "Path to the configuration file"
+
+  run do |opts, _args|
+    GemMirror::CLI.load_configuration(opts[:c])
+
+    config = GemMirror.configuration
+
+    unless File.directory?(config.checksums)
+      config.logger("The directory #{config.checksums} does not exist")
+      abort
+    end
+
+    unless File.directory?(config.destination)
+      config.logger("The directory #{config.destination} does not exist")
+      abort
+    end
+
+    Dir[File.join(config.gems_directory, "*.gem")].each do |gem|
+      basename = File.basename(gem)
+      name     = "#{basename}.sha512"
+
+      config.logger.info("Creating checksum for #{basename}")
+
+      hash   = Digest::SHA512.hexdigest(File.read(gem))
+      handle = File.open(File.join(config.checksums, name), "w")
+
+      handle.write(hash)
+      handle.close
+    end
+  end
+end

data/lib/gem_mirror/cli/index.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+GemMirror::CLI.options.command "index" do
+  banner      "Usage: gem_mirror index [OPTIONS]"
+  description "Indexes a list of Gems"
+  separator   "\nOptions:\n"
+
+  on :h, :help, "Shows this help message" do
+    puts self
+    exit
+  end
+
+  on :c=, :config=, "Path to the configuration file"
+
+  run do |opts, _args|
+    GemMirror::CLI.load_configuration(opts[:c])
+
+    config = GemMirror.configuration
+
+    unless File.directory?(config.destination)
+      config.logger.error("The directory #{config.destination} does not exist")
+      abort
+    end
+
+    indexer = Gem::Indexer.new(config.destination, build_legacy: opts[:l])
+
+    config.logger.info("Generating Indexes")
+
+    indexer.generate_index
+  end
+end

data/lib/gem_mirror/cli/init.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+GemMirror::CLI.options.command "init" do
+  banner      "Usage: gem_mirror init [DIRECTORY] [OPTIONS]"
+  description "Sets up a new mirror"
+  separator   "\nOptions:\n"
+
+  on :h, :help, "Shows this help message" do
+    puts self
+    exit
+  end
+
+  run do |_opts, args|
+    directory = File.expand_path(args[0] || Dir.pwd)
+    template  = GemMirror::Configuration.template_directory
+
+    Dir.mkdir(directory) unless File.directory?(directory)
+
+    FileUtils.cp_r(File.join(template, "."), directory)
+
+    puts "Initialized empty mirror in #{directory}"
+  end
+end

data/lib/gem_mirror/cli/update.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+GemMirror::CLI.options.command "update" do
+  banner      "Usage: gem_mirror update [OPTIONS]"
+  description "Updates the list of Gems"
+  separator   "\nOptions:\n"
+
+  on :h, :help, "Shows this help message" do
+    puts self
+    exit
+  end
+
+  on :c=, :config=, "Path to the configuration file"
+
+  run do |opts, _args|
+    GemMirror::CLI.load_configuration(opts[:c])
+
+    GemMirror.configuration.sources.each do |source|
+      versions = GemMirror::VersionsFetcher.new(source).fetch
+      gems     = GemMirror::GemsFetcher.new(source, versions)
+
+      gems.fetch
+    end
+  end
+end

data/lib/gem_mirror/configuration.rb

@@ -0,0 +1,140 @@
+# frozen_string_literal: true
+
+# Namespace for GemMirror
+module GemMirror
+  ##
+  # @return [GemMirror::Configuration]
+  #
+  def self.configuration
+    @configuration ||= Configuration.new
+  end
+
+  ##
+  # Configuration class used for storing data about a mirror such as the
+  # destination directory, sources, ignored Gems, etc.
+  #
+  class Configuration < Confstruct::Configuration
+    ##
+    # @return [Logger]
+    #
+    def logger
+      @logger = Logger.new($stdout,
+                           progname: "gem_mirror",
+                           formatter: proc do |severity, datetime, progname, msg|
+                             d_format = datetime.strftime("%Y%m%dT%H%M%S%z")
+                             "[#{d_format}] severity=#{severity} prog=#{progname} pid=#{Process.pid} message=#{msg}\n"
+                           end)
+    end
+
+    ##
+    # @return [String]
+    #
+    def self.template_directory
+      File.expand_path("../../template", __dir__)
+    end
+
+    ##
+    # @return [String]
+    #
+    def self.default_configuration_file
+      File.expand_path("config.rb", Dir.pwd)
+    end
+
+    ##
+    # Returns the name of the directory that contains the quick
+    # specification files.
+    #
+    # @return [String]
+    #
+    def self.marshal_identifier
+      "Marshal.#{marshal_version}"
+    end
+
+    ##
+    # Returns the name of the file that contains an index of all the versions.
+    #
+    # @return [String]
+    #
+    def self.versions_file
+      "specs.#{marshal_version}.gz"
+    end
+
+    ##
+    # Returns a String containing the Marshal version.
+    #
+    # @return [String]
+    #
+    def self.marshal_version
+      "#{Marshal::MAJOR_VERSION}.#{Marshal::MINOR_VERSION}"
+    end
+
+    ##
+    # @return [GemMirror::MirrorDirectory]
+    #
+    def mirror_directory
+      @mirror_directory ||= MirrorDirectory.new(gems_directory)
+    end
+
+    ##
+    # @return [String]
+    #
+    def gems_directory
+      File.join(destination, "gems")
+    end
+
+    ##
+    # Returns a Hash containing various Gems to ignore and their versions.
+    #
+    # @return [Hash]
+    #
+    def ignored_gems
+      @ignored_gems ||= Hash.new { |hash, key| hash[key] = [] }
+    end
+
+    ##
+    # Adds a Gem to the list of Gems to ignore.
+    #
+    # @param [String] name
+    # @param [String] version
+    #
+    def ignore_gem(name, version)
+      ignored_gems[name] ||= []
+      ignored_gems[name] << version
+    end
+
+    ##
+    # Checks if a Gem should be ignored.
+    #
+    # @param [String] name
+    # @param [String] version
+    # @return [TrueClass|FalseClass]
+    #
+    def ignore_gem?(name, version)
+      ignored_gems[name].include?(version)
+    end
+
+    ##
+    # Returns a list of sources to mirror.
+    #
+    # @return [Array]
+    #
+    def sources
+      @sources ||= []
+    end
+
+    ##
+    # Adds a new source to mirror.
+    #
+    # @param [String] name
+    # @param [String] url
+    # @param [Proc] block
+    # @yieldparam [GemMirror::Source] source
+    #
+    def source(name, url, &block)
+      source = Source.new(name, url)
+      source.instance_eval(&block)
+
+      sources << source
+    end
+  end
+end