gem 0.0.1.alpha

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Samuel Cochran
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,75 @@
+# Gem
+
+Just enough not-Rubygems to index a collection of gems for download... and maybe more.
+
+There are some severe caveats. Some are by design, others will be addressed later.
+
+Created by [Samuel Cochran](http://sj26.com) for [Railscamp X](http://railscamps.com).
+
+## Thoughts
+
+Rubygems is a bit... horrible. It was a good fit for the problem it solved for a long time, but we've outgrown it. Here are a list of my thoughts on what's wrong and how we could fix it. Many of these ideas are or will permeate this `gem` gem as a (mostly backwards-compatible) drop-in replacement for rubygems.
+
+Local index should be more granular and allow stream processing. Same with specs lists. Marshalling and constantly re-downloading a whole index, or even just a list of specs, is ludicrous at current sizes.
+
+Break index up into more files for efficient lookup?
+
+Learn from PIP, have quick lookups for common cases.
+
+JSON is a better choice than YAML—it's just plain faster and becoming ubiquitous. Built-in (usually reasonably fast) support in 1.9, including streaming, which is cross-platform friendly (much moreso than YAML).
+
+Auto-negotiated transport compression should counteract any size difference and work for partial updates.
+
+Compression is purely a transport/storage concern.
+
+Index journal for efficient partial updates via HTTP.
+
+Use plain old HTTP (but the full functionality of 1.1) and static files to allow serving at edges by cloudfront, easy proxying/caching, etc.
+
+Same directory structure and index format locally and remotely, backwards compatible:
+
+Given the following definitions:
+
+ * `<basename>`: `<name>-<version>[-<platform>]`
+   * `<name>`: gem name, i.e. "rails"
+   * `<version>`: gem version, i.e. "3.2.2"
+   * `<platform>`: gem platform, omittted if platform is "ruby", the default
+
+Proposed directory structure:
+
+ * `cache/` -> `gems/` — a symlink for old gemball location, backwards compat only.
+ * `index/` — index of gemspecs.
+   * All gemspec indexes are stored as simple tuples, `[[<name>, <version>, <platform>[, <yanked?>]]+]`
+     * Journalled (append-only).
+     * Yanked gems add another entry with <yanked?> set to true.
+   * `specs.json[.gz]` — index of all (non-prerelease) gemspecs
+   * `latest_specs.json[.gz]` — index of latest (non-prerelease) gemspec
+   * `prerelease_specs.json[.gz] — index of all (non-prerelease) gemspecs ([[name, version, platform]*]), journalled.
+   * `<name>/` — gem name specific indexes
+     * `specs.json[.gz]` — index of gemspecs for a gem name (for complex requirement resolution)
+     * `latest_specs.json[.gz]` — latest gemspec for each platform for a gem name (`gem install <name>`)
+     * `prerelease_specs.json[.gz]` — latest prerelease gemspec for each platform for a gem name (`gem install —prerelease <name>`)
+     * eventually, we might need to replicate by version segment for targeted requirements if it would provide efficiency gains — gems with many semantic versions, etc:
+       `version-<version-prefix>/{specs,latest_specs,prerelease_specs}.json.gz`
+     * could also introduce something for platform, ala:
+       `version-<version-prefix>/]platform-<platform>/`
+ * `sources/` — cached source indexes
+   * `<source-sha>/` — SHA of source URL (i.e. `http://rubygems.org`)
+     * `index/` — a cache of the top-level index/ for a particular source
+ * `gems/` — installed gems, backwards compatible.
+   * `<basename>.gem` — the gem ball
+   * `<basename>/` — the unpacked gem tree
+ * `specifications/` — gem specifications of installed gems, backwards compatible.
+   * `<basename>.gemspec` — ruby format, without file/test lists
+
+Use net/http/persistent per-source for gem operations like rubygems-mirror, respecting HTTP content-type, transport-encoding (compression), range and freshness controls, and negating overhead of many small files.
+
+Multiplex mirroring and potentially other operations over a pool of threads like rubygems-mirror.
+
+Break old marshalling support into modules, only include them when backwards-compatible behaviour required (i.e. during upgrade, when compat is requested/configured).
+
+Add more checks and guards to make sure the index/gemspecs/gems can't get into an invalid state.
+
+## License
+
+MIT (see LICENSE). Some parts adapted from Rubygems, which is under the Ruby or MIT license.

data/bin/gem

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby -I ../lib
+
+require 'gem'
+
+raise ArgumentError, "Bad command" unless Gem.respond_to? ARGV.first
+
+Gem.public_send ARGV.shift, *ARGV

data/lib/gem.rb

@@ -0,0 +1,249 @@
+require 'fileutils'
+require 'net/http/persistent'
+require 'rbconfig'
+require 'time'
+require 'yaml'
+require 'zlib'
+
+module Gem
+  VERSION = '1.8.11'.freeze
+
+  # XXX: Find methods not implemented yet
+  def self.method_missing symbol, *args
+    # XXX: WHUT, why doesn't "[athing]"[1, -1] not give "athing"? Grr.
+    puts "TODO: #{name}.#{symbol}(#{args.inspect.tap { |s| s.slice!(0, 1); s.slice!(-1) } })"
+  end
+
+  def self.[] name, version=nil, platform=nil
+    name.gsub! /\.gem\Z/, ""
+    if version.nil?
+      versions = Dir[File.join(path, "cache", "#{name}-*.gem")].map do |filename|
+        File.basename(filename)
+      end.map do |basename|
+        basename.slice(name.length + 1, basename.length - name.length - 1 - 4)
+      end.map do |version|
+        Version.new version
+      end.reject(&:prerelease?).sort
+      version = versions.last.to_s unless versions.empty?
+    end
+
+    specification = Specification.new name, version, platform
+    filename = File.join path, "cache", "#{specification.basename}.gem"
+
+    Specification.from_gem filename
+  end
+
+  def self.all
+    # If we have an index, run the block if given
+    @index.gems.each(&proc) if block_given? and @index
+
+    # Otherwise build the index and run the block for each built spec
+    @index ||= SourceIndex.new.tap do |index|
+      progress = nil
+      Dir.foreach("gems").select do |path|
+        path =~ /\.gem\Z/
+      end.tap do |names|
+        progress = ProgressBar.new("Loading index", names.length)
+      end.each do |name|
+        begin
+          if specification = self[name].for_cache!
+            index.gems << specification
+            yield specification if block_given?
+          end
+        rescue StandardError
+          puts "Failed to load gem #{name.inspect}: #{$!}", $!.inspect, $!.backtrace
+        end
+        progress.inc
+      end
+      progress.finish
+      puts "#{index.gems.length} gems loaded into index"
+      index.gems.sort!
+    end
+  end
+
+  def self.quick_index specification
+    File.write("quick/#{specification.basename}.gemspec.rz", Zlib.deflate(YAML.dump(specification)))
+    File.write("quick/Marshal.#{marshal_version}/#{specification.basename}.gemspec.rz", Zlib.deflate(Marshal.dump(specification)))
+  end
+
+  def self.index
+    FileUtils.mkdir_p "quick/Marshal.#{marshal_version}"
+
+    all(&method(:quick_index))
+
+    print "Marshal index... "
+    File.write("Marshal.#{marshal_version}.Z", Zlib.deflate(Marshal.dump(all.gems.map { |spec| [spec.basename, spec] })))
+    puts "done."
+
+    # deprecated: Marshal.dump(all, File.open("Marshal.#{marshal_version}", "w"))
+
+    # deprecated:
+    #puts "Quick index"
+    #File.open('quick/index', 'w') do |quick_index|
+    #  all.gems.each do |specification|
+    #    quick_index.write("#{specification.name.to_s}-#{specification.version.version}\n")
+    #  end
+    #end
+
+    # deprecated:
+    #puts "Master index"
+    #YAML.dump(all, File.open("yaml", "w"))
+    #File.write("yaml.Z", Zlib.deflate(File.read("yaml")))
+
+    # un-gzipped indexes are deprecated, so generate gzipped directly:
+
+    print "Writing specs... "
+    Marshal.dump(all.gems.reject(&:prerelease?).map do |specification|
+      platform = specification.platform
+      platform = "ruby" if platform.nil? or platform.empty?
+      [specification.name, specification.version, platform]
+    end, Zlib::GzipWriter.new(File.open("specs.#{marshal_version}.gz", "w")))
+    puts "done."
+
+    print "Writing lastest_specs... "
+    Marshal.dump(all.gems.group_by(&:name).map do |name, specifications|
+      specification = specifications.reject(&:prerelease?).last
+      platform = specification.platform
+      platform = "ruby" if platform.nil? or platform.empty?
+      [specification.name, specification.version, platform]
+    end, Zlib::GzipWriter.new(File.open("latest_specs.#{marshal_version}.gz", "w")))
+    puts "done."
+
+    print "Writing prerelease_specs... "
+    Marshal.dump(all.gems.select(&:prerelease?).map do |specification|
+      platform = specification.platform
+      platform = "ruby" if platform.nil? or platform.empty?
+      [specification.name, specification.version, platform]
+    end, Zlib::GzipWriter.new(File.open("prerelease_specs.#{marshal_version}.gz", "w")))
+    puts "done."
+
+    # TODO: index.rss
+  end
+
+  def self.mirror source=source
+    http = Net::HTTP::Persistent.new "gem-mirror"
+    print "#{File.exist? "specs.#{marshal_version}.gz" and "Updating" or "Fetching"} specifications... "
+    ["specs", "latest_specs", "prerelease_specs"].each.in_threads do |specs_name|
+      path = "#{specs_name}.#{marshal_version}.gz"
+      uri = URI "#{source}/#{specs_name}.#{marshal_version}.gz"
+      headers = {}
+      headers['If-Modified-Since'] = File.mtime(path).rfc2822 if File.exist? path
+      catch :done do
+        loop do
+          request = Net::HTTP::Get.new uri.path, headers
+          http.request(uri, request) do |response|
+            puts response.inspect
+            if response.code == "304"
+              # Nothing to do, we already have latest version
+              throw :done
+            elsif response.code[0] == "3" and response["Location"]
+              # Redirect
+              url = URI.join uri.to_s, response["Location"]
+            elsif response.code == "206"
+              File.open(path, 'a') do |file|
+                response.read_body do |chunk|
+                  file.write chunk
+                end
+              end
+            elsif response.code == "200"
+              File.open(path, 'w') do |file|
+                response.read_body do |chunk|
+                  file.write chunk
+                end
+              end
+              last_modified = Time.parse response['Last-Modified']
+              File.utime last_modified, last_modified, path
+              throw :done
+            else
+              raise StandardError, "Unknown response: #{response.inspect}"
+            end
+          end
+        end
+      end
+    end
+    puts "done."
+    FileUtils.mkdir_p "gems"
+    progress = nil
+    ["latest_specs", "specs", "prerelease_specs"].each do |specs_name|
+      Marshal.load(IO.popen("gunzip -c #{specs_name}.#{marshal_version}.gz", "r", err: nil)).tap do |tuples|
+        progress = ProgressBar.new("Mirroring #{specs_name.gsub('_', ' ')}", tuples.length)
+      end.each.in_thread_pool(of: 8) do |tuple|
+        name, version, platform = tuple
+        begin
+          specification = Specification.new name: name, version: version, platform: platform
+          path = "gems/#{specification.basename}.gem"
+          unless File.exist? path and Specification.try_from_gem(path)
+            uri = URI "#{source}/#{path}"
+            headers = {}
+            headers["Range"] = "bytes=#{File.size(path)}-" if File.exist? path
+            catch :done do
+              loop do
+                request = Net::HTTP::Get.new uri.path, headers
+                http.request uri, request do |response|
+                  puts response.inspect
+                  if response.code == "304"
+                    # Nothing to do, we already have latest version
+                    throw :done
+                  elsif response.code[0] == "3" and response["Location"]
+                    # Redirect
+                    url = URI.join uri.to_s, response["Location"]
+                  elsif response.code == "200" or response.code == "206"
+                    # TODO: Check range properly
+                    File.open(path, response.code == '206' ? 'a' : 'w') do |file|
+                      response.read_body do |chunk|
+                        file.write chunk
+                      end
+                    end
+                    last_modified = Time.parse response['Last-Modified']
+                    File.utime last_modified, last_modified, path
+                    throw :done
+                  else
+                    raise StandardError, "Unknown response: #{response.inspect}"
+                  end
+                end
+              end
+            end
+            progress.puts specification.basename
+          end
+        rescue StandardError
+          progress.puts "Failed to mirror gem #{name.inspect}: #{$!}", $!.inspect, $!.backtrace
+        end
+        progress.inc
+      end
+      progress.finish
+    end
+    puts "#{`/bin/ls -1f | wc -l`.to_i - 2} gems mirrored."
+    index
+  end
+
+protected
+
+  def self.shellescape arg
+    if not arg.is_a? String
+      arg.to_s
+    else
+      arg.dup
+    end.tap do |arg|
+      # Process as a single byte sequence because not all shell
+      # implementations are multibyte aware.
+      arg.gsub!(/([^A-Za-z0-9_\-.,:\/@\n])/n, "\\\\\\1")
+
+      # A LF cannot be escaped with a backslash because a backslash + LF
+      # combo is regarded as line continuation and simply ignored.
+      arg.gsub!(/\n/, "'\n'")
+    end
+  end
+end
+
+require 'gem/tar'
+require 'gem/thread_poolable'
+
+require 'gem/configuration'
+require 'gem/version'
+require 'gem/requirement'
+require 'gem/dependency'
+require 'gem/platform'
+require 'gem/specification'
+require 'gem/progressbar'
+
+require 'gem/require'

data/lib/gem/configuration.rb

@@ -0,0 +1,43 @@
+module Gem::Configuration
+  def sources
+    @sources ||= %w(http://rubygems.org)
+  end
+
+  def sources= value
+    @sources = value
+  end
+
+  def source
+    sources.first
+  end
+
+  def source= value
+    self.sources = [value]
+  end
+
+  def ruby_engine
+    if defined? RUBY_ENGINE then
+      RUBY_ENGINE
+    else
+      'ruby'
+    end
+  end
+
+  def path
+    @path ||= begin
+      File.join *if defined? RUBY_FRAMEWORK_VERSION
+        [File.dirname(RbConfig::CONFIG["sitedir"]), 'Gems', RbConfig::CONFIG["ruby_version"]]
+      elsif RbConfig::CONFIG["rubylibprefix"] then
+        [RbConfig::CONFIG["rubylibprefix"], 'gems', RbConfig::CONFIG["ruby_version"]]
+      else
+        [RbConfig::CONFIG["libdir"], ruby_engine, 'gems', RbConfig::CONFIG["ruby_version"]]
+      end
+    end
+  end
+
+  def marshal_version
+    "#{Marshal::MAJOR_VERSION}.#{Marshal::MINOR_VERSION}"
+  end
+end
+
+Gem.extend Gem::Configuration

data/lib/gem/dependency.rb

@@ -0,0 +1,2 @@
+class Gem::Dependency
+end

data/lib/gem/platform.rb

@@ -0,0 +1,76 @@
+class Gem::Platform
+  RUBY = 'ruby'
+
+  attr_accessor :cpu, :os, :version
+
+  def initialize arch
+    case arch
+    when Array then
+      @cpu, @os, @version = arch
+    when String then
+      arch = arch.split '-'
+
+      if arch.length > 2 and arch.last !~ /\d/ then # reassemble x86-linux-gnu
+        extra = arch.pop
+        arch.last << "-#{extra}"
+      end
+
+      cpu = arch.shift
+
+      @cpu = case cpu
+         when /i\d86/ then 'x86'
+         else cpu
+       end
+
+      if arch.length == 2 and arch.last =~ /^\d+(\.\d+)?$/ then # for command-line
+        @os, @version = arch
+        return
+      end
+
+      os, = arch
+      @cpu, os = nil, cpu if os.nil? # legacy jruby
+
+      @os, @version = case os
+        when /aix(\d+)/ then             ['aix',       $1 ]
+        when /cygwin/ then               ['cygwin',    nil]
+        when /darwin(\d+)?/ then         ['darwin',    $1 ]
+        when /freebsd(\d+)/ then         ['freebsd',   $1 ]
+        when /hpux(\d+)/ then            ['hpux',      $1 ]
+        when /^java$/, /^jruby$/ then    ['java',      nil]
+        when /^java([\d.]*)/ then        ['java',      $1 ]
+        when /^dotnet$/ then             ['dotnet',    nil]
+        when /^dotnet([\d.]*)/ then      ['dotnet',    $1 ]
+        when /linux/ then                ['linux',     $1 ]
+        when /mingw32/ then              ['mingw32',   nil]
+        when /(mswin\d+)(\_(\d+))?/ then
+          os, version = $1, $3
+          @cpu = 'x86' if @cpu.nil? and os =~ /32$/
+          [os, version]
+        when /netbsdelf/ then            ['netbsdelf', nil]
+        when /openbsd(\d+\.\d+)/ then    ['openbsd',   $1 ]
+        when /solaris(\d+\.\d+)/ then    ['solaris',   $1 ]
+        # test
+        when /^(\w+_platform)(\d+)/ then [$1,          $2 ]
+        else                             ['unknown',   nil]
+      end
+    when Gem::Platform then
+      @cpu = arch.cpu
+      @os = arch.os
+      @version = arch.version
+    else
+      raise ArgumentError, "invalid argument #{arch.inspect}"
+    end
+  end
+
+  def to_a
+    [@cpu, @os, @version]
+  end
+
+  def to_s
+    to_a.compact.join '-'
+  end
+
+  def empty?
+    to_s.empty?
+  end
+end