pkgr-deb-s3 0.6.0

data/README.md

@@ -0,0 +1,112 @@
+# deb-s3
+
+`deb-s3` is a simple utility to make creating and managing APT repositories on
+S3.
+
+Most existing existing guides on using S3 to host an APT repository have you
+using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
+the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
+sync the files to S3.
+
+The annoying thing about this process is it requires you to maintain a local
+copy of the file tree for regenerating and syncing the next time. Personally,
+my process is to use one-off virtual machines with
+[Vagrant](http://vagrantup.com), script out the build process, and then would
+prefer to just upload the final `.deb` from my Mac.
+
+With `deb-s3`, there is no need for this. `deb-s3` features:
+
+* Downloads the existing package manifest and parses it.
+* Updates it with the new package, replacing the existing entry if already
+  there or adding a new one if not.
+* Uploads the package itself, the Packages manifest, and the Packages.gz
+  manifest.
+* Updates the Release file with the new hashes and file sizes.
+
+## Getting Started
+
+You can simply install it from rubygems:
+
+```console
+$ gem install deb-s3
+```
+
+Or to run the code directly, just check out the repo and run Bundler to ensure
+all dependencies are installed:
+
+```console
+$ git clone https://github.com/krobertson/deb-s3.git
+$ cd deb-s3
+$ bundle install
+```
+
+Now to upload a package, simply use:
+
+```console
+$ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
+>> Examining package file my-deb-package-1.0.0_amd64.deb
+>> Retrieving existing package manifest
+>> Uploading package and new manifests to S3
+   -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
+   -- Transferring dists/stable/main/binary-amd64/Packages
+   -- Transferring dists/stable/main/binary-amd64/Packages.gz
+   -- Transferring dists/stable/Release
+>> Update complete.
+```
+
+```
+Usage:
+  deb-s3 upload FILES
+
+Options:
+  -a, [--arch=ARCH]                     # The architecture of the package in the APT repository.
+      [--sign=SIGN]                     # Sign the Release file. Use --sign with your key ID to use a specific key.
+  -p, [--preserve-versions]             # Whether to preserve other versions of a package in the repository when uploading one.
+  -b, [--bucket=BUCKET]                 # The name of the S3 bucket to upload to.
+  -c, [--codename=CODENAME]             # The codename of the APT repository.
+                                        # Default: stable
+  -m, [--component=COMPONENT]           # The component of the APT repository.
+                                        # Default: main
+      [--access-key-id=ACCESS_KEY]      # The access key for connecting to S3.
+      [--secret-access-key=SECRET_KEY]  # The secret key for connecting to S3.
+  -v, [--visibility=VISIBILITY]         # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                        # Default: public
+
+Uploads the given files to a S3 bucket as an APT repository.
+```
+
+You can also verify an existing APT repository on S3 using the `verify` command:
+
+```console
+deb-s3 verify -b my-bucket
+>> Retrieving existing manifests
+>> Checking for missing packages in: stable/main i386
+>> Checking for missing packages in: stable/main amd64
+>> Checking for missing packages in: stable/main all
+```
+
+```
+Usage:
+  deb-s3 verify
+
+Options:
+  -f, [--fix-manifests]                 # Whether to fix problems in manifests when verifying.
+      [--sign=SIGN]                     # Sign the Release file. Use --sign with your key ID to use a specific key.
+  -b, [--bucket=BUCKET]                 # The name of the S3 bucket to upload to.
+  -c, [--codename=CODENAME]             # The codename of the APT repository.
+                                        # Default: stable
+  -m, [--component=COMPONENT]           # The component of the APT repository.
+                                        # Default: main
+      [--access-key-id=ACCESS_KEY]      # The access key for connecting to S3.
+      [--secret-access-key=SECRET_KEY]  # The secret key for connecting to S3.
+  -v, [--visibility=VISIBILITY]         # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                        # Default: public
+
+Verifies that the files in the package manifests exist
+```
+
+## TODO
+
+This is still experimental.  These are several things to be done:
+
+* Don't re-upload a package if it already exists and has the same hashes.

data/bin/deb-s3

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+$:.unshift File.expand_path('../../lib', __FILE__)
+
+require 'rubygems'
+require 'deb/s3/cli'
+
+Deb::S3::CLI.start
+

data/lib/deb/s3.rb

@@ -0,0 +1,5 @@
+module Deb
+  module S3
+    VERSION = "0.6.0"
+  end
+end

data/lib/deb/s3/cli.rb

@@ -0,0 +1,234 @@
+require "aws"
+require "thor"
+
+# Hack: aws requires this!
+require "json"
+
+require "deb/s3"
+require "deb/s3/utils"
+require "deb/s3/manifest"
+require "deb/s3/package"
+require "deb/s3/release"
+
+class Deb::S3::CLI < Thor
+  class_option :bucket,
+  :type     => :string,
+  :aliases  => "-b",
+  :desc     => "The name of the S3 bucket to upload to."
+
+  class_option :prefix,
+  :type     => :string,
+  :desc     => "The path prefix to use when storing on S3."
+
+  class_option :codename,
+  :default  => "stable",
+  :type     => :string,
+  :aliases  => "-c",
+  :desc     => "The codename of the APT repository."
+
+  class_option :component,
+  :default  => "main",
+  :type     => :string,
+  :aliases  => "-m",
+  :desc     => "The component of the APT repository."
+
+  class_option :section,
+  :type     => :string,
+  :aliases  => "-s",
+  :hide     => true
+
+  class_option :access_key_id,
+  :type     => :string,
+  :desc     => "The access key for connecting to S3."
+
+  class_option :secret_access_key,
+  :type     => :string,
+  :desc     => "The secret key for connecting to S3."
+
+  class_option :visibility,
+  :default  => "public",
+  :type     => :string,
+  :aliases  => "-v",
+  :desc     => "The access policy for the uploaded files. " +
+    "Can be public, private, or authenticated."
+
+  class_option :sign,
+  :type     => :string,
+  :desc     => "Sign the Release file when uploading a package," +
+    "or when verifying it after removing a package." +
+    "Use --sign with your key ID to use a specific key."
+
+  class_option :gpg_options,
+  :default => "",
+  :type    => :string,
+  :desc    => "Additional command line options to pass to GPG when signing"
+
+  desc "upload FILES",
+  "Uploads the given files to a S3 bucket as an APT repository."
+
+  option :arch,
+  :type     => :string,
+  :aliases  => "-a",
+  :desc     => "The architecture of the package in the APT repository."
+
+  option :preserve_versions,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-p",
+  :desc     => "Whether to preserve other versions of a package " +
+    "in the repository when uploading one."
+
+  def upload(*files)
+    component = options[:component]
+    if options[:section]
+      component = options[:section]
+      warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
+    end
+
+    if files.nil? || files.empty?
+      error("You must specify at least one file to upload")
+    end
+
+    # make sure all the files exists
+    if missing_file = files.detect { |f| !File.exists?(f) }
+      error("File '#{missing_file}' doesn't exist")
+    end
+
+    # configure AWS::S3
+    configure_s3_client
+
+    # retrieve the existing manifests
+    log("Retrieving existing manifests")
+    release  = Deb::S3::Release.retrieve(options[:codename])
+    manifests = {}
+
+    # examine all the files
+    files.collect { |f| Dir.glob(f) }.flatten.each do |file|
+      log("Examining package file #{File.basename(file)}")
+      pkg = Deb::S3::Package.parse_file(file)
+
+      # copy over some options if they weren't given
+      arch = options[:arch] || pkg.architecture
+
+      # validate we have them
+      error("No architcture given and unable to determine one for #{file}. " +
+            "Please specify one with --arch [i386,amd64].") unless arch
+
+      # retrieve the manifest for the arch if we don't have it already
+      manifests[arch] ||= Deb::S3::Manifest.retrieve(options[:codename], component, arch)
+
+      # add in the package
+      manifests[arch].add(pkg, options[:preserve_versions])
+    end
+
+    # upload the manifest
+    log("Uploading packages and new manifests to S3")
+    manifests.each_value do |manifest|
+      manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
+      release.update_manifest(manifest)
+    end
+    release.write_to_s3 { |f| sublog("Transferring #{f}") }
+
+    log("Update complete.")
+  end
+
+  desc "verify", "Verifies that the files in the package manifests exist"
+
+  option :fix_manifests,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-f",
+  :desc     => "Whether to fix problems in manifests when verifying."
+
+  def verify
+    component = options[:component]
+    if options[:section]
+      component = options[:section]
+      warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
+    end
+
+    configure_s3_client
+
+    log("Retrieving existing manifests")
+    release = Deb::S3::Release.retrieve(options[:codename])
+
+    %w[amd64 armel i386 all].each do |arch|
+      log("Checking for missing packages in: #{options[:codename]}/#{options[:component]} #{arch}")
+      manifest = Deb::S3::Manifest.retrieve(options[:codename], component, arch)
+      missing_packages = []
+
+      manifest.packages.each do |p|
+        unless Deb::S3::Utils.s3_exists? p.url_filename_encoded
+          sublog("The following packages are missing:\n\n") if missing_packages.empty?
+          puts(p.generate)
+          puts("")
+
+          missing_packages << p
+        end
+      end
+
+      if options[:fix_manifests] && !missing_packages.empty?
+        log("Removing #{missing_packages.length} package(s) from the manifest...")
+        missing_packages.each { |p| manifest.packages.delete(p) }
+        manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
+        release.update_manifest(manifest)
+        release.write_to_s3 { |f| sublog("Transferring #{f}") }
+
+        log("Update complete.")
+      end
+    end
+  end
+
+  private
+
+  def log(message)
+    puts ">> #{message}"
+  end
+
+  def sublog(message)
+    puts "   -- #{message}"
+  end
+
+  def error(message)
+    puts "!! #{message}"
+    exit 1
+  end
+
+  def provider
+    access_key_id     = options[:access_key_id]
+    secret_access_key = options[:secret_access_key]
+
+    if access_key_id.nil? ^ secret_access_key.nil?
+      error("If you specify one of --access-key-id or --secret-access-key, you must specify the other.")
+    end
+
+    static_credentials = {}
+    static_credentials[:access_key_id]     = access_key_id     if access_key_id
+    static_credentials[:secret_access_key] = secret_access_key if secret_access_key
+
+    AWS::Core::CredentialProviders::DefaultProvider.new(static_credentials)
+  end
+
+  def configure_s3_client
+    error("No value provided for required options '--bucket'") unless options[:bucket]
+
+    Deb::S3::Utils.s3          = AWS::S3.new(provider.credentials)
+    Deb::S3::Utils.bucket      = options[:bucket]
+    Deb::S3::Utils.signing_key = options[:sign]
+    Deb::S3::Utils.gpg_options = options[:gpg_options]
+    Deb::S3::Utils.prefix      = options[:prefix]
+
+    # make sure we have a valid visibility setting
+    Deb::S3::Utils.access_policy =
+      case options[:visibility]
+      when "public"
+        :public_read
+      when "private"
+        :private
+      when "authenticated"
+        :authenticated_read
+      else
+        error("Invalid visibility setting given. Can be public, private, or authenticated.")
+      end
+  end
+end

data/lib/deb/s3/manifest.rb

@@ -0,0 +1,104 @@
+require "tempfile"
+require "zlib"
+
+class Deb::S3::Manifest
+  include Deb::S3::Utils
+
+  attr_accessor :codename
+  attr_accessor :component
+  attr_accessor :architecture
+
+  attr_accessor :files
+
+  def initialize
+    @packages = []
+    @component = nil
+    @architecture = nil
+    @files = {}
+  end
+
+  class << self
+    def retrieve(codename, component, architecture)
+      m = if s = Deb::S3::Utils.s3_read("dists/#{codename}/#{component}/binary-#{architecture}/Packages")
+        self.parse_packages(s)
+      else
+        self.new
+      end
+
+      m.codename = codename
+      m.component = component
+      m.architecture = architecture
+      m
+    end
+
+    def parse_packages(str)
+      m = self.new
+      str.split("\n\n").each do |s|
+        next if s.chomp.empty?
+        m.packages << Deb::S3::Package.parse_string(s)
+      end
+      m
+    end
+  end
+
+  def packages
+    @packages
+  end
+
+  def add(pkg, preserve_versions)
+    if preserve_versions
+      @packages.delete_if { |p| p.name == pkg.name && p.version == pkg.version }
+    else
+      @packages.delete_if { |p| p.name == pkg.name }
+    end
+    @packages << pkg
+    pkg
+  end
+
+  def generate
+    @packages.collect { |pkg| pkg.generate }.join("\n")
+  end
+
+  def write_to_s3
+    manifest = self.generate
+
+    # store any packages that need to be stored
+    @packages.each do |pkg|
+      if pkg.needs_uploading?
+        yield pkg.url_filename if block_given?
+        s3_store(pkg.filename, pkg.url_filename_encoded)
+      end
+    end
+
+    # generate the Packages file
+    pkgs_temp = Tempfile.new("Packages")
+    pkgs_temp.write manifest
+    pkgs_temp.close
+    f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages"
+    yield f if block_given?
+    s3_store(pkgs_temp.path, f)
+    @files["#{@component}/binary-#{@architecture}/Packages"] = hashfile(pkgs_temp.path)
+    pkgs_temp.unlink
+
+    # generate the Packages.gz file
+    gztemp = Tempfile.new("Packages.gz")
+    gztemp.close
+    Zlib::GzipWriter.open(gztemp.path) { |gz| gz.write manifest }
+    f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages.gz"
+    yield f if block_given?
+    s3_store(gztemp.path, f)
+    @files["#{@component}/binary-#{@architecture}/Packages.gz"] = hashfile(gztemp.path)
+    gztemp.unlink
+
+    nil
+  end
+
+  def hashfile(path)
+    {
+      :size   => File.size(path),
+      :sha1   => Digest::SHA1.file(path).hexdigest,
+      :sha256 => Digest::SHA2.file(path).hexdigest,
+      :md5    => Digest::MD5.file(path).hexdigest
+    }
+  end
+end

data/lib/deb/s3/package.rb

@@ -0,0 +1,273 @@
+require "digest/sha1"
+require "digest/sha2"
+require "digest/md5"
+require "socket"
+require "tmpdir"
+
+class Deb::S3::Package
+  include Deb::S3::Utils
+
+  attr_accessor :name
+  attr_accessor :version
+  attr_accessor :epoch
+  attr_accessor :iteration
+  attr_accessor :maintainer
+  attr_accessor :vendor
+  attr_accessor :url
+  attr_accessor :category
+  attr_accessor :license
+  attr_accessor :architecture
+  attr_accessor :description
+
+  attr_accessor :dependencies
+  attr_accessor :provides
+  attr_accessor :conflicts
+  attr_accessor :replaces
+  attr_accessor :excludes
+
+
+  # Any other attributes specific to this package.
+  # This is where you'd put rpm, deb, or other specific attributes.
+  attr_accessor :attributes
+
+  # hashes
+  attr_accessor :url_filename
+  attr_accessor :sha1
+  attr_accessor :sha256
+  attr_accessor :md5
+  attr_accessor :size
+
+  attr_accessor :filename
+
+  class << self
+    include Deb::S3::Utils
+
+    def parse_file(package)
+      p = self.new
+      p.extract_info(extract_control(package))
+      p.apply_file_info(package)
+      p.filename = package
+      p
+    end
+
+    def parse_string(s)
+      p = self.new
+      p.extract_info(s)
+      p
+    end
+
+    def extract_control(package)
+      if system("which dpkg 2>&1 >/dev/null")
+        `dpkg -f #{package}`
+      else
+        Dir.mktmpdir do |path|
+          safesystem("ar p #{package} control.tar.gz | tar -zxf - -C #{path}")
+          File.read(File.join(path, "control"))
+        end
+      end
+    end
+  end
+
+  def initialize
+    @attributes = {}
+
+    # Reference
+    # http://www.debian.org/doc/manuals/maint-guide/first.en.html
+    # http://wiki.debian.org/DeveloperConfiguration
+    # https://github.com/jordansissel/fpm/issues/37
+    if ENV.include?("DEBEMAIL") and ENV.include?("DEBFULLNAME")
+      # Use DEBEMAIL and DEBFULLNAME as the default maintainer if available.
+      @maintainer = "#{ENV["DEBFULLNAME"]} <#{ENV["DEBEMAIL"]}>"
+    else
+      # TODO(sissel): Maybe support using 'git config' for a default as well?
+      # git config --get user.name, etc can be useful.
+      #
+      # Otherwise default to user@currenthost
+      @maintainer = "<#{ENV["USER"]}@#{Socket.gethostname}>"
+    end
+
+    @name = nil
+    @architecture = "native"
+    @description = "no description given"
+    @version = nil
+    @epoch = nil
+    @iteration = nil
+    @url = nil
+    @category = "default"
+    @license = "unknown"
+    @vendor = "none"
+    @sha1 = nil
+    @sha256 = nil
+    @md5 = nil
+    @size = nil
+    @filename = nil
+    @url_filename = nil
+
+    @provides = []
+    @conflicts = []
+    @replaces = []
+    @dependencies = []
+
+    @needs_uploading = false
+  end
+
+  def filename=(f)
+    @filename = f
+    @needs_uploading = true
+    @filename
+  end
+
+  def url_filename
+    @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{File.basename(self.filename)}"
+  end
+
+  def url_filename_encoded
+    @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{s3_escape(File.basename(self.filename))}"
+  end
+
+  def needs_uploading?
+    @needs_uploading
+  end
+
+  def generate
+    template("package.erb").result(binding)
+  end
+
+  # from fpm
+  def parse_depends(data)
+    return [] if data.nil? or data.empty?
+    # parse dependencies. Debian dependencies come in one of two forms:
+    # * name
+    # * name (op version)
+    # They are all on one line, separated by ", "
+
+    dep_re = /^([^ ]+)(?: \(([>=<]+) ([^)]+)\))?$/
+    return data.split(/, */).collect do |dep|
+      m = dep_re.match(dep)
+      if m
+        name, op, version = m.captures
+        # this is the proper form of dependency
+        if op && version && op != "" && version != ""
+          "#{name} (#{op} #{version})".strip
+        else
+          name.strip
+        end
+      else
+        # Assume normal form dependency, "name op version".
+        dep
+      end
+    end
+  end # def parse_depends
+
+  # from fpm
+  def fix_dependency(dep)
+    # Deb dependencies are: NAME (OP VERSION), like "zsh (> 3.0)"
+    # Convert anything that looks like 'NAME OP VERSION' to this format.
+    if dep =~ /[\(,\|]/
+      # Don't "fix" ones that could appear well formed already.
+    else
+      # Convert ones that appear to be 'name op version'
+      name, op, version = dep.split(/ +/)
+      if !version.nil?
+        # Convert strings 'foo >= bar' to 'foo (>= bar)'
+        dep = "#{name} (#{debianize_op(op)} #{version})"
+      end
+    end
+
+    name_re = /^[^ \(]+/
+    name = dep[name_re]
+    if name =~ /[A-Z]/
+      dep = dep.gsub(name_re) { |n| n.downcase }
+    end
+
+    if dep.include?("_")
+      dep = dep.gsub("_", "-")
+    end
+
+    # Convert gem ~> X.Y.Z to '>= X.Y.Z' and << X.Y+1.0
+    if dep =~ /\(~>/
+      name, version = dep.gsub(/[()~>]/, "").split(/ +/)[0..1]
+      nextversion = version.split(".").collect { |v| v.to_i }
+      l = nextversion.length
+      nextversion[l-2] += 1
+      nextversion[l-1] = 0
+      nextversion = nextversion.join(".")
+      return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
+    elsif (m = dep.match(/(\S+)\s+\(!= (.+)\)/))
+      # Append this to conflicts
+      self.conflicts += [dep.gsub(/!=/,"=")]
+      return []
+    elsif (m = dep.match(/(\S+)\s+\(= (.+)\)/)) and
+        self.attributes[:deb_ignore_iteration_in_dependencies?]
+      # Convert 'foo (= x)' to 'foo (>= x)' and 'foo (<< x+1)'
+      # but only when flag --ignore-iteration-in-dependencies is passed.
+      name, version = m[1..2]
+      nextversion = version.split('.').collect { |v| v.to_i }
+      nextversion[-1] += 1
+      nextversion = nextversion.join(".")
+      return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
+    else
+      # otherwise the dep is probably fine
+      return dep.rstrip
+    end
+  end # def fix_dependency
+
+  # from fpm
+  def extract_info(control)
+    parse = lambda do |field| 
+      value = control[/^#{field}: .*/]
+      if value.nil?
+        return nil
+      else
+        return value.split(": ",2).last
+      end
+    end
+
+    # Parse 'epoch:version-iteration' in the version string
+    version_re = /^(?:([0-9]+):)?(.+?)(?:-(.*))?$/
+    m = version_re.match(parse.call("Version"))
+    if !m
+      raise "Unsupported version string '#{parse.call("Version")}'"
+    end
+    self.epoch, self.version, self.iteration = m.captures
+
+    self.architecture = parse.call("Architecture")
+    self.category = parse.call("Section")
+    self.license = parse.call("License") || self.license
+    self.maintainer = parse.call("Maintainer")
+    self.name = parse.call("Package")
+    self.url = parse.call("Homepage")
+    self.vendor = parse.call("Vendor") || self.vendor
+    self.attributes[:deb_priority] = parse.call("Priority")
+    self.attributes[:deb_installed_size] = parse.call("Installed-Size")
+
+    # Packages manifest fields
+    self.url_filename = parse.call("Filename")
+    self.sha1 = parse.call("SHA1")
+    self.sha256 = parse.call("SHA256")
+    self.md5 = parse.call("MD5sum")
+    self.size = parse.call("Size")
+
+    # The description field is a special flower, parse it that way.
+    # The description is the first line as a normal Description field, but also continues
+    # on future lines indented by one space, until the end of the file. Blank
+    # lines are marked as ' .'
+    description = control[/^Description: .*[^\Z]/m]
+    description = description.gsub(/^[^(Description|\s)].*$/, "").split(": ", 2).last
+    self.description = description.gsub(/^ /, "").gsub(/^\.$/, "")
+
+    #self.config_files = config_files
+
+    self.dependencies += Array(parse_depends(parse.call("Depends")))
+    self.conflicts += Array(parse_depends(parse.call("Conflicts")))
+    self.provides += Array(parse_depends(parse.call("Provides")))
+    self.replaces += Array(parse_depends(parse.call("Replaces")))
+  end # def extract_info
+
+  def apply_file_info(file)
+    self.size = File.size(file)
+    self.sha1 = Digest::SHA1.file(file).hexdigest
+    self.sha256 = Digest::SHA2.file(file).hexdigest
+    self.md5 = Digest::MD5.file(file).hexdigest
+  end
+end

data/lib/deb/s3/release.rb

@@ -0,0 +1,139 @@
+require "tempfile"
+
+class Deb::S3::Release
+  include Deb::S3::Utils
+
+  attr_accessor :codename
+  attr_accessor :architectures
+  attr_accessor :components
+
+  attr_accessor :files
+  attr_accessor :policy
+
+  def initialize
+    @codename = nil
+    @architectures = []
+    @components = []
+    @files = {}
+    @policy = :public_read
+  end
+
+  class << self
+    def retrieve(codename)
+      if s = Deb::S3::Utils.s3_read("dists/#{codename}/Release")
+        self.parse_release(s)
+      else
+        rel = self.new
+        rel.codename = codename
+        rel
+      end
+    end
+
+    def parse_release(str)
+      rel = self.new
+      rel.parse(str)
+      rel
+    end
+  end
+
+  def filename
+    "dists/#{@codename}/Release"
+  end
+
+  def parse(str)
+    parse = lambda do |field|
+      value = str[/^#{field}: .*/]
+      if value.nil?
+        return nil
+      else
+        return value.split(": ",2).last
+      end
+    end
+
+    # grab basic fields
+    self.codename = parse.call("Codename")
+    self.architectures = (parse.call("Architectures") || "").split(/\s+/)
+    self.components = (parse.call("Components") || "").split(/\s+/)
+
+    # find all the hashes
+    str.scan(/^\s+([^\s]+)\s+(\d+)\s+(.+)$/).each do |(hash,size,name)|
+      self.files[name] ||= { :size => size.to_i }
+      case hash.length
+      when 32
+        self.files[name][:md5] = hash
+      when 40
+        self.files[name][:sha1] = hash
+      when 64
+        self.files[name][:sha256] = hash
+      end
+    end
+  end
+
+  def generate
+    template("release.erb").result(binding)
+  end
+
+  def write_to_s3
+    # validate some other files are present
+    if block_given?
+      self.validate_others { |f| yield f }
+    else
+      self.validate_others
+    end
+
+    # generate the Release file
+    release_tmp = Tempfile.new("Release")
+    release_tmp.puts self.generate
+    release_tmp.close
+    yield self.filename if block_given?
+    s3_store(release_tmp.path, self.filename)
+
+    # sign the file, if necessary
+    if Deb::S3::Utils.signing_key
+      key_param = Deb::S3::Utils.signing_key != "" ? "--default-key=#{Deb::S3::Utils.signing_key}" : ""
+      if system("gpg -a #{key_param} #{Deb::S3::Utils.gpg_options} -b #{release_tmp.path}")
+        local_file = release_tmp.path+".asc"
+        remote_file = self.filename+".gpg"
+        yield remote_file if block_given?
+        raise "Unable to locate Release signature file" unless File.exists?(local_file)
+        s3_store(local_file, remote_file)
+        File.unlink(local_file)
+      else
+        raise "Signing the Release file failed."
+      end
+    else
+      # remove an existing Release.gpg, if it was there
+      s3_remove(self.filename+".gpg")
+    end
+
+    release_tmp.unlink
+  end
+
+  def update_manifest(manifest)
+    self.components << manifest.component unless self.components.include?(manifest.component)
+    self.architectures << manifest.architecture unless self.architectures.include?(manifest.architecture)
+    self.files.merge!(manifest.files)
+  end
+
+  def validate_others
+    to_apply = []
+    self.components.each do |comp|
+      %w(amd64 i386).each do |arch|
+        next if self.files.has_key?("#{comp}/binary-#{arch}/Packages")
+
+        m = Deb::S3::Manifest.new
+        m.codename = self.codename
+        m.component = comp
+        m.architecture = arch
+        if block_given?
+          m.write_to_s3 { |f| yield f }
+        else
+          m.write_to_s3
+        end
+        to_apply << m
+      end
+    end
+
+    to_apply.each { |m| self.update_manifest(m) }
+  end
+end

data/lib/deb/s3/templates/package.erb

@@ -0,0 +1,57 @@
+Package: <%= name %>
+Version: <%= "#{epoch}:" if epoch %><%= version %><%= "-" + iteration.to_s if iteration %>
+License: <%= license %>
+Vendor: <%= vendor %>
+Architecture: <%= architecture %>
+Maintainer: <%= maintainer  %>
+Installed-Size: <%= attributes[:deb_installed_size] %>
+<% if !dependencies.empty? and !attributes[:no_depends?] -%>
+Depends: <%= dependencies.collect { |d| fix_dependency(d) }.flatten.join(", ") %>
+<% end -%>
+<% if !conflicts.empty? -%>
+Conflicts: <%= conflicts.join(", ") %>
+<% end -%>
+<% if attributes[:deb_pre_depends_given?] -%>
+Pre-Depends: <%= attributes[:deb_pre_depends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
+<% end -%>
+<% if !provides.empty? -%>
+<%# Turn each provides from 'foo = 123' to simply 'foo' because Debian :\ -%>
+<%#  http://www.debian.org/doc/debian-policy/ch-relationships.html -%>
+Provides: <%= provides.map {|p| p.split(" ").first}.join ", " %>
+<% end -%>
+<% if !replaces.empty? -%>
+Replaces: <%= replaces.join(", ") %>
+<% end -%>
+<% if attributes[:deb_recommends_given?] -%>
+Recommends: <%= attributes[:deb_recommends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
+<% end -%>
+<% if attributes[:deb_suggests_given?] -%>
+Suggests: <%= attributes[:deb_suggests].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
+<% end -%>
+Section: <%= category %>
+Priority: <%= attributes[:deb_priority] %>
+Homepage: <%= url or "http://nourlgiven.example.com/" %>
+Filename: <%= url_filename_encoded %>
+<% if size -%>
+Size: <%= size %>
+<% end -%>
+<% if sha1 -%>
+SHA1: <%= sha1 %>
+<% end -%>
+<% if sha256 -%>
+SHA256: <%= sha256 %>
+<% end -%>
+<% if md5 -%>
+MD5sum: <%= md5 %>
+<% end -%>
+<% lines = (description or "no description given").split("\n") -%>
+<% firstline, *remainder = lines -%>
+Description: <%= firstline %>
+<% if remainder.any? -%>
+<%= remainder.collect { |l| l =~ /^ *$/ ? " ." : " #{l}" }.join("\n") %>
+<% end -%>
+<% if attributes[:deb_field_given?] -%>
+<%   attributes[:deb_field].each do |field, value| -%>
+<%= field %>: <%= value %>
+<%   end -%>
+<% end -%>

data/lib/deb/s3/templates/release.erb

@@ -0,0 +1,16 @@
+Codename: <%= codename %>
+Date: <%= Time.now.utc.strftime("%a, %d %b %Y %T %Z") %>
+Architectures: <%= architectures.join(" ") %>
+Components: <%= components.join(" ") %>
+MD5Sum:
+<% files.each do |f,p| -%>
+ <%= p[:md5] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
+<% end -%>
+SHA1:
+<% files.each do |f,p| -%>
+ <%= p[:sha1] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
+<% end -%>
+SHA256:
+<% files.each do |f,p| -%>
+ <%= p[:sha256] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
+<% end -%>

data/lib/deb/s3/utils.rb

@@ -0,0 +1,71 @@
+require "erb"
+require "tmpdir"
+
+module Deb::S3::Utils
+  module_function
+  def s3; @s3 end
+  def s3= v; @s3 = v end
+  def bucket; @bucket end
+  def bucket= v; @bucket = v end
+  def access_policy; @access_policy end
+  def access_policy= v; @access_policy = v end
+  def signing_key; @signing_key end
+  def signing_key= v; @signing_key = v end
+  def gpg_options; @gpg_options end
+  def gpg_options= v; @gpg_options = v end
+  def prefix; @prefix end
+  def prefix= v; @prefix = v end
+
+  def safesystem(*args)
+    success = system(*args)
+    if !success
+      raise "'system(#{args.inspect})' failed with error code: #{$?.exitstatus}"
+    end
+    return success
+  end
+
+  def debianize_op(op)
+    # Operators in debian packaging are <<, <=, =, >= and >>
+    # So any operator like < or > must be replaced
+    {:< => "<<", :> => ">>"}[op.to_sym] or op
+  end
+
+  def template(path)
+    template_file = File.join(File.dirname(__FILE__), "templates", path)
+    template_code = File.read(template_file)
+    ERB.new(template_code, nil, "-")
+  end
+
+  def s3_path(path)
+    File.join(*[Deb::S3::Utils.prefix, path].compact)
+  end
+
+  # from fog, Fog::AWS.escape
+  def s3_escape(string)
+    string.gsub(/([^a-zA-Z0-9_.\-~]+)/) {
+      "%" + $1.unpack("H2" * $1.bytesize).join("%").upcase
+    }
+  end
+
+  def s3_exists?(path)
+    Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].exists?
+  end
+
+  def s3_read(path)
+    return nil unless s3_exists?(path)
+    Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].read
+  end
+
+  def s3_store(path, filename=nil)
+    filename = File.basename(path) unless filename
+    File.open(path) do |file|
+      o = Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(filename)]
+      o.write(file)
+      o.acl = Deb::S3::Utils.access_policy
+    end
+  end
+
+  def s3_remove(path)
+    Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].delete if s3_exists?(path)
+  end
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: pkgr-deb-s3
+version: !ruby/object:Gem::Version
+  version: 0.6.0
+  prerelease: 
+platform: ruby
+authors:
+- Ken Robertson
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-10-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.18.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.18.0
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.18'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.18'
+description: Easily create and manage an APT repository on S3.
+email: ken@invalidlogic.com
+executables:
+- deb-s3
+extensions: []
+extra_rdoc_files: []
+files:
+- bin/deb-s3
+- lib/deb/s3/cli.rb
+- lib/deb/s3/manifest.rb
+- lib/deb/s3/package.rb
+- lib/deb/s3/release.rb
+- lib/deb/s3/templates/package.erb
+- lib/deb/s3/templates/release.erb
+- lib/deb/s3/utils.rb
+- lib/deb/s3.rb
+- README.md
+homepage: http://invalidlogic.com/
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: Easily create and manage an APT repository on S3.
+test_files: []