dpkg-s3 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8085118424eabe4c62a7016b26f53033f27719496830531aa12dadde91887553
+  data.tar.gz: 8894ff4df72564465523ba841ea680f0e370970ce2fe5922c478d13236c69da1
+SHA512:
+  metadata.gz: 66a81617d7e57fd104e9ec3a03b682e607f2dcdd454148e4fd6a25b778d393ee349cf62a9336f9100cfa51e1e6c7c5b53ed533708f92b4d85452c4eb558b1db0
+  data.tar.gz: 712883d85bc9704a0a009be8674c672911b51351ab74101130c981698b24156f53d3066833d9a1c95cd97610165b39efd211009eb0f21af4e9cab0a18d43bfb8

data/README.md

@@ -0,0 +1,181 @@
+# dpkg-s3
+
+[![Build Status](https://travis-ci.org/gamunu/dpkg-s3.svg?branch=master)](https://travis-ci.org/gamunu/dpkg-s3)
+
+`dpkg-s3` is a simple utility to make creating and managing APT repositories on
+S3.
+
+Most existing guides on using S3 to host an APT repository have you
+using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
+the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
+sync the files to S3.
+
+The annoying thing about this process is it requires you to maintain a local
+copy of the file tree for regenerating and syncing the next time. Personally,
+my process is to use one-off virtual machines with
+[Vagrant](http://vagrantup.com), script out the build process, and then would
+prefer to just upload the final `.deb` from my Mac.
+
+With `dpkg-s3`, there is no need for this. `dpkg-s3` features:
+
+* Downloads the existing package manifest and parses it.
+* Updates it with the new package, replacing the existing entry if already
+  there or adding a new one if not.
+* Uploads the package itself, the Packages manifest, and the Packages.gz
+  manifest. It will skip the uploading if the package is already there.
+* Updates the Release file with the new hashes and file sizes.
+
+## Getting Started
+
+You can simply install it from rubygems:
+
+```console
+$ gem install dpkg-s3
+```
+
+Or to run the code directly, just check out the repo and run Bundler to ensure
+all dependencies are installed:
+
+```console
+$ git clone https://github.com/gamunu/dpkg-s3.git
+$ cd dpkg-s3
+$ bundle install
+```
+
+Now to upload a package, simply use:
+
+```console
+$ dpkg-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
+>> Examining package file my-deb-package-1.0.0_amd64.deb
+>> Retrieving existing package manifest
+>> Uploading package and new manifests to S3
+   -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
+   -- Transferring dists/stable/main/binary-amd64/Packages
+   -- Transferring dists/stable/main/binary-amd64/Packages.gz
+   -- Transferring dists/stable/Release
+>> Update complete.
+```
+
+```
+Usage:
+  dpkg-s3 upload FILES
+
+Options:
+  -a, [--arch=ARCH]                                        # The architecture of the package in the APT repository.
+  -p, [--preserve-versions], [--no-preserve-versions]      # Whether to preserve other versions of a package in the repository when uploading one.
+  -l, [--lock], [--no-lock]                                # Whether to check for an existing lock on the repository to prevent simultaneous updates
+      [--fail-if-exists], [--no-fail-if-exists]            # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
+      [--skip-package-upload], [--no-skip-package-upload]  # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
+  -b, [--bucket=BUCKET]                                    # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                                    # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                                    # The origin to use in the repository Release file.
+      [--suite=SUITE]                                      # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                                # The codename of the APT repository.
+                                                           # Default: stable
+  -m, [--component=COMPONENT]                              # The component of the APT repository.
+                                                           # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                      # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]              # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                              # The region for connecting to S3.
+                                                           # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]        # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                              # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                            # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                           # Default: public
+      [--sign=SIGN]                                        # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                          # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]                    # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                              # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                      # Add cache-control headers to S3 objects.
+
+Uploads the given files to a S3 bucket as an APT repository.
+```
+
+You can also delete packages from the APT repository. Please keep in mind that
+this does NOT delete the .deb file itself, it only removes it from the list of
+packages in the specified component, codename and architecture.
+
+Now to delete the package:
+```console
+$ dpkg-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
+>> Retrieving existing manifests
+   -- Deleting my-deb-package version 1.0.0
+>> Uploading new manifests to S3
+   -- Transferring dists/stable/main/binary-amd64/Packages
+   -- Transferring dists/stable/main/binary-amd64/Packages.gz
+   -- Transferring dists/stable/Release
+>> Update complete.
+```
+
+```
+Usage:
+  dpkg-s3 delete PACKAGE
+
+Options:
+  -a, [--arch=ARCH]                                  # The architecture of the package in the APT repository.
+      [--versions=one two three]                     # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
+  -b, [--bucket=BUCKET]                              # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                              # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                              # The origin to use in the repository Release file.
+      [--suite=SUITE]                                # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                          # The codename of the APT repository.
+                                                     # Default: stable
+  -m, [--component=COMPONENT]                        # The component of the APT repository.
+                                                     # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]        # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                        # The region for connecting to S3.
+                                                     # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]  # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                        # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                      # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                     # Default: public
+      [--sign=SIGN]                                  # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                    # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]              # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                        # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                # Add cache-control headers to S3 objects.
+
+Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
+```
+
+You can also verify an existing APT repository on S3 using the `verify` command:
+
+```console
+dpkg-s3 verify -b my-bucket
+>> Retrieving existing manifests
+>> Checking for missing packages in: stable/main i386
+>> Checking for missing packages in: stable/main amd64
+>> Checking for missing packages in: stable/main all
+```
+
+```
+Usage:
+  dpkg-s3 verify
+
+Options:
+  -f, [--fix-manifests], [--no-fix-manifests]        # Whether to fix problems in manifests when verifying.
+  -b, [--bucket=BUCKET]                              # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                              # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                              # The origin to use in the repository Release file.
+      [--suite=SUITE]                                # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                          # The codename of the APT repository.
+                                                     # Default: stable
+  -m, [--component=COMPONENT]                        # The component of the APT repository.
+                                                     # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]        # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                        # The region for connecting to S3.
+                                                     # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]  # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                        # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                      # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                     # Default: public
+      [--sign=SIGN]                                  # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                    # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]              # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                        # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                # Add cache-control headers to S3 objects.
+
+Verifies that the files in the package manifests exist
+```

data/bin/dpkg-s3

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+require 'pathname'
+$:.unshift File.join(Pathname.new(__FILE__).realpath,'../../lib')
+
+require 'rubygems'
+require 'dpkg/s3/cli'
+
+Dpkg::S3::CLI.start
+

data/lib/dpkg/s3.rb

@@ -0,0 +1,6 @@
+# -*- encoding : utf-8 -*-
+module Dpkg
+  module S3
+    VERSION = "0.1.0"
+  end
+end

data/lib/dpkg/s3/cli.rb

@@ -0,0 +1,641 @@
+# -*- encoding : utf-8 -*-
+require "aws-sdk"
+require "thor"
+
+# Hack: aws requires this!
+require "json"
+
+require "dpkg/s3"
+require "dpkg/s3/utils"
+require "dpkg/s3/manifest"
+require "dpkg/s3/package"
+require "dpkg/s3/release"
+require "dpkg/s3/lock"
+
+class Dpkg::S3::CLI < Thor
+  class_option :bucket,
+  :type     => :string,
+  :aliases  => "-b",
+  :desc     => "The name of the S3 bucket to upload to."
+
+  class_option :prefix,
+  :type     => :string,
+  :desc     => "The path prefix to use when storing on S3."
+
+  class_option :origin,
+  :type     => :string,
+  :aliases  => "-o",
+  :desc     => "The origin to use in the repository Release file."
+
+  class_option :suite,
+  :type     => :string,
+  :desc     => "The suite to use in the repository Release file."
+
+  class_option :codename,
+  :default  => "stable",
+  :type     => :string,
+  :aliases  => "-c",
+  :desc     => "The codename of the APT repository."
+
+  class_option :component,
+  :default  => "main",
+  :type     => :string,
+  :aliases  => "-m",
+  :desc     => "The component of the APT repository."
+
+  class_option :section,
+  :type     => :string,
+  :aliases  => "-s",
+  :hide     => true
+
+  class_option :access_key_id,
+  :type     => :string,
+  :desc     => "The access key for connecting to S3."
+
+  class_option :secret_access_key,
+  :type     => :string,
+  :desc     => "The secret key for connecting to S3."
+
+  class_option :session_token,
+  :type     => :string,
+  :desc     => "The (optional) session token for connecting to S3."
+
+  class_option :endpoint,
+  :type     => :string,
+  :desc     => "The URL endpoint to the S3 API."
+
+  class_option :s3_region,
+  :type     => :string,
+  :desc     => "The region for connecting to S3.",
+  :default  => ENV["AWS_DEFAULT_REGION"] || "us-east-1"
+
+  class_option :force_path_style,
+  :default  => false,
+  :type     => :boolean,
+  :desc     => "Use S3 path style instead of subdomains."
+
+  class_option :proxy_uri,
+  :type     => :string,
+  :desc     => "The URI of the proxy to send service requests through."
+
+  class_option :visibility,
+  :default  => "public",
+  :type     => :string,
+  :aliases  => "-v",
+  :desc     => "The access policy for the uploaded files. " +
+    "Can be public, private, or authenticated."
+
+  class_option :sign,
+  :type     => :string,
+  :desc     => "GPG Sign the Release file when uploading a package, " +
+    "or when verifying it after removing a package. " +
+    "Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B)."
+
+  class_option :gpg_options,
+  :default => "",
+  :type    => :string,
+  :desc    => "Additional command line options to pass to GPG when signing."
+
+  class_option :encryption,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-e",
+  :desc     => "Use S3 server side encryption."
+
+  class_option :quiet,
+  :type => :boolean,
+  :aliases => "-q",
+  :desc => "Doesn't output information, just returns status appropriately."
+
+  class_option :cache_control,
+  :type     => :string,
+  :aliases  => "-C",
+  :desc     => "Add cache-control headers to S3 objects."
+
+  desc "upload FILES",
+  "Uploads the given files to a S3 bucket as an APT repository."
+
+  option :arch,
+  :type     => :string,
+  :aliases  => "-a",
+  :desc     => "The architecture of the package in the APT repository."
+
+  option :preserve_versions,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-p",
+  :desc     => "Whether to preserve other versions of a package " +
+    "in the repository when uploading one."
+
+  option :lock,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-l",
+  :desc     => "Whether to check for an existing lock on the repository " +
+    "to prevent simultaneous updates "
+
+  option :fail_if_exists,
+  :default  => false,
+  :type     => :boolean,
+  :desc     => "Whether to overwrite any existing package that has the same " +
+    "filename in the pool or the same name and version in the manifest but " +
+	"different contents."
+
+  option :skip_package_upload,
+  :default  => false,
+  :type     => :boolean,
+  :desc     => "Whether to skip all package uploads." +
+    "This is useful when hosting .deb files outside of the bucket."
+
+  def upload(*files)
+    if files.nil? || files.empty?
+      error("You must specify at least one file to upload")
+    end
+
+    # make sure all the files exists
+    if missing_file = files.find { |pattern| Dir.glob(pattern).empty? }
+      error("File '#{missing_file}' doesn't exist")
+    end
+
+    # configure AWS::S3
+    configure_s3_client
+
+    begin
+      if options[:lock]
+        log("Checking for existing lock file")
+        if Dpkg::S3::Lock.locked?(options[:codename], component, options[:arch], options[:cache_control])
+          lock = Dpkg::S3::Lock.current(options[:codename], component, options[:arch], options[:cache_control])
+          log("Repository is locked by another user: #{lock.user} at host #{lock.host}")
+          log("Attempting to obtain a lock")
+          Dpkg::S3::Lock.wait_for_lock(options[:codename], component, options[:arch], options[:cache_control])
+        end
+        log("Locking repository for updates")
+        Dpkg::S3::Lock.lock(options[:codename], component, options[:arch], options[:cache_control])
+        @lock_acquired = true
+      end
+
+      # retrieve the existing manifests
+      log("Retrieving existing manifests")
+      release  = Dpkg::S3::Release.retrieve(options[:codename], options[:origin], options[:suite], options[:cache_control])
+      manifests = {}
+      release.architectures.each do |arch|
+        manifests[arch] = Dpkg::S3::Manifest.retrieve(options[:codename], component, arch, options[:cache_control], options[:fail_if_exists], options[:skip_package_upload])
+      end
+
+      packages_arch_all = []
+
+      # examine all the files
+      files.collect { |f| Dir.glob(f) }.flatten.each do |file|
+        log("Examining package file #{File.basename(file)}")
+        pkg = Dpkg::S3::Package.parse_file(file)
+
+        # copy over some options if they weren't given
+        arch = options[:arch] || pkg.architecture
+
+        # If they've specified an arch type that doesn't match the package let them know
+        if options.key?("arch") && options[:arch] != pkg.architecture
+          warn("You specified architecture #{options[:arch]} but package #{pkg.name} has architecture type of #{pkg.architecture}")
+        end
+
+        # validate we have them
+        error("No architcture given and unable to determine one for #{file}. " +
+              "Please specify one with --arch [i386|amd64|armhf].") unless arch
+
+        # If the arch is all and the list of existing manifests is none, then
+        # throw an error. This is mainly the case when initializing a brand new
+        # repository. With "all", we won't know which architectures they're using.
+        if arch == "all" && manifests.count == 0
+          manifests['amd64'] = Dpkg::S3::Manifest.retrieve(options[:codename], component,'amd64', options[:cache_control], options[:fail_if_exists], options[:skip_package_upload])
+          manifests['i386'] = Dpkg::S3::Manifest.retrieve(options[:codename], component,'i386', options[:cache_control], options[:fail_if_exists], options[:skip_package_upload])
+          manifests['armhf'] = Dpkg::S3::Manifest.retrieve(options[:codename], component,'armhf', options[:cache_control], options[:fail_if_exists], options[:skip_package_upload])
+
+         # error("Package #{File.basename(file)} had architecture \"all\", " +
+         #       "however noexisting package lists exist. This can often happen " +
+         #       "if the first package you are add to a new repository is an " +
+         #       "\"all\" architecture file. Please use --arch [i386|amd64|armhf] or " +
+         #       "another platform type to upload the file.")
+        end
+
+        # retrieve the manifest for the arch if we don't have it already
+        manifests[arch] ||= Dpkg::S3::Manifest.retrieve(options[:codename], component, arch, options[:cache_control], options[:fail_if_exists], options[:skip_package_upload])
+
+        # add package in manifests
+        begin
+          manifests[arch].add(pkg, options[:preserve_versions])
+        rescue Dpkg::S3::Utils::AlreadyExistsError => e
+          error("Preparing manifest failed because: #{e}")
+        end
+
+        # If arch is all, we must add this package in all arch available
+        if arch == 'all'
+          packages_arch_all << pkg
+        end
+      end
+
+      manifests.each do |arch, manifest|
+        next if arch == 'all'
+        packages_arch_all.each do |pkg|
+          begin
+            manifest.add(pkg, options[:preserve_versions], false)
+          rescue Dpkg::S3::Utils::AlreadyExistsError => e
+            error("Preparing manifest failed because: #{e}")
+          end
+        end
+      end
+
+      # upload the manifest
+      log("Uploading packages and new manifests to S3")
+      manifests.each_value do |manifest|
+        begin
+          manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
+        rescue Dpkg::S3::Utils::AlreadyExistsError => e
+          error("Uploading manifest failed because: #{e}")
+        end
+        release.update_manifest(manifest)
+      end
+      release.write_to_s3 { |f| sublog("Transferring #{f}") }
+
+      log("Update complete.")
+    ensure
+      if options[:lock] && @lock_acquired
+        Dpkg::S3::Lock.unlock(options[:codename], component, options[:arch], options[:cache_control])
+        log("Lock released.")
+      end
+    end
+  end
+
+  desc "list", "Lists packages in given codename, component, and optionally architecture"
+
+  option :long,
+  :type     => :boolean,
+  :aliases  => '-l',
+  :desc     => "Shows all package information in original format.",
+  :default  => false
+
+  option :arch,
+  :type     => :string,
+  :aliases  => "-a",
+  :desc     => "The architecture of the package in the APT repository."
+
+  def list
+    configure_s3_client
+
+    release = Dpkg::S3::Release.retrieve(options[:codename])
+    archs = release.architectures
+    archs &= [options[:arch]] if options[:arch] && options[:arch] != "all"
+    widths = [0, 0]
+    rows = archs.map { |arch|
+      manifest = Dpkg::S3::Manifest.retrieve(options[:codename], component,
+                                            arch, options[:cache_control],
+                                            false, false)
+      manifest.packages.map do |package|
+        if options[:long]
+          package.generate(options[:codename])
+        else
+          [package.name, package.full_version, package.architecture].tap do |row|
+            row.each_with_index do |col, i|
+              widths[i] = [widths[i], col.size].max if widths[i]
+            end
+          end
+        end
+      end
+    }.flatten(1)
+
+    if options[:long]
+      $stdout.puts rows.join("\n")
+    else
+      rows.each do |row|
+        $stdout.puts "% -#{widths[0]}s  % -#{widths[1]}s  %s" % row
+      end
+    end
+  end
+
+  desc "show PACKAGE VERSION ARCH", "Shows information about a package."
+
+  def show(package_name, version, arch)
+    if version.nil?
+      error "You must specify the name of the package to show."
+    end
+    if version.nil?
+      error "You must specify the version of the package to show."
+    end
+    if arch.nil?
+      error "You must specify the architecture of the package to show."
+    end
+
+    configure_s3_client
+
+    # retrieve the existing manifests
+    manifest = Dpkg::S3::Manifest.retrieve(options[:codename], component, arch,
+                                          options[:cache_control], false, false)
+    package = manifest.packages.detect { |p|
+      p.name == package_name && p.full_version == version
+    }
+    if package.nil?
+      error "No such package found."
+    end
+
+    puts package.generate(options[:codename])
+  end
+
+  desc "copy PACKAGE TO_CODENAME TO_COMPONENT ",
+    "Copy the package named PACKAGE to given codename and component. If --versions is not specified, copy all versions of PACKAGE. Otherwise, only the specified versions will be copied. Source codename and component is given by --codename and --component options."
+
+  option :cache_control,
+  :type     => :string,
+  :aliases  => "-C",
+  :desc     => "Add cache-control headers to S3 objects."
+
+  option :arch,
+    :type     => :string,
+    :aliases  => "-a",
+    :desc     => "The architecture of the package in the APT repository."
+
+  option :versions,
+    :default  => nil,
+    :type     => :array,
+    :desc     => "The space-delimited versions of PACKAGE to delete. If not " +
+    "specified, ALL VERSIONS will be deleted. Fair warning. " +
+    "E.g. --versions \"0.1 0.2 0.3\""
+
+  option :preserve_versions,
+    :default  => false,
+    :type     => :boolean,
+    :aliases  => "-p",
+    :desc     => "Whether to preserve other versions of a package " +
+    "in the repository when uploading one."
+
+  option :fail_if_exists,
+  :default  => true,
+  :type     => :boolean,
+  :desc     => "Whether to overwrite any existing package that has the same " +
+    "filename in the pool or the same name and version in the manifest."
+
+  def copy(package_name, to_codename, to_component)
+    if package_name.nil?
+      error "You must specify a package name."
+    end
+    if to_codename.nil?
+      error "You must specify a codename to copy to."
+    end
+    if to_component.nil?
+      error "You must specify a component to copy to."
+    end
+
+    arch = options[:arch]
+    if arch.nil?
+      error "You must specify the architecture of the package to copy."
+    end
+
+    versions = options[:versions]
+    if versions.nil?
+      warn "===> WARNING: Copying all versions of #{package_name}"
+    else
+      log "Versions to copy: #{versions.join(', ')}"
+    end
+
+    configure_s3_client
+
+    # retrieve the existing manifests
+    log "Retrieving existing manifests"
+    from_manifest = Dpkg::S3::Manifest.retrieve(options[:codename],
+                                               component, arch,
+                                               options[:cache_control],
+                                               false, options[:skip_package_upload])
+    to_release = Dpkg::S3::Release.retrieve(to_codename)
+    to_manifest = Dpkg::S3::Manifest.retrieve(to_codename, to_component, arch,
+                                             options[:cache_control],
+                                             options[:fail_if_exists],
+                                             options[:skip_package_upload])
+    packages = from_manifest.packages.select { |p|
+      p.name == package_name &&
+        (versions.nil? || versions.include?(p.full_version))
+    }
+    if packages.size == 0
+      error "No packages found in repository."
+    end
+
+    packages.each do |package|
+      begin
+        to_manifest.add package, options[:preserve_versions], false
+      rescue Dpkg::S3::Utils::AlreadyExistsError => e
+        error("Preparing manifest failed because: #{e}")
+      end
+    end
+
+    begin
+      to_manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
+    rescue Dpkg::S3::Utils::AlreadyExistsError => e
+      error("Copying manifest failed because: #{e}")
+    end
+    to_release.update_manifest(to_manifest)
+    to_release.write_to_s3 { |f| sublog("Transferring #{f}") }
+
+    log "Copy complete."
+  end
+
+  desc "delete PACKAGE",
+    "Remove the package named PACKAGE. If --versions is not specified, delete" +
+    "all versions of PACKAGE. Otherwise, only the specified versions will be " +
+    "deleted."
+
+  option :arch,
+    :type     => :string,
+    :aliases  => "-a",
+    :desc     => "The architecture of the package in the APT repository."
+
+  option :versions,
+    :default  => nil,
+    :type     => :array,
+    :desc     => "The space-delimited versions of PACKAGE to delete. If not " +
+    "specified, ALL VERSIONS will be deleted. Fair warning. " +
+    "E.g. --versions \"0.1 0.2 0.3\""
+
+  def delete(package)
+    if package.nil?
+      error("You must specify a package name.")
+    end
+
+    versions = options[:versions]
+    if versions.nil?
+      warn("===> WARNING: Deleting all versions of #{package}")
+    else
+      log("Versions to delete: #{versions.join(', ')}")
+    end
+
+    arch = options[:arch]
+    if arch.nil?
+      error("You must specify the architecture of the package to remove.")
+    end
+
+    configure_s3_client
+
+    # retrieve the existing manifests
+    log("Retrieving existing manifests")
+    release  = Dpkg::S3::Release.retrieve(options[:codename], options[:origin], options[:suite])
+    if arch == 'all'
+      selected_arch = release.architectures
+    else
+      selected_arch = [arch]
+    end
+    all_found = 0
+    selected_arch.each { |ar|
+      manifest = Dpkg::S3::Manifest.retrieve(options[:codename], component, ar, options[:cache_control], false, options[:skip_package_upload])
+
+      deleted = manifest.delete_package(package, versions)
+      all_found += deleted.length
+      if deleted.length == 0
+          if versions.nil?
+              sublog("No packages were deleted. #{package} not found in arch #{ar}.")
+              next
+          else
+              sublog("No packages were deleted. #{package} versions #{versions.join(', ')} could not be found in arch #{ar}.")
+              next
+          end
+      else
+          deleted.each { |p|
+              sublog("Deleting #{p.name} version #{p.full_version} from arch #{ar}")
+          }
+      end
+
+      log("Uploading new manifests to S3")
+      manifest.write_to_s3 {|f| sublog("Transferring #{f}") }
+      release.update_manifest(manifest)
+      release.write_to_s3 {|f| sublog("Transferring #{f}") }
+
+      log("Update complete.")
+    }
+    if all_found == 0
+      if versions.nil?
+        error("No packages were deleted. #{package} not found.")
+      else
+        error("No packages were deleted. #{package} versions #{versions.join(', ')} could not be found.")
+      end
+    end
+
+  end
+
+
+  desc "verify", "Verifies that the files in the package manifests exist"
+
+  option :fix_manifests,
+  :default  => false,
+  :type     => :boolean,
+  :aliases  => "-f",
+  :desc     => "Whether to fix problems in manifests when verifying."
+
+  def verify
+    configure_s3_client
+
+    log("Retrieving existing manifests")
+    release = Dpkg::S3::Release.retrieve(options[:codename], options[:origin], options[:suite])
+
+    release.architectures.each do |arch|
+      log("Checking for missing packages in: #{options[:codename]}/#{options[:component]} #{arch}")
+      manifest = Dpkg::S3::Manifest.retrieve(options[:codename], component,
+                                            arch, options[:cache_control], false,
+                                            options[:skip_package_upload])
+      missing_packages = []
+
+      manifest.packages.each do |p|
+        unless Dpkg::S3::Utils.s3_exists? p.url_filename_encoded(options[:codename])
+          sublog("The following packages are missing:\n\n") if missing_packages.empty?
+          puts(p.generate(options[:codename]))
+          puts("")
+
+          missing_packages << p
+        end
+      end
+
+      if options[:sign] || (options[:fix_manifests] && !missing_packages.empty?)
+        log("Removing #{missing_packages.length} package(s) from the manifest...")
+        missing_packages.each { |p| manifest.packages.delete(p) }
+        manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
+        release.update_manifest(manifest)
+        release.write_to_s3 { |f| sublog("Transferring #{f}") }
+
+        log("Update complete.")
+      end
+    end
+  end
+
+  private
+
+  def component
+    return @component if @component
+    @component = if (section = options[:section])
+                   warn("===> WARNING: The --section/-s argument is " \
+                        "deprecated, please use --component/-m.")
+                   section
+                 else
+                   options[:component]
+                 end
+  end
+
+  def puts(*args)
+    $stdout.puts(*args) unless options[:quiet]
+  end
+
+  def log(message)
+    puts ">> #{message}" unless options[:quiet]
+  end
+
+  def sublog(message)
+    puts "   -- #{message}" unless options[:quiet]
+  end
+
+  def error(message)
+    $stderr.puts "!! #{message}" unless options[:quiet]
+    exit 1
+  end
+
+  def provider
+    access_key_id     = options[:access_key_id]
+    secret_access_key = options[:secret_access_key]
+    session_token     = options[:session_token]
+
+    if access_key_id.nil? ^ secret_access_key.nil?
+      error("If you specify one of --access-key-id or --secret-access-key, you must specify the other.")
+    end
+    static_credentials = {}
+    static_credentials[:access_key_id]     = access_key_id     if access_key_id
+    static_credentials[:secret_access_key] = secret_access_key if secret_access_key
+    static_credentials[:session_token]     = session_token     if session_token
+
+    static_credentials
+  end
+
+  def configure_s3_client
+    error("No value provided for required options '--bucket'") unless options[:bucket]
+
+    settings = {
+      :region => options[:s3_region],
+      :http_proxy   => options[:proxy_uri],
+      :force_path_style => options[:force_path_style]
+    }
+    settings[:endpoint] = options[:endpoint] if options[:endpoint]
+    settings.merge!(provider)
+
+    Dpkg::S3::Utils.s3          = Aws::S3::Client.new(settings)
+    Dpkg::S3::Utils.bucket      = options[:bucket]
+    Dpkg::S3::Utils.signing_key = options[:sign]
+    Dpkg::S3::Utils.gpg_options = options[:gpg_options]
+    Dpkg::S3::Utils.prefix      = options[:prefix]
+    Dpkg::S3::Utils.encryption  = options[:encryption]
+
+    # make sure we have a valid visibility setting
+    Dpkg::S3::Utils.access_policy =
+      case options[:visibility]
+      when "public"
+        "public-read"
+      when "private"
+        "private"
+      when "authenticated"
+        "authenticated-read"
+      when "bucket_owner"
+        "bucket-owner-full-control"
+      else
+        error("Invalid visibility setting given. Can be public, private, authenticated, or bucket_owner.")
+      end
+  end
+end