pkgr-deb-s3 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,112 @@
1
+ # deb-s3
2
+
3
+ `deb-s3` is a simple utility to make creating and managing APT repositories on
4
+ S3.
5
+
6
+ Most existing existing guides on using S3 to host an APT repository have you
7
+ using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
8
+ the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
9
+ sync the files to S3.
10
+
11
+ The annoying thing about this process is it requires you to maintain a local
12
+ copy of the file tree for regenerating and syncing the next time. Personally,
13
+ my process is to use one-off virtual machines with
14
+ [Vagrant](http://vagrantup.com), script out the build process, and then would
15
+ prefer to just upload the final `.deb` from my Mac.
16
+
17
+ With `deb-s3`, there is no need for this. `deb-s3` features:
18
+
19
+ * Downloads the existing package manifest and parses it.
20
+ * Updates it with the new package, replacing the existing entry if already
21
+ there or adding a new one if not.
22
+ * Uploads the package itself, the Packages manifest, and the Packages.gz
23
+ manifest.
24
+ * Updates the Release file with the new hashes and file sizes.
25
+
26
+ ## Getting Started
27
+
28
+ You can simply install it from rubygems:
29
+
30
+ ```console
31
+ $ gem install deb-s3
32
+ ```
33
+
34
+ Or to run the code directly, just check out the repo and run Bundler to ensure
35
+ all dependencies are installed:
36
+
37
+ ```console
38
+ $ git clone https://github.com/krobertson/deb-s3.git
39
+ $ cd deb-s3
40
+ $ bundle install
41
+ ```
42
+
43
+ Now to upload a package, simply use:
44
+
45
+ ```console
46
+ $ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
47
+ >> Examining package file my-deb-package-1.0.0_amd64.deb
48
+ >> Retrieving existing package manifest
49
+ >> Uploading package and new manifests to S3
50
+ -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
51
+ -- Transferring dists/stable/main/binary-amd64/Packages
52
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
53
+ -- Transferring dists/stable/Release
54
+ >> Update complete.
55
+ ```
56
+
57
+ ```
58
+ Usage:
59
+ deb-s3 upload FILES
60
+
61
+ Options:
62
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
63
+ [--sign=SIGN] # Sign the Release file. Use --sign with your key ID to use a specific key.
64
+ -p, [--preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
65
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
66
+ -c, [--codename=CODENAME] # The codename of the APT repository.
67
+ # Default: stable
68
+ -m, [--component=COMPONENT] # The component of the APT repository.
69
+ # Default: main
70
+ [--access-key-id=ACCESS_KEY] # The access key for connecting to S3.
71
+ [--secret-access-key=SECRET_KEY] # The secret key for connecting to S3.
72
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
73
+ # Default: public
74
+
75
+ Uploads the given files to a S3 bucket as an APT repository.
76
+ ```
77
+
78
+ You can also verify an existing APT repository on S3 using the `verify` command:
79
+
80
+ ```console
81
+ deb-s3 verify -b my-bucket
82
+ >> Retrieving existing manifests
83
+ >> Checking for missing packages in: stable/main i386
84
+ >> Checking for missing packages in: stable/main amd64
85
+ >> Checking for missing packages in: stable/main all
86
+ ```
87
+
88
+ ```
89
+ Usage:
90
+ deb-s3 verify
91
+
92
+ Options:
93
+ -f, [--fix-manifests] # Whether to fix problems in manifests when verifying.
94
+ [--sign=SIGN] # Sign the Release file. Use --sign with your key ID to use a specific key.
95
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
96
+ -c, [--codename=CODENAME] # The codename of the APT repository.
97
+ # Default: stable
98
+ -m, [--component=COMPONENT] # The component of the APT repository.
99
+ # Default: main
100
+ [--access-key-id=ACCESS_KEY] # The access key for connecting to S3.
101
+ [--secret-access-key=SECRET_KEY] # The secret key for connecting to S3.
102
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
103
+ # Default: public
104
+
105
+ Verifies that the files in the package manifests exist
106
+ ```
107
+
108
+ ## TODO
109
+
110
+ This is still experimental. These are several things to be done:
111
+
112
+ * Don't re-upload a package if it already exists and has the same hashes.
@@ -0,0 +1,9 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $:.unshift File.expand_path('../../lib', __FILE__)
4
+
5
+ require 'rubygems'
6
+ require 'deb/s3/cli'
7
+
8
+ Deb::S3::CLI.start
9
+
@@ -0,0 +1,5 @@
1
+ module Deb
2
+ module S3
3
+ VERSION = "0.6.0"
4
+ end
5
+ end
@@ -0,0 +1,234 @@
1
+ require "aws"
2
+ require "thor"
3
+
4
+ # Hack: aws requires this!
5
+ require "json"
6
+
7
+ require "deb/s3"
8
+ require "deb/s3/utils"
9
+ require "deb/s3/manifest"
10
+ require "deb/s3/package"
11
+ require "deb/s3/release"
12
+
13
+ class Deb::S3::CLI < Thor
14
+ class_option :bucket,
15
+ :type => :string,
16
+ :aliases => "-b",
17
+ :desc => "The name of the S3 bucket to upload to."
18
+
19
+ class_option :prefix,
20
+ :type => :string,
21
+ :desc => "The path prefix to use when storing on S3."
22
+
23
+ class_option :codename,
24
+ :default => "stable",
25
+ :type => :string,
26
+ :aliases => "-c",
27
+ :desc => "The codename of the APT repository."
28
+
29
+ class_option :component,
30
+ :default => "main",
31
+ :type => :string,
32
+ :aliases => "-m",
33
+ :desc => "The component of the APT repository."
34
+
35
+ class_option :section,
36
+ :type => :string,
37
+ :aliases => "-s",
38
+ :hide => true
39
+
40
+ class_option :access_key_id,
41
+ :type => :string,
42
+ :desc => "The access key for connecting to S3."
43
+
44
+ class_option :secret_access_key,
45
+ :type => :string,
46
+ :desc => "The secret key for connecting to S3."
47
+
48
+ class_option :visibility,
49
+ :default => "public",
50
+ :type => :string,
51
+ :aliases => "-v",
52
+ :desc => "The access policy for the uploaded files. " +
53
+ "Can be public, private, or authenticated."
54
+
55
+ class_option :sign,
56
+ :type => :string,
57
+ :desc => "Sign the Release file when uploading a package," +
58
+ "or when verifying it after removing a package." +
59
+ "Use --sign with your key ID to use a specific key."
60
+
61
+ class_option :gpg_options,
62
+ :default => "",
63
+ :type => :string,
64
+ :desc => "Additional command line options to pass to GPG when signing"
65
+
66
+ desc "upload FILES",
67
+ "Uploads the given files to a S3 bucket as an APT repository."
68
+
69
+ option :arch,
70
+ :type => :string,
71
+ :aliases => "-a",
72
+ :desc => "The architecture of the package in the APT repository."
73
+
74
+ option :preserve_versions,
75
+ :default => false,
76
+ :type => :boolean,
77
+ :aliases => "-p",
78
+ :desc => "Whether to preserve other versions of a package " +
79
+ "in the repository when uploading one."
80
+
81
+ def upload(*files)
82
+ component = options[:component]
83
+ if options[:section]
84
+ component = options[:section]
85
+ warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
86
+ end
87
+
88
+ if files.nil? || files.empty?
89
+ error("You must specify at least one file to upload")
90
+ end
91
+
92
+ # make sure all the files exists
93
+ if missing_file = files.detect { |f| !File.exists?(f) }
94
+ error("File '#{missing_file}' doesn't exist")
95
+ end
96
+
97
+ # configure AWS::S3
98
+ configure_s3_client
99
+
100
+ # retrieve the existing manifests
101
+ log("Retrieving existing manifests")
102
+ release = Deb::S3::Release.retrieve(options[:codename])
103
+ manifests = {}
104
+
105
+ # examine all the files
106
+ files.collect { |f| Dir.glob(f) }.flatten.each do |file|
107
+ log("Examining package file #{File.basename(file)}")
108
+ pkg = Deb::S3::Package.parse_file(file)
109
+
110
+ # copy over some options if they weren't given
111
+ arch = options[:arch] || pkg.architecture
112
+
113
+ # validate we have them
114
+ error("No architcture given and unable to determine one for #{file}. " +
115
+ "Please specify one with --arch [i386,amd64].") unless arch
116
+
117
+ # retrieve the manifest for the arch if we don't have it already
118
+ manifests[arch] ||= Deb::S3::Manifest.retrieve(options[:codename], component, arch)
119
+
120
+ # add in the package
121
+ manifests[arch].add(pkg, options[:preserve_versions])
122
+ end
123
+
124
+ # upload the manifest
125
+ log("Uploading packages and new manifests to S3")
126
+ manifests.each_value do |manifest|
127
+ manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
128
+ release.update_manifest(manifest)
129
+ end
130
+ release.write_to_s3 { |f| sublog("Transferring #{f}") }
131
+
132
+ log("Update complete.")
133
+ end
134
+
135
+ desc "verify", "Verifies that the files in the package manifests exist"
136
+
137
+ option :fix_manifests,
138
+ :default => false,
139
+ :type => :boolean,
140
+ :aliases => "-f",
141
+ :desc => "Whether to fix problems in manifests when verifying."
142
+
143
+ def verify
144
+ component = options[:component]
145
+ if options[:section]
146
+ component = options[:section]
147
+ warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
148
+ end
149
+
150
+ configure_s3_client
151
+
152
+ log("Retrieving existing manifests")
153
+ release = Deb::S3::Release.retrieve(options[:codename])
154
+
155
+ %w[amd64 armel i386 all].each do |arch|
156
+ log("Checking for missing packages in: #{options[:codename]}/#{options[:component]} #{arch}")
157
+ manifest = Deb::S3::Manifest.retrieve(options[:codename], component, arch)
158
+ missing_packages = []
159
+
160
+ manifest.packages.each do |p|
161
+ unless Deb::S3::Utils.s3_exists? p.url_filename_encoded
162
+ sublog("The following packages are missing:\n\n") if missing_packages.empty?
163
+ puts(p.generate)
164
+ puts("")
165
+
166
+ missing_packages << p
167
+ end
168
+ end
169
+
170
+ if options[:fix_manifests] && !missing_packages.empty?
171
+ log("Removing #{missing_packages.length} package(s) from the manifest...")
172
+ missing_packages.each { |p| manifest.packages.delete(p) }
173
+ manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
174
+ release.update_manifest(manifest)
175
+ release.write_to_s3 { |f| sublog("Transferring #{f}") }
176
+
177
+ log("Update complete.")
178
+ end
179
+ end
180
+ end
181
+
182
+ private
183
+
184
+ def log(message)
185
+ puts ">> #{message}"
186
+ end
187
+
188
+ def sublog(message)
189
+ puts " -- #{message}"
190
+ end
191
+
192
+ def error(message)
193
+ puts "!! #{message}"
194
+ exit 1
195
+ end
196
+
197
+ def provider
198
+ access_key_id = options[:access_key_id]
199
+ secret_access_key = options[:secret_access_key]
200
+
201
+ if access_key_id.nil? ^ secret_access_key.nil?
202
+ error("If you specify one of --access-key-id or --secret-access-key, you must specify the other.")
203
+ end
204
+
205
+ static_credentials = {}
206
+ static_credentials[:access_key_id] = access_key_id if access_key_id
207
+ static_credentials[:secret_access_key] = secret_access_key if secret_access_key
208
+
209
+ AWS::Core::CredentialProviders::DefaultProvider.new(static_credentials)
210
+ end
211
+
212
+ def configure_s3_client
213
+ error("No value provided for required options '--bucket'") unless options[:bucket]
214
+
215
+ Deb::S3::Utils.s3 = AWS::S3.new(provider.credentials)
216
+ Deb::S3::Utils.bucket = options[:bucket]
217
+ Deb::S3::Utils.signing_key = options[:sign]
218
+ Deb::S3::Utils.gpg_options = options[:gpg_options]
219
+ Deb::S3::Utils.prefix = options[:prefix]
220
+
221
+ # make sure we have a valid visibility setting
222
+ Deb::S3::Utils.access_policy =
223
+ case options[:visibility]
224
+ when "public"
225
+ :public_read
226
+ when "private"
227
+ :private
228
+ when "authenticated"
229
+ :authenticated_read
230
+ else
231
+ error("Invalid visibility setting given. Can be public, private, or authenticated.")
232
+ end
233
+ end
234
+ end
@@ -0,0 +1,104 @@
1
+ require "tempfile"
2
+ require "zlib"
3
+
4
+ class Deb::S3::Manifest
5
+ include Deb::S3::Utils
6
+
7
+ attr_accessor :codename
8
+ attr_accessor :component
9
+ attr_accessor :architecture
10
+
11
+ attr_accessor :files
12
+
13
+ def initialize
14
+ @packages = []
15
+ @component = nil
16
+ @architecture = nil
17
+ @files = {}
18
+ end
19
+
20
+ class << self
21
+ def retrieve(codename, component, architecture)
22
+ m = if s = Deb::S3::Utils.s3_read("dists/#{codename}/#{component}/binary-#{architecture}/Packages")
23
+ self.parse_packages(s)
24
+ else
25
+ self.new
26
+ end
27
+
28
+ m.codename = codename
29
+ m.component = component
30
+ m.architecture = architecture
31
+ m
32
+ end
33
+
34
+ def parse_packages(str)
35
+ m = self.new
36
+ str.split("\n\n").each do |s|
37
+ next if s.chomp.empty?
38
+ m.packages << Deb::S3::Package.parse_string(s)
39
+ end
40
+ m
41
+ end
42
+ end
43
+
44
+ def packages
45
+ @packages
46
+ end
47
+
48
+ def add(pkg, preserve_versions)
49
+ if preserve_versions
50
+ @packages.delete_if { |p| p.name == pkg.name && p.version == pkg.version }
51
+ else
52
+ @packages.delete_if { |p| p.name == pkg.name }
53
+ end
54
+ @packages << pkg
55
+ pkg
56
+ end
57
+
58
+ def generate
59
+ @packages.collect { |pkg| pkg.generate }.join("\n")
60
+ end
61
+
62
+ def write_to_s3
63
+ manifest = self.generate
64
+
65
+ # store any packages that need to be stored
66
+ @packages.each do |pkg|
67
+ if pkg.needs_uploading?
68
+ yield pkg.url_filename if block_given?
69
+ s3_store(pkg.filename, pkg.url_filename_encoded)
70
+ end
71
+ end
72
+
73
+ # generate the Packages file
74
+ pkgs_temp = Tempfile.new("Packages")
75
+ pkgs_temp.write manifest
76
+ pkgs_temp.close
77
+ f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages"
78
+ yield f if block_given?
79
+ s3_store(pkgs_temp.path, f)
80
+ @files["#{@component}/binary-#{@architecture}/Packages"] = hashfile(pkgs_temp.path)
81
+ pkgs_temp.unlink
82
+
83
+ # generate the Packages.gz file
84
+ gztemp = Tempfile.new("Packages.gz")
85
+ gztemp.close
86
+ Zlib::GzipWriter.open(gztemp.path) { |gz| gz.write manifest }
87
+ f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages.gz"
88
+ yield f if block_given?
89
+ s3_store(gztemp.path, f)
90
+ @files["#{@component}/binary-#{@architecture}/Packages.gz"] = hashfile(gztemp.path)
91
+ gztemp.unlink
92
+
93
+ nil
94
+ end
95
+
96
+ def hashfile(path)
97
+ {
98
+ :size => File.size(path),
99
+ :sha1 => Digest::SHA1.file(path).hexdigest,
100
+ :sha256 => Digest::SHA2.file(path).hexdigest,
101
+ :md5 => Digest::MD5.file(path).hexdigest
102
+ }
103
+ end
104
+ end
@@ -0,0 +1,273 @@
1
+ require "digest/sha1"
2
+ require "digest/sha2"
3
+ require "digest/md5"
4
+ require "socket"
5
+ require "tmpdir"
6
+
7
+ class Deb::S3::Package
8
+ include Deb::S3::Utils
9
+
10
+ attr_accessor :name
11
+ attr_accessor :version
12
+ attr_accessor :epoch
13
+ attr_accessor :iteration
14
+ attr_accessor :maintainer
15
+ attr_accessor :vendor
16
+ attr_accessor :url
17
+ attr_accessor :category
18
+ attr_accessor :license
19
+ attr_accessor :architecture
20
+ attr_accessor :description
21
+
22
+ attr_accessor :dependencies
23
+ attr_accessor :provides
24
+ attr_accessor :conflicts
25
+ attr_accessor :replaces
26
+ attr_accessor :excludes
27
+
28
+
29
+ # Any other attributes specific to this package.
30
+ # This is where you'd put rpm, deb, or other specific attributes.
31
+ attr_accessor :attributes
32
+
33
+ # hashes
34
+ attr_accessor :url_filename
35
+ attr_accessor :sha1
36
+ attr_accessor :sha256
37
+ attr_accessor :md5
38
+ attr_accessor :size
39
+
40
+ attr_accessor :filename
41
+
42
+ class << self
43
+ include Deb::S3::Utils
44
+
45
+ def parse_file(package)
46
+ p = self.new
47
+ p.extract_info(extract_control(package))
48
+ p.apply_file_info(package)
49
+ p.filename = package
50
+ p
51
+ end
52
+
53
+ def parse_string(s)
54
+ p = self.new
55
+ p.extract_info(s)
56
+ p
57
+ end
58
+
59
+ def extract_control(package)
60
+ if system("which dpkg 2>&1 >/dev/null")
61
+ `dpkg -f #{package}`
62
+ else
63
+ Dir.mktmpdir do |path|
64
+ safesystem("ar p #{package} control.tar.gz | tar -zxf - -C #{path}")
65
+ File.read(File.join(path, "control"))
66
+ end
67
+ end
68
+ end
69
+ end
70
+
71
+ def initialize
72
+ @attributes = {}
73
+
74
+ # Reference
75
+ # http://www.debian.org/doc/manuals/maint-guide/first.en.html
76
+ # http://wiki.debian.org/DeveloperConfiguration
77
+ # https://github.com/jordansissel/fpm/issues/37
78
+ if ENV.include?("DEBEMAIL") and ENV.include?("DEBFULLNAME")
79
+ # Use DEBEMAIL and DEBFULLNAME as the default maintainer if available.
80
+ @maintainer = "#{ENV["DEBFULLNAME"]} <#{ENV["DEBEMAIL"]}>"
81
+ else
82
+ # TODO(sissel): Maybe support using 'git config' for a default as well?
83
+ # git config --get user.name, etc can be useful.
84
+ #
85
+ # Otherwise default to user@currenthost
86
+ @maintainer = "<#{ENV["USER"]}@#{Socket.gethostname}>"
87
+ end
88
+
89
+ @name = nil
90
+ @architecture = "native"
91
+ @description = "no description given"
92
+ @version = nil
93
+ @epoch = nil
94
+ @iteration = nil
95
+ @url = nil
96
+ @category = "default"
97
+ @license = "unknown"
98
+ @vendor = "none"
99
+ @sha1 = nil
100
+ @sha256 = nil
101
+ @md5 = nil
102
+ @size = nil
103
+ @filename = nil
104
+ @url_filename = nil
105
+
106
+ @provides = []
107
+ @conflicts = []
108
+ @replaces = []
109
+ @dependencies = []
110
+
111
+ @needs_uploading = false
112
+ end
113
+
114
+ def filename=(f)
115
+ @filename = f
116
+ @needs_uploading = true
117
+ @filename
118
+ end
119
+
120
+ def url_filename
121
+ @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{File.basename(self.filename)}"
122
+ end
123
+
124
+ def url_filename_encoded
125
+ @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{s3_escape(File.basename(self.filename))}"
126
+ end
127
+
128
+ def needs_uploading?
129
+ @needs_uploading
130
+ end
131
+
132
+ def generate
133
+ template("package.erb").result(binding)
134
+ end
135
+
136
+ # from fpm
137
+ def parse_depends(data)
138
+ return [] if data.nil? or data.empty?
139
+ # parse dependencies. Debian dependencies come in one of two forms:
140
+ # * name
141
+ # * name (op version)
142
+ # They are all on one line, separated by ", "
143
+
144
+ dep_re = /^([^ ]+)(?: \(([>=<]+) ([^)]+)\))?$/
145
+ return data.split(/, */).collect do |dep|
146
+ m = dep_re.match(dep)
147
+ if m
148
+ name, op, version = m.captures
149
+ # this is the proper form of dependency
150
+ if op && version && op != "" && version != ""
151
+ "#{name} (#{op} #{version})".strip
152
+ else
153
+ name.strip
154
+ end
155
+ else
156
+ # Assume normal form dependency, "name op version".
157
+ dep
158
+ end
159
+ end
160
+ end # def parse_depends
161
+
162
+ # from fpm
163
+ def fix_dependency(dep)
164
+ # Deb dependencies are: NAME (OP VERSION), like "zsh (> 3.0)"
165
+ # Convert anything that looks like 'NAME OP VERSION' to this format.
166
+ if dep =~ /[\(,\|]/
167
+ # Don't "fix" ones that could appear well formed already.
168
+ else
169
+ # Convert ones that appear to be 'name op version'
170
+ name, op, version = dep.split(/ +/)
171
+ if !version.nil?
172
+ # Convert strings 'foo >= bar' to 'foo (>= bar)'
173
+ dep = "#{name} (#{debianize_op(op)} #{version})"
174
+ end
175
+ end
176
+
177
+ name_re = /^[^ \(]+/
178
+ name = dep[name_re]
179
+ if name =~ /[A-Z]/
180
+ dep = dep.gsub(name_re) { |n| n.downcase }
181
+ end
182
+
183
+ if dep.include?("_")
184
+ dep = dep.gsub("_", "-")
185
+ end
186
+
187
+ # Convert gem ~> X.Y.Z to '>= X.Y.Z' and << X.Y+1.0
188
+ if dep =~ /\(~>/
189
+ name, version = dep.gsub(/[()~>]/, "").split(/ +/)[0..1]
190
+ nextversion = version.split(".").collect { |v| v.to_i }
191
+ l = nextversion.length
192
+ nextversion[l-2] += 1
193
+ nextversion[l-1] = 0
194
+ nextversion = nextversion.join(".")
195
+ return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
196
+ elsif (m = dep.match(/(\S+)\s+\(!= (.+)\)/))
197
+ # Append this to conflicts
198
+ self.conflicts += [dep.gsub(/!=/,"=")]
199
+ return []
200
+ elsif (m = dep.match(/(\S+)\s+\(= (.+)\)/)) and
201
+ self.attributes[:deb_ignore_iteration_in_dependencies?]
202
+ # Convert 'foo (= x)' to 'foo (>= x)' and 'foo (<< x+1)'
203
+ # but only when flag --ignore-iteration-in-dependencies is passed.
204
+ name, version = m[1..2]
205
+ nextversion = version.split('.').collect { |v| v.to_i }
206
+ nextversion[-1] += 1
207
+ nextversion = nextversion.join(".")
208
+ return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
209
+ else
210
+ # otherwise the dep is probably fine
211
+ return dep.rstrip
212
+ end
213
+ end # def fix_dependency
214
+
215
+ # from fpm
216
+ def extract_info(control)
217
+ parse = lambda do |field|
218
+ value = control[/^#{field}: .*/]
219
+ if value.nil?
220
+ return nil
221
+ else
222
+ return value.split(": ",2).last
223
+ end
224
+ end
225
+
226
+ # Parse 'epoch:version-iteration' in the version string
227
+ version_re = /^(?:([0-9]+):)?(.+?)(?:-(.*))?$/
228
+ m = version_re.match(parse.call("Version"))
229
+ if !m
230
+ raise "Unsupported version string '#{parse.call("Version")}'"
231
+ end
232
+ self.epoch, self.version, self.iteration = m.captures
233
+
234
+ self.architecture = parse.call("Architecture")
235
+ self.category = parse.call("Section")
236
+ self.license = parse.call("License") || self.license
237
+ self.maintainer = parse.call("Maintainer")
238
+ self.name = parse.call("Package")
239
+ self.url = parse.call("Homepage")
240
+ self.vendor = parse.call("Vendor") || self.vendor
241
+ self.attributes[:deb_priority] = parse.call("Priority")
242
+ self.attributes[:deb_installed_size] = parse.call("Installed-Size")
243
+
244
+ # Packages manifest fields
245
+ self.url_filename = parse.call("Filename")
246
+ self.sha1 = parse.call("SHA1")
247
+ self.sha256 = parse.call("SHA256")
248
+ self.md5 = parse.call("MD5sum")
249
+ self.size = parse.call("Size")
250
+
251
+ # The description field is a special flower, parse it that way.
252
+ # The description is the first line as a normal Description field, but also continues
253
+ # on future lines indented by one space, until the end of the file. Blank
254
+ # lines are marked as ' .'
255
+ description = control[/^Description: .*[^\Z]/m]
256
+ description = description.gsub(/^[^(Description|\s)].*$/, "").split(": ", 2).last
257
+ self.description = description.gsub(/^ /, "").gsub(/^\.$/, "")
258
+
259
+ #self.config_files = config_files
260
+
261
+ self.dependencies += Array(parse_depends(parse.call("Depends")))
262
+ self.conflicts += Array(parse_depends(parse.call("Conflicts")))
263
+ self.provides += Array(parse_depends(parse.call("Provides")))
264
+ self.replaces += Array(parse_depends(parse.call("Replaces")))
265
+ end # def extract_info
266
+
267
+ def apply_file_info(file)
268
+ self.size = File.size(file)
269
+ self.sha1 = Digest::SHA1.file(file).hexdigest
270
+ self.sha256 = Digest::SHA2.file(file).hexdigest
271
+ self.md5 = Digest::MD5.file(file).hexdigest
272
+ end
273
+ end
@@ -0,0 +1,139 @@
1
+ require "tempfile"
2
+
3
+ class Deb::S3::Release
4
+ include Deb::S3::Utils
5
+
6
+ attr_accessor :codename
7
+ attr_accessor :architectures
8
+ attr_accessor :components
9
+
10
+ attr_accessor :files
11
+ attr_accessor :policy
12
+
13
+ def initialize
14
+ @codename = nil
15
+ @architectures = []
16
+ @components = []
17
+ @files = {}
18
+ @policy = :public_read
19
+ end
20
+
21
+ class << self
22
+ def retrieve(codename)
23
+ if s = Deb::S3::Utils.s3_read("dists/#{codename}/Release")
24
+ self.parse_release(s)
25
+ else
26
+ rel = self.new
27
+ rel.codename = codename
28
+ rel
29
+ end
30
+ end
31
+
32
+ def parse_release(str)
33
+ rel = self.new
34
+ rel.parse(str)
35
+ rel
36
+ end
37
+ end
38
+
39
+ def filename
40
+ "dists/#{@codename}/Release"
41
+ end
42
+
43
+ def parse(str)
44
+ parse = lambda do |field|
45
+ value = str[/^#{field}: .*/]
46
+ if value.nil?
47
+ return nil
48
+ else
49
+ return value.split(": ",2).last
50
+ end
51
+ end
52
+
53
+ # grab basic fields
54
+ self.codename = parse.call("Codename")
55
+ self.architectures = (parse.call("Architectures") || "").split(/\s+/)
56
+ self.components = (parse.call("Components") || "").split(/\s+/)
57
+
58
+ # find all the hashes
59
+ str.scan(/^\s+([^\s]+)\s+(\d+)\s+(.+)$/).each do |(hash,size,name)|
60
+ self.files[name] ||= { :size => size.to_i }
61
+ case hash.length
62
+ when 32
63
+ self.files[name][:md5] = hash
64
+ when 40
65
+ self.files[name][:sha1] = hash
66
+ when 64
67
+ self.files[name][:sha256] = hash
68
+ end
69
+ end
70
+ end
71
+
72
+ def generate
73
+ template("release.erb").result(binding)
74
+ end
75
+
76
+ def write_to_s3
77
+ # validate some other files are present
78
+ if block_given?
79
+ self.validate_others { |f| yield f }
80
+ else
81
+ self.validate_others
82
+ end
83
+
84
+ # generate the Release file
85
+ release_tmp = Tempfile.new("Release")
86
+ release_tmp.puts self.generate
87
+ release_tmp.close
88
+ yield self.filename if block_given?
89
+ s3_store(release_tmp.path, self.filename)
90
+
91
+ # sign the file, if necessary
92
+ if Deb::S3::Utils.signing_key
93
+ key_param = Deb::S3::Utils.signing_key != "" ? "--default-key=#{Deb::S3::Utils.signing_key}" : ""
94
+ if system("gpg -a #{key_param} #{Deb::S3::Utils.gpg_options} -b #{release_tmp.path}")
95
+ local_file = release_tmp.path+".asc"
96
+ remote_file = self.filename+".gpg"
97
+ yield remote_file if block_given?
98
+ raise "Unable to locate Release signature file" unless File.exists?(local_file)
99
+ s3_store(local_file, remote_file)
100
+ File.unlink(local_file)
101
+ else
102
+ raise "Signing the Release file failed."
103
+ end
104
+ else
105
+ # remove an existing Release.gpg, if it was there
106
+ s3_remove(self.filename+".gpg")
107
+ end
108
+
109
+ release_tmp.unlink
110
+ end
111
+
112
+ def update_manifest(manifest)
113
+ self.components << manifest.component unless self.components.include?(manifest.component)
114
+ self.architectures << manifest.architecture unless self.architectures.include?(manifest.architecture)
115
+ self.files.merge!(manifest.files)
116
+ end
117
+
118
+ def validate_others
119
+ to_apply = []
120
+ self.components.each do |comp|
121
+ %w(amd64 i386).each do |arch|
122
+ next if self.files.has_key?("#{comp}/binary-#{arch}/Packages")
123
+
124
+ m = Deb::S3::Manifest.new
125
+ m.codename = self.codename
126
+ m.component = comp
127
+ m.architecture = arch
128
+ if block_given?
129
+ m.write_to_s3 { |f| yield f }
130
+ else
131
+ m.write_to_s3
132
+ end
133
+ to_apply << m
134
+ end
135
+ end
136
+
137
+ to_apply.each { |m| self.update_manifest(m) }
138
+ end
139
+ end
@@ -0,0 +1,57 @@
1
+ Package: <%= name %>
2
+ Version: <%= "#{epoch}:" if epoch %><%= version %><%= "-" + iteration.to_s if iteration %>
3
+ License: <%= license %>
4
+ Vendor: <%= vendor %>
5
+ Architecture: <%= architecture %>
6
+ Maintainer: <%= maintainer %>
7
+ Installed-Size: <%= attributes[:deb_installed_size] %>
8
+ <% if !dependencies.empty? and !attributes[:no_depends?] -%>
9
+ Depends: <%= dependencies.collect { |d| fix_dependency(d) }.flatten.join(", ") %>
10
+ <% end -%>
11
+ <% if !conflicts.empty? -%>
12
+ Conflicts: <%= conflicts.join(", ") %>
13
+ <% end -%>
14
+ <% if attributes[:deb_pre_depends_given?] -%>
15
+ Pre-Depends: <%= attributes[:deb_pre_depends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
16
+ <% end -%>
17
+ <% if !provides.empty? -%>
18
+ <%# Turn each provides from 'foo = 123' to simply 'foo' because Debian :\ -%>
19
+ <%# http://www.debian.org/doc/debian-policy/ch-relationships.html -%>
20
+ Provides: <%= provides.map {|p| p.split(" ").first}.join ", " %>
21
+ <% end -%>
22
+ <% if !replaces.empty? -%>
23
+ Replaces: <%= replaces.join(", ") %>
24
+ <% end -%>
25
+ <% if attributes[:deb_recommends_given?] -%>
26
+ Recommends: <%= attributes[:deb_recommends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
27
+ <% end -%>
28
+ <% if attributes[:deb_suggests_given?] -%>
29
+ Suggests: <%= attributes[:deb_suggests].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
30
+ <% end -%>
31
+ Section: <%= category %>
32
+ Priority: <%= attributes[:deb_priority] %>
33
+ Homepage: <%= url or "http://nourlgiven.example.com/" %>
34
+ Filename: <%= url_filename_encoded %>
35
+ <% if size -%>
36
+ Size: <%= size %>
37
+ <% end -%>
38
+ <% if sha1 -%>
39
+ SHA1: <%= sha1 %>
40
+ <% end -%>
41
+ <% if sha256 -%>
42
+ SHA256: <%= sha256 %>
43
+ <% end -%>
44
+ <% if md5 -%>
45
+ MD5sum: <%= md5 %>
46
+ <% end -%>
47
+ <% lines = (description or "no description given").split("\n") -%>
48
+ <% firstline, *remainder = lines -%>
49
+ Description: <%= firstline %>
50
+ <% if remainder.any? -%>
51
+ <%= remainder.collect { |l| l =~ /^ *$/ ? " ." : " #{l}" }.join("\n") %>
52
+ <% end -%>
53
+ <% if attributes[:deb_field_given?] -%>
54
+ <% attributes[:deb_field].each do |field, value| -%>
55
+ <%= field %>: <%= value %>
56
+ <% end -%>
57
+ <% end -%>
@@ -0,0 +1,16 @@
1
+ Codename: <%= codename %>
2
+ Date: <%= Time.now.utc.strftime("%a, %d %b %Y %T %Z") %>
3
+ Architectures: <%= architectures.join(" ") %>
4
+ Components: <%= components.join(" ") %>
5
+ MD5Sum:
6
+ <% files.each do |f,p| -%>
7
+ <%= p[:md5] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
8
+ <% end -%>
9
+ SHA1:
10
+ <% files.each do |f,p| -%>
11
+ <%= p[:sha1] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
12
+ <% end -%>
13
+ SHA256:
14
+ <% files.each do |f,p| -%>
15
+ <%= p[:sha256] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
16
+ <% end -%>
@@ -0,0 +1,71 @@
1
+ require "erb"
2
+ require "tmpdir"
3
+
4
+ module Deb::S3::Utils
5
+ module_function
6
+ def s3; @s3 end
7
+ def s3= v; @s3 = v end
8
+ def bucket; @bucket end
9
+ def bucket= v; @bucket = v end
10
+ def access_policy; @access_policy end
11
+ def access_policy= v; @access_policy = v end
12
+ def signing_key; @signing_key end
13
+ def signing_key= v; @signing_key = v end
14
+ def gpg_options; @gpg_options end
15
+ def gpg_options= v; @gpg_options = v end
16
+ def prefix; @prefix end
17
+ def prefix= v; @prefix = v end
18
+
19
+ def safesystem(*args)
20
+ success = system(*args)
21
+ if !success
22
+ raise "'system(#{args.inspect})' failed with error code: #{$?.exitstatus}"
23
+ end
24
+ return success
25
+ end
26
+
27
+ def debianize_op(op)
28
+ # Operators in debian packaging are <<, <=, =, >= and >>
29
+ # So any operator like < or > must be replaced
30
+ {:< => "<<", :> => ">>"}[op.to_sym] or op
31
+ end
32
+
33
+ def template(path)
34
+ template_file = File.join(File.dirname(__FILE__), "templates", path)
35
+ template_code = File.read(template_file)
36
+ ERB.new(template_code, nil, "-")
37
+ end
38
+
39
+ def s3_path(path)
40
+ File.join(*[Deb::S3::Utils.prefix, path].compact)
41
+ end
42
+
43
+ # from fog, Fog::AWS.escape
44
+ def s3_escape(string)
45
+ string.gsub(/([^a-zA-Z0-9_.\-~]+)/) {
46
+ "%" + $1.unpack("H2" * $1.bytesize).join("%").upcase
47
+ }
48
+ end
49
+
50
+ def s3_exists?(path)
51
+ Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].exists?
52
+ end
53
+
54
+ def s3_read(path)
55
+ return nil unless s3_exists?(path)
56
+ Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].read
57
+ end
58
+
59
+ def s3_store(path, filename=nil)
60
+ filename = File.basename(path) unless filename
61
+ File.open(path) do |file|
62
+ o = Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(filename)]
63
+ o.write(file)
64
+ o.acl = Deb::S3::Utils.access_policy
65
+ end
66
+ end
67
+
68
+ def s3_remove(path)
69
+ Deb::S3::Utils.s3.buckets[Deb::S3::Utils.bucket].objects[s3_path(path)].delete if s3_exists?(path)
70
+ end
71
+ end
metadata ADDED
@@ -0,0 +1,87 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: pkgr-deb-s3
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.6.0
5
+ prerelease:
6
+ platform: ruby
7
+ authors:
8
+ - Ken Robertson
9
+ autorequire:
10
+ bindir: bin
11
+ cert_chain: []
12
+ date: 2013-10-04 00:00:00.000000000 Z
13
+ dependencies:
14
+ - !ruby/object:Gem::Dependency
15
+ name: thor
16
+ requirement: !ruby/object:Gem::Requirement
17
+ none: false
18
+ requirements:
19
+ - - ~>
20
+ - !ruby/object:Gem::Version
21
+ version: 0.18.0
22
+ type: :runtime
23
+ prerelease: false
24
+ version_requirements: !ruby/object:Gem::Requirement
25
+ none: false
26
+ requirements:
27
+ - - ~>
28
+ - !ruby/object:Gem::Version
29
+ version: 0.18.0
30
+ - !ruby/object:Gem::Dependency
31
+ name: aws-sdk
32
+ requirement: !ruby/object:Gem::Requirement
33
+ none: false
34
+ requirements:
35
+ - - ~>
36
+ - !ruby/object:Gem::Version
37
+ version: '1.18'
38
+ type: :runtime
39
+ prerelease: false
40
+ version_requirements: !ruby/object:Gem::Requirement
41
+ none: false
42
+ requirements:
43
+ - - ~>
44
+ - !ruby/object:Gem::Version
45
+ version: '1.18'
46
+ description: Easily create and manage an APT repository on S3.
47
+ email: ken@invalidlogic.com
48
+ executables:
49
+ - deb-s3
50
+ extensions: []
51
+ extra_rdoc_files: []
52
+ files:
53
+ - bin/deb-s3
54
+ - lib/deb/s3/cli.rb
55
+ - lib/deb/s3/manifest.rb
56
+ - lib/deb/s3/package.rb
57
+ - lib/deb/s3/release.rb
58
+ - lib/deb/s3/templates/package.erb
59
+ - lib/deb/s3/templates/release.erb
60
+ - lib/deb/s3/utils.rb
61
+ - lib/deb/s3.rb
62
+ - README.md
63
+ homepage: http://invalidlogic.com/
64
+ licenses: []
65
+ post_install_message:
66
+ rdoc_options: []
67
+ require_paths:
68
+ - lib
69
+ required_ruby_version: !ruby/object:Gem::Requirement
70
+ none: false
71
+ requirements:
72
+ - - ! '>='
73
+ - !ruby/object:Gem::Version
74
+ version: '0'
75
+ required_rubygems_version: !ruby/object:Gem::Requirement
76
+ none: false
77
+ requirements:
78
+ - - ! '>='
79
+ - !ruby/object:Gem::Version
80
+ version: '0'
81
+ requirements: []
82
+ rubyforge_project:
83
+ rubygems_version: 1.8.23
84
+ signing_key:
85
+ specification_version: 3
86
+ summary: Easily create and manage an APT repository on S3.
87
+ test_files: []