deb-fog 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,15 @@
1
+ ---
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ MWMwZjlmN2FiZWM3Y2FlOThlOWE1ZjczOTk1YmVjOWM4ZjAzY2I0Mg==
5
+ data.tar.gz: !binary |-
6
+ NDNiNDRiOGIwNTYxYTAzNjZhMGNmMDFiZjZiZjU4MzE5OThlYzcxYg==
7
+ SHA512:
8
+ metadata.gz: !binary |-
9
+ ZGQwZWJkMjhiOThlMjk4MjlkYTkxODRhMzcwZmU2OGUwODI5NWU5MWU4YjRh
10
+ MDQyYmI2MGI5NjNlOGY3YjAwMmU2ODYzODEzYzU3NjY0YmQ3ZWE0ZDkxNDFh
11
+ N2E4YjNmOGEwMzIxNjU5YzQ3YjUyMmQxYzMzZTgzODZkOWUzMzk=
12
+ data.tar.gz: !binary |-
13
+ YjMyY2E1MDg3YmY2MWM5YTY1MjM4MDM5NDJmM2NhMmMwMDlmMWViYWZlNDQ0
14
+ NDk2MmU2OWRhYTg4NWQ2MThkOTViM2YyZmQ5N2I1NDQ3MDBmOWM4OGI5ODIx
15
+ NGE4MzlmYTdiMGU3YzEwMjdhYzA3YzUwMTY4NWUxNmQ2MTVhNzc=
@@ -0,0 +1,139 @@
1
+ # deb-fog
2
+
3
+ `deb-fog` is a simple utility to make creating and managing APT repositories on
4
+ Cloud object storage platforms. It is based on a fork of the `deb-fog` repository found [here](https://github.com/krobertson/deb-fog)
5
+
6
+ Most existing existing guides on using object storage to host an APT repository have you
7
+ using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
8
+ the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) or similar to sync the files to object storage.
9
+
10
+ The annoying thing about this process is it requires you to maintain a local
11
+ copy of the file tree for regenerating and syncing the next time. Personally,
12
+ my process is to use one-off virtual machines with
13
+ [Vagrant](http://vagrantup.com), script out the build process, and then would
14
+ prefer to just upload the final `.deb` from my Mac.
15
+
16
+ With `deb-fog`, there is no need for this. `deb-fog` features:
17
+
18
+ * Downloads the existing package manifest and parses it.
19
+ * Updates it with the new package, replacing the existing entry if already
20
+ there or adding a new one if not.
21
+ * Uploads the package itself, the Packages manifest, and the Packages.gz
22
+ manifest.
23
+ * Updates the Release file with the new hashes and file sizes.
24
+
25
+ ## Getting Started
26
+
27
+ You can simply install it from rubygems:
28
+
29
+ ```console
30
+ $ gem install deb-fog
31
+ ```
32
+
33
+ Or to run the code directly, just check out the repo and run Bundler to ensure
34
+ all dependencies are installed:
35
+
36
+ ```console
37
+ $ git clone https://github.com/krobertson/deb-fog.git
38
+ $ cd deb-fog
39
+ $ bundle install
40
+ ```
41
+
42
+ Now to upload a package, simply use:
43
+
44
+ ```console
45
+ $ deb-fog upload --provider Rackspace --bucket my-bucket my-deb-package-1.0.0_amd64.deb
46
+ >> Examining package file my-deb-package-1.0.0_amd64.deb
47
+ >> Retrieving existing package manifest
48
+ >> Uploading package and new manifests to S3
49
+ -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
50
+ -- Transferring dists/stable/main/binary-amd64/Packages
51
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
52
+ -- Transferring dists/stable/Release
53
+ >> Update complete.
54
+ ```
55
+
56
+ ```
57
+ Usage:
58
+ deb-fog upload FILES
59
+
60
+ Options:
61
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
62
+ [--sign=SIGN] # Sign the Release file. Use --sign with your key ID to use a specific key.
63
+ -p, [--preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
64
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
65
+ -c, [--codename=CODENAME] # The codename of the APT repository.
66
+ # Default: stable
67
+ -m, [--component=COMPONENT] # The component of the APT repository.
68
+ # Default: main
69
+ [--access-key-id=ACCESS_KEY] # The access key or username for
70
+ # authenticating with your cloud
71
+ # platform
72
+ [--secret-access-key=SECRET_KEY] # The secret key or API key for
73
+ # authenticating with your cloud
74
+ # platform
75
+ [--provider=CLOUD_PROVIDER] # the cloud to connect to Rackspace|AWS
76
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded
77
+ files. Can be public, private, or authenticated.
78
+ # Default: public
79
+
80
+ Uploads the given files to a S3 bucket as an APT repository.
81
+ ```
82
+
83
+ You can also delete packages from the APT repository. Please keep in mind that
84
+ this does NOT delete the .deb file itself, it only removes it from the list of
85
+ packages in the specified component, codename and architecture.
86
+
87
+ Now to delete the package:
88
+ ```console
89
+ $ deb-fog delete --provider Rackspace --arch amd64 --bucket my-bucket --versions 1.0.0 my-deb-package
90
+ >> Retrieving existing manifests
91
+ -- Deleting my-deb-package version 1.0.0
92
+ >> Uploading new manifests to S3
93
+ -- Transferring dists/stable/main/binary-amd64/Packages
94
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
95
+ -- Transferring dists/stable/Release
96
+ >> Update complete.
97
+
98
+ ````
99
+
100
+ You can also verify an existing APT repository on S3 using the `verify` command:
101
+
102
+ ```console
103
+ deb-fog verify --provider Rackspace -b my-bucket
104
+ >> Retrieving existing manifests
105
+ >> Checking for missing packages in: stable/main i386
106
+ >> Checking for missing packages in: stable/main amd64
107
+ >> Checking for missing packages in: stable/main all
108
+ ```
109
+
110
+ ```
111
+ Usage:
112
+ deb-fog verify
113
+
114
+ Options:
115
+ -f, [--fix-manifests] # Whether to fix problems in manifests when verifying.
116
+ [--sign=SIGN] # Sign the Release file. Use --sign with your key ID to use a specific key.
117
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
118
+ -c, [--codename=CODENAME] # The codename of the APT repository.
119
+ # Default: stable
120
+ -m, [--component=COMPONENT] # The component of the APT repository.
121
+ # Default: main
122
+ [--access-key-id=ACCESS_KEY] # The access key or username for
123
+ # authenticating with your cloud
124
+ # platform
125
+ [--secret-access-key=SECRET_KEY] # The secret key or API key for
126
+ # authenticating with your cloud
127
+ # platform
128
+ [--provider=CLOUD_PROVIDER] # the cloud to connect to Rackspace|AWS
129
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
130
+ # Default: public
131
+
132
+ Verifies that the files in the package manifests exist
133
+ ```
134
+
135
+ ## TODO
136
+
137
+ This is still experimental. These are several things to be done:
138
+
139
+ * Don't re-upload a package if it already exists and has the same hashes.
@@ -0,0 +1,9 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $:.unshift File.expand_path('../../lib', __FILE__)
4
+
5
+ require 'rubygems'
6
+ require 'deb/fog/cli'
7
+
8
+ Deb::Fog::CLI.start
9
+
@@ -0,0 +1,5 @@
1
+ module Deb
2
+ module Fog
3
+ VERSION = "0.1.0"
4
+ end
5
+ end
@@ -0,0 +1,306 @@
1
+ require "fog"
2
+ require "thor"
3
+
4
+ # Hack: aws requires this!
5
+ require "json"
6
+
7
+ require "deb/fog"
8
+ require "deb/fog/utils"
9
+ require "deb/fog/manifest"
10
+ require "deb/fog/package"
11
+ require "deb/fog/release"
12
+
13
+ class Deb::Fog::CLI < Thor
14
+ class_option :bucket,
15
+ :type => :string,
16
+ :aliases => "-b",
17
+ :desc => "The name of the Fog bucket to upload to."
18
+
19
+ class_option :prefix,
20
+ :type => :string,
21
+ :desc => "The path prefix to use when storing on Fog."
22
+
23
+ class_option :codename,
24
+ :default => "stable",
25
+ :type => :string,
26
+ :aliases => "-c",
27
+ :desc => "The codename of the APT repository."
28
+
29
+ class_option :component,
30
+ :default => "main",
31
+ :type => :string,
32
+ :aliases => "-m",
33
+ :desc => "The component of the APT repository."
34
+
35
+ class_option :section,
36
+ :type => :string,
37
+ :aliases => "-s",
38
+ :hide => true
39
+
40
+ class_option :provider,
41
+ :type => :string,
42
+ :desc => "The Cloud Provider to use: AWS|Google|Rackspace"
43
+
44
+ class_option :access_key_id,
45
+ :type => :string,
46
+ :desc => "The access key for connecting to Fog."
47
+
48
+ class_option :secret_access_key,
49
+ :type => :string,
50
+ :desc => "The secret key for connecting to Fog."
51
+
52
+ class_option :endpoint,
53
+ :type => :string,
54
+ :desc => "The region endpoint for connecting to Fog.",
55
+ :default => "fog.amazonaws.com"
56
+
57
+ class_option :visibility,
58
+ :default => "public",
59
+ :type => :string,
60
+ :aliases => "-v",
61
+ :desc => "The access policy for the uploaded files. " +
62
+ "Can be public, private, or authenticated."
63
+
64
+ class_option :sign,
65
+ :type => :string,
66
+ :desc => "Sign the Release file when uploading a package," +
67
+ "or when verifying it after removing a package." +
68
+ "Use --sign with your key ID to use a specific key."
69
+
70
+ class_option :gpg_options,
71
+ :default => "",
72
+ :type => :string,
73
+ :desc => "Additional command line options to pass to GPG when signing"
74
+
75
+ desc "upload FILES",
76
+ "Uploads the given files to a Fog bucket as an APT repository."
77
+
78
+ option :arch,
79
+ :type => :string,
80
+ :aliases => "-a",
81
+ :desc => "The architecture of the package in the APT repository."
82
+
83
+ option :preserve_versions,
84
+ :default => false,
85
+ :type => :boolean,
86
+ :aliases => "-p",
87
+ :desc => "Whether to preserve other versions of a package " +
88
+ "in the repository when uploading one."
89
+
90
+ def upload(*files)
91
+ component = options[:component]
92
+ if options[:section]
93
+ component = options[:section]
94
+ warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
95
+ end
96
+
97
+ if files.nil? || files.empty?
98
+ error("You must specify at least one file to upload")
99
+ end
100
+
101
+ # make sure all the files exists
102
+ if missing_file = files.detect { |f| !File.exists?(f) }
103
+ error("File '#{missing_file}' doesn't exist")
104
+ end
105
+
106
+ # configure AWS::Fog
107
+ configure_fog_client
108
+
109
+ # retrieve the existing manifests
110
+ log("Retrieving existing manifests")
111
+ release = Deb::Fog::Release.retrieve(options[:codename])
112
+ manifests = {}
113
+
114
+ # examine all the files
115
+ files.collect { |f| Dir.glob(f) }.flatten.each do |file|
116
+ log("Examining package file #{File.basename(file)}")
117
+ pkg = Deb::Fog::Package.parse_file(file)
118
+
119
+ # copy over some options if they weren't given
120
+ arch = options[:arch] || pkg.architecture
121
+
122
+ # validate we have them
123
+ error("No architcture given and unable to determine one for #{file}. " +
124
+ "Please specify one with --arch [i386,amd64].") unless arch
125
+
126
+ # retrieve the manifest for the arch if we don't have it already
127
+ manifests[arch] ||= Deb::Fog::Manifest.retrieve(options[:codename], component, arch)
128
+
129
+ # add in the package
130
+ manifests[arch].add(pkg, options[:preserve_versions])
131
+ end
132
+
133
+ # upload the manifest
134
+ log("Uploading packages and new manifests to Fog")
135
+ manifests.each_value do |manifest|
136
+ manifest.write_to_fog { |f| sublog("Transferring #{f}") }
137
+ release.update_manifest(manifest)
138
+ end
139
+ release.write_to_fog { |f| sublog("Transferring #{f}") }
140
+
141
+ log("Update complete.")
142
+ end
143
+
144
+ desc "delete PACKAGE",
145
+ "Remove the package named PACKAGE. If --versions is not specified, delete" +
146
+ "all versions of PACKAGE. Otherwise, only the specified versions will be " +
147
+ "deleted."
148
+
149
+ option :arch,
150
+ :type => :string,
151
+ :aliases => "-a",
152
+ :desc => "The architecture of the package in the APT repository."
153
+
154
+ option :versions,
155
+ :default => nil,
156
+ :type => :array,
157
+ :desc => "The space-delimited versions of PACKAGE to delete. If not" +
158
+ "specified, ALL VERSIONS will be deleted. Fair warning." +
159
+ "E.g. --versions \"0.1 0.2 0.3\""
160
+
161
+ def delete(package)
162
+ component = options[:component]
163
+ if options[:section]
164
+ component = options[:section]
165
+ warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
166
+ end
167
+
168
+ if package.nil?
169
+ error("You must specify a package name.")
170
+ end
171
+
172
+ versions = options[:versions]
173
+ if versions.nil?
174
+ warn("===> WARNING: Deleting all versions of #{package}")
175
+ else
176
+ log("Versions to delete: #{versions.join(', ')}")
177
+ end
178
+
179
+ arch = options[:arch]
180
+ if arch.nil?
181
+ error("You must specify the architecture of the package to remove.")
182
+ end
183
+
184
+ configure_fog_client
185
+
186
+ # retrieve the existing manifests
187
+ log("Retrieving existing manifests")
188
+ release = Deb::Fog::Release.retrieve(options[:codename])
189
+ manifest = Deb::Fog::Manifest.retrieve(options[:codename], component, options[:arch])
190
+
191
+ deleted = manifest.delete_package(package, versions)
192
+ if deleted.length == 0
193
+ if versions.nil?
194
+ error("No packages were deleted. #{package} not found.")
195
+ else
196
+ error("No packages were deleted. #{package} versions #{versions.join(', ')} could not be found.")
197
+ end
198
+ else
199
+ deleted.each { |p|
200
+ sublog("Deleting #{p.name} version #{p.version}")
201
+ }
202
+ end
203
+
204
+ log("Uploading new manifests to Fog")
205
+ manifest.write_to_fog {|f| sublog("Transferring #{f}") }
206
+ release.update_manifest(manifest)
207
+ release.write_to_fog {|f| sublog("Transferring #{f}") }
208
+
209
+ log("Update complete.")
210
+ end
211
+
212
+
213
+ desc "verify", "Verifies that the files in the package manifests exist"
214
+
215
+ option :fix_manifests,
216
+ :default => false,
217
+ :type => :boolean,
218
+ :aliases => "-f",
219
+ :desc => "Whether to fix problems in manifests when verifying."
220
+
221
+ def verify
222
+ component = options[:component]
223
+ if options[:section]
224
+ component = options[:section]
225
+ warn("===> WARNING: The --section/-s argument is deprecated, please use --component/-m.")
226
+ end
227
+
228
+ configure_fog_client
229
+
230
+ log("Retrieving existing manifests")
231
+ release = Deb::Fog::Release.retrieve(options[:codename])
232
+
233
+ %w[amd64 armel i386 all].each do |arch|
234
+ log("Checking for missing packages in: #{options[:codename]}/#{options[:component]} #{arch}")
235
+ manifest = Deb::Fog::Manifest.retrieve(options[:codename], component, arch)
236
+ missing_packages = []
237
+
238
+ manifest.packages.each do |p|
239
+ unless Deb::Fog::Utils.fog_exists? p.url_filename_encoded
240
+ sublog("The following packages are missing:\n\n") if missing_packages.empty?
241
+ puts(p.generate)
242
+ puts("")
243
+
244
+ missing_packages << p
245
+ end
246
+ end
247
+
248
+ if options[:fix_manifests] && !missing_packages.empty?
249
+ log("Removing #{missing_packages.length} package(s) from the manifest...")
250
+ missing_packages.each { |p| manifest.packages.delete(p) }
251
+ manifest.write_to_fog { |f| sublog("Transferring #{f}") }
252
+ release.update_manifest(manifest)
253
+ release.write_to_fog { |f| sublog("Transferring #{f}") }
254
+
255
+ log("Update complete.")
256
+ end
257
+ end
258
+ end
259
+
260
+ private
261
+
262
+ def log(message)
263
+ puts ">> #{message}"
264
+ end
265
+
266
+ def sublog(message)
267
+ puts " -- #{message}"
268
+ end
269
+
270
+ def error(message)
271
+ puts "!! #{message}"
272
+ exit 1
273
+ end
274
+
275
+ def configure_fog_client
276
+ error("No value provided for required options '--bucket'") unless options[:bucket]
277
+ credentials = {:provider => options[:provider]}
278
+ case credentials[:provider]
279
+ when 'AWS'
280
+ credentials[:aws_access_key_id] = options[:access_key_id] if options[:access_key_id]
281
+ credentials[:aws_secret_access_key] = options[:secret_access_key] if options[:secret_access_key]
282
+ when 'Rackspace'
283
+ credentials[:rackspace_username] = options[:access_key_id] if options[:access_key_id]
284
+ credentials[:rackspace_api_key] = options[:secret_access_key] if options[:secret_access_key]
285
+ else
286
+ error("Invalid provider. Can be AWS or Rackspace")
287
+ end
288
+ Deb::Fog::Utils.fog = Fog::Storage.new(credentials)
289
+ Deb::Fog::Utils.bucket = Deb::Fog::Utils.fog.directories.new :key => options[:bucket]
290
+ Deb::Fog::Utils.bucket.reload
291
+ Deb::Fog::Utils.signing_key = options[:sign]
292
+ Deb::Fog::Utils.gpg_options = options[:gpg_options]
293
+ Deb::Fog::Utils.prefix = options[:prefix]
294
+
295
+ # make sure we have a valid visibility setting
296
+ Deb::Fog::Utils.is_public =
297
+ case options[:visibility]
298
+ when "public"
299
+ true
300
+ when "private"
301
+ false
302
+ else
303
+ error("Invalid visibility setting given. Can be public or private")
304
+ end
305
+ end
306
+ end
@@ -0,0 +1,121 @@
1
+ require "tempfile"
2
+ require "zlib"
3
+
4
+ class Deb::Fog::Manifest
5
+ include Deb::Fog::Utils
6
+
7
+ attr_accessor :codename
8
+ attr_accessor :component
9
+ attr_accessor :architecture
10
+
11
+ attr_accessor :files
12
+
13
+ def initialize
14
+ @packages = []
15
+ @component = nil
16
+ @architecture = nil
17
+ @files = {}
18
+ end
19
+
20
+ class << self
21
+ def retrieve(codename, component, architecture)
22
+ m = if s = Deb::Fog::Utils.fog_read("dists/#{codename}/#{component}/binary-#{architecture}/Packages")
23
+ self.parse_packages(s)
24
+ else
25
+ self.new
26
+ end
27
+
28
+ m.codename = codename
29
+ m.component = component
30
+ m.architecture = architecture
31
+ m
32
+ end
33
+
34
+ def parse_packages(str)
35
+ m = self.new
36
+ str.split("\n\n").each do |s|
37
+ next if s.chomp.empty?
38
+ m.packages << Deb::Fog::Package.parse_string(s)
39
+ end
40
+ m
41
+ end
42
+ end
43
+
44
+ def packages
45
+ @packages
46
+ end
47
+
48
+ def add(pkg, preserve_versions)
49
+ if preserve_versions
50
+ @packages.delete_if { |p| p.name == pkg.name && p.version == pkg.version }
51
+ else
52
+ @packages.delete_if { |p| p.name == pkg.name }
53
+ end
54
+ @packages << pkg
55
+ pkg
56
+ end
57
+
58
+ def delete_package(pkg, versions=nil)
59
+ deleted = []
60
+ new_packages = @packages.select { |p|
61
+ # Include packages we didn't name
62
+ if p.name != pkg
63
+ p
64
+ # Also include the packages not matching a specified version
65
+ elsif (!versions.nil? and p.name == pkg and !versions.include? p.version)
66
+ p
67
+ end
68
+ }
69
+ deleted = @packages - new_packages
70
+ @packages = new_packages
71
+ deleted
72
+ end
73
+
74
+ def generate
75
+ @packages.collect { |pkg| pkg.generate }.join("\n")
76
+ end
77
+
78
+ def write_to_fog
79
+ manifest = self.generate
80
+
81
+ # store any packages that need to be stored
82
+ @packages.each do |pkg|
83
+ if pkg.needs_uploading?
84
+ yield pkg.url_filename if block_given?
85
+ fog_store(pkg.filename, pkg.url_filename, 'application/octet-stream; charset=binary')
86
+ end
87
+ end
88
+
89
+ # generate the Packages file
90
+ pkgs_temp = Tempfile.new("Packages")
91
+ pkgs_temp.write manifest
92
+ puts "package contents: #{manifest}!!!!! to dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages"
93
+ pkgs_temp.close
94
+ f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages"
95
+ yield f if block_given?
96
+ fog_store(pkgs_temp.path, f, 'text/plain; charset=us-ascii')
97
+ @files["#{@component}/binary-#{@architecture}/Packages"] = hashfile(pkgs_temp.path)
98
+ pkgs_temp.unlink
99
+
100
+ # generate the Packages.gz file
101
+ gztemp = Tempfile.new("Packages.gz")
102
+ gztemp.close
103
+ Zlib::GzipWriter.open(gztemp.path) { |gz| gz.write manifest }
104
+ f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages.gz"
105
+ yield f if block_given?
106
+ fog_store(gztemp.path, f, 'application/x-gzip; charset=binary')
107
+ @files["#{@component}/binary-#{@architecture}/Packages.gz"] = hashfile(gztemp.path)
108
+ gztemp.unlink
109
+
110
+ nil
111
+ end
112
+
113
+ def hashfile(path)
114
+ {
115
+ :size => File.size(path),
116
+ :sha1 => Digest::SHA1.file(path).hexdigest,
117
+ :sha256 => Digest::SHA2.file(path).hexdigest,
118
+ :md5 => Digest::MD5.file(path).hexdigest
119
+ }
120
+ end
121
+ end
@@ -0,0 +1,286 @@
1
+ require "digest/sha1"
2
+ require "digest/sha2"
3
+ require "digest/md5"
4
+ require "socket"
5
+ require "tmpdir"
6
+
7
+ class Deb::Fog::Package
8
+ include Deb::Fog::Utils
9
+
10
+ attr_accessor :name
11
+ attr_accessor :version
12
+ attr_accessor :epoch
13
+ attr_accessor :iteration
14
+ attr_accessor :maintainer
15
+ attr_accessor :vendor
16
+ attr_accessor :url
17
+ attr_accessor :category
18
+ attr_accessor :license
19
+ attr_accessor :architecture
20
+ attr_accessor :description
21
+
22
+ attr_accessor :dependencies
23
+ attr_accessor :provides
24
+ attr_accessor :conflicts
25
+ attr_accessor :replaces
26
+ attr_accessor :excludes
27
+
28
+
29
+ # Any other attributes specific to this package.
30
+ # This is where you'd put rpm, deb, or other specific attributes.
31
+ attr_accessor :attributes
32
+
33
+ # hashes
34
+ attr_accessor :url_filename
35
+ attr_accessor :sha1
36
+ attr_accessor :sha256
37
+ attr_accessor :md5
38
+ attr_accessor :size
39
+
40
+ attr_accessor :filename
41
+
42
+ class << self
43
+ include Deb::Fog::Utils
44
+
45
+ def parse_file(package)
46
+ p = self.new
47
+ p.extract_info(extract_control(package))
48
+ p.apply_file_info(package)
49
+ p.filename = package
50
+ p
51
+ end
52
+
53
+ def parse_string(s)
54
+ p = self.new
55
+ p.extract_info(s)
56
+ p
57
+ end
58
+
59
+ def extract_control(package)
60
+ if system("which dpkg &> /dev/null")
61
+ `dpkg -f #{package}`
62
+ else
63
+ # ar fails to find the control.tar.gz tarball within the .deb
64
+ # on Mac OS. Try using ar to list the control file, if found,
65
+ # use ar to extract, otherwise attempt with tar which works on OS X.
66
+ extract_control_tarball_cmd = "ar p #{package} control.tar.gz"
67
+
68
+ begin
69
+ safesystem("ar t #{package} control.tar.gz &> /dev/null")
70
+ rescue SafeSystemError
71
+ warn "Failed to find control data in .deb with ar, trying tar."
72
+ extract_control_tarball_cmd = "tar zxf #{package} --to-stdout control.tar.gz"
73
+ end
74
+
75
+ Dir.mktmpdir do |path|
76
+ safesystem("#{extract_control_tarball_cmd} | tar -zxf - -C #{path}")
77
+ File.read(File.join(path, "control"))
78
+ end
79
+ end
80
+ end
81
+ end
82
+
83
+ def initialize
84
+ @attributes = {}
85
+
86
+ # Reference
87
+ # http://www.debian.org/doc/manuals/maint-guide/first.en.html
88
+ # http://wiki.debian.org/DeveloperConfiguration
89
+ # https://github.com/jordansissel/fpm/issues/37
90
+ if ENV.include?("DEBEMAIL") and ENV.include?("DEBFULLNAME")
91
+ # Use DEBEMAIL and DEBFULLNAME as the default maintainer if available.
92
+ @maintainer = "#{ENV["DEBFULLNAME"]} <#{ENV["DEBEMAIL"]}>"
93
+ else
94
+ # TODO(sissel): Maybe support using 'git config' for a default as well?
95
+ # git config --get user.name, etc can be useful.
96
+ #
97
+ # Otherwise default to user@currenthost
98
+ @maintainer = "<#{ENV["USER"]}@#{Socket.gethostname}>"
99
+ end
100
+
101
+ @name = nil
102
+ @architecture = "native"
103
+ @description = "no description given"
104
+ @version = nil
105
+ @epoch = nil
106
+ @iteration = nil
107
+ @url = nil
108
+ @category = "default"
109
+ @license = "unknown"
110
+ @vendor = "none"
111
+ @sha1 = nil
112
+ @sha256 = nil
113
+ @md5 = nil
114
+ @size = nil
115
+ @filename = nil
116
+ @url_filename = nil
117
+
118
+ @provides = []
119
+ @conflicts = []
120
+ @replaces = []
121
+ @dependencies = []
122
+
123
+ @needs_uploading = false
124
+ end
125
+
126
+ def filename=(f)
127
+ @filename = f
128
+ @needs_uploading = true
129
+ @filename
130
+ end
131
+
132
+ def url_filename
133
+ @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{File.basename(self.filename)}"
134
+ end
135
+
136
+ def url_filename_encoded
137
+ @url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{fog_escape(File.basename(self.filename))}"
138
+ end
139
+
140
+ def needs_uploading?
141
+ @needs_uploading
142
+ end
143
+
144
+ def generate
145
+ template("package.erb").result(binding)
146
+ end
147
+
148
+ # from fpm
149
+ def parse_depends(data)
150
+ return [] if data.nil? or data.empty?
151
+ # parse dependencies. Debian dependencies come in one of two forms:
152
+ # * name
153
+ # * name (op version)
154
+ # They are all on one line, separated by ", "
155
+
156
+ dep_re = /^([^ ]+)(?: \(([>=<]+) ([^)]+)\))?$/
157
+ return data.split(/, */).collect do |dep|
158
+ m = dep_re.match(dep)
159
+ if m
160
+ name, op, version = m.captures
161
+ # this is the proper form of dependency
162
+ if op && version && op != "" && version != ""
163
+ "#{name} (#{op} #{version})".strip
164
+ else
165
+ name.strip
166
+ end
167
+ else
168
+ # Assume normal form dependency, "name op version".
169
+ dep
170
+ end
171
+ end
172
+ end # def parse_depends
173
+
174
+ # from fpm
175
+ def fix_dependency(dep)
176
+ # Deb dependencies are: NAME (OP VERSION), like "zsh (> 3.0)"
177
+ # Convert anything that looks like 'NAME OP VERSION' to this format.
178
+ if dep =~ /[\(,\|]/
179
+ # Don't "fix" ones that could appear well formed already.
180
+ else
181
+ # Convert ones that appear to be 'name op version'
182
+ name, op, version = dep.split(/ +/)
183
+ if !version.nil?
184
+ # Convert strings 'foo >= bar' to 'foo (>= bar)'
185
+ dep = "#{name} (#{debianize_op(op)} #{version})"
186
+ end
187
+ end
188
+
189
+ name_re = /^[^ \(]+/
190
+ name = dep[name_re]
191
+ if name =~ /[A-Z]/
192
+ dep = dep.gsub(name_re) { |n| n.downcase }
193
+ end
194
+
195
+ if dep.include?("_")
196
+ dep = dep.gsub("_", "-")
197
+ end
198
+
199
+ # Convert gem ~> X.Y.Z to '>= X.Y.Z' and << X.Y+1.0
200
+ if dep =~ /\(~>/
201
+ name, version = dep.gsub(/[()~>]/, "").split(/ +/)[0..1]
202
+ nextversion = version.split(".").collect { |v| v.to_i }
203
+ l = nextversion.length
204
+ nextversion[l-2] += 1
205
+ nextversion[l-1] = 0
206
+ nextversion = nextversion.join(".")
207
+ return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
208
+ elsif (m = dep.match(/(\S+)\s+\(!= (.+)\)/))
209
+ # Append this to conflicts
210
+ self.conflicts += [dep.gsub(/!=/,"=")]
211
+ return []
212
+ elsif (m = dep.match(/(\S+)\s+\(= (.+)\)/)) and
213
+ self.attributes[:deb_ignore_iteration_in_dependencies?]
214
+ # Convert 'foo (= x)' to 'foo (>= x)' and 'foo (<< x+1)'
215
+ # but only when flag --ignore-iteration-in-dependencies is passed.
216
+ name, version = m[1..2]
217
+ nextversion = version.split('.').collect { |v| v.to_i }
218
+ nextversion[-1] += 1
219
+ nextversion = nextversion.join(".")
220
+ return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
221
+ else
222
+ # otherwise the dep is probably fine
223
+ return dep.rstrip
224
+ end
225
+ end # def fix_dependency
226
+
227
+ # from fpm
228
+ def extract_info(control)
229
+ parse = lambda do |field|
230
+ value = control[/^#{field}: .*/]
231
+ if value.nil?
232
+ return nil
233
+ else
234
+ return value.split(": ",2).last
235
+ end
236
+ end
237
+
238
+ # Parse 'epoch:version-iteration' in the version string
239
+ version_re = /^(?:([0-9]+):)?(.+?)(?:-(.*))?$/
240
+ m = version_re.match(parse.call("Version"))
241
+ if !m
242
+ raise "Unsupported version string '#{parse.call("Version")}'"
243
+ end
244
+ self.epoch, self.version, self.iteration = m.captures
245
+
246
+ self.architecture = parse.call("Architecture")
247
+ self.category = parse.call("Section")
248
+ self.license = parse.call("License") || self.license
249
+ self.maintainer = parse.call("Maintainer")
250
+ self.name = parse.call("Package")
251
+ self.url = parse.call("Homepage")
252
+ self.vendor = parse.call("Vendor") || self.vendor
253
+ self.attributes[:deb_priority] = parse.call("Priority")
254
+ self.attributes[:deb_origin] = parse.call("Origin")
255
+ self.attributes[:deb_installed_size] = parse.call("Installed-Size")
256
+
257
+ # Packages manifest fields
258
+ self.url_filename = parse.call("Filename")
259
+ self.sha1 = parse.call("SHA1")
260
+ self.sha256 = parse.call("SHA256")
261
+ self.md5 = parse.call("MD5sum")
262
+ self.size = parse.call("Size")
263
+
264
+ # The description field is a special flower, parse it that way.
265
+ # The description is the first line as a normal Description field, but also continues
266
+ # on future lines indented by one space, until the end of the file. Blank
267
+ # lines are marked as ' .'
268
+ description = control[/^Description: .*[^\Z]/m]
269
+ description = description.gsub(/^[^(Description|\s)].*$/, "").split(": ", 2).last
270
+ self.description = description.gsub(/^ /, "").gsub(/^\.$/, "")
271
+
272
+ #self.config_files = config_files
273
+
274
+ self.dependencies += Array(parse_depends(parse.call("Depends")))
275
+ self.conflicts += Array(parse_depends(parse.call("Conflicts")))
276
+ self.provides += Array(parse_depends(parse.call("Provides")))
277
+ self.replaces += Array(parse_depends(parse.call("Replaces")))
278
+ end # def extract_info
279
+
280
+ def apply_file_info(file)
281
+ self.size = File.size(file)
282
+ self.sha1 = Digest::SHA1.file(file).hexdigest
283
+ self.sha256 = Digest::SHA2.file(file).hexdigest
284
+ self.md5 = Digest::MD5.file(file).hexdigest
285
+ end
286
+ end
@@ -0,0 +1,139 @@
1
+ require "tempfile"
2
+
3
+ class Deb::Fog::Release
4
+ include Deb::Fog::Utils
5
+
6
+ attr_accessor :codename
7
+ attr_accessor :architectures
8
+ attr_accessor :components
9
+
10
+ attr_accessor :files
11
+ attr_accessor :policy
12
+
13
+ def initialize
14
+ @codename = nil
15
+ @architectures = []
16
+ @components = []
17
+ @files = {}
18
+ @policy = :public_read
19
+ end
20
+
21
+ class << self
22
+ def retrieve(codename)
23
+ if s = Deb::Fog::Utils.fog_read("dists/#{codename}/Release")
24
+ self.parse_release(s)
25
+ else
26
+ rel = self.new
27
+ rel.codename = codename
28
+ rel
29
+ end
30
+ end
31
+
32
+ def parse_release(str)
33
+ rel = self.new
34
+ rel.parse(str)
35
+ rel
36
+ end
37
+ end
38
+
39
+ def filename
40
+ "dists/#{@codename}/Release"
41
+ end
42
+
43
+ def parse(str)
44
+ parse = lambda do |field|
45
+ value = str[/^#{field}: .*/]
46
+ if value.nil?
47
+ return nil
48
+ else
49
+ return value.split(": ",2).last
50
+ end
51
+ end
52
+
53
+ # grab basic fields
54
+ self.codename = parse.call("Codename")
55
+ self.architectures = (parse.call("Architectures") || "").split(/\s+/)
56
+ self.components = (parse.call("Components") || "").split(/\s+/)
57
+
58
+ # find all the hashes
59
+ str.scan(/^\s+([^\s]+)\s+(\d+)\s+(.+)$/).each do |(hash,size,name)|
60
+ self.files[name] ||= { :size => size.to_i }
61
+ case hash.length
62
+ when 32
63
+ self.files[name][:md5] = hash
64
+ when 40
65
+ self.files[name][:sha1] = hash
66
+ when 64
67
+ self.files[name][:sha256] = hash
68
+ end
69
+ end
70
+ end
71
+
72
+ def generate
73
+ template("release.erb").result(binding)
74
+ end
75
+
76
+ def write_to_fog
77
+ # validate some other files are present
78
+ if block_given?
79
+ self.validate_others { |f| yield f }
80
+ else
81
+ self.validate_others
82
+ end
83
+
84
+ # generate the Release file
85
+ release_tmp = Tempfile.new("Release")
86
+ release_tmp.puts self.generate
87
+ release_tmp.close
88
+ yield self.filename if block_given?
89
+ fog_store(release_tmp.path, self.filename, 'text/plain; charset=us-ascii')
90
+
91
+ # sign the file, if necessary
92
+ if Deb::Fog::Utils.signing_key
93
+ key_param = Deb::Fog::Utils.signing_key != "" ? "--default-key=#{Deb::Fog::Utils.signing_key}" : ""
94
+ if system("gpg -a #{key_param} #{Deb::Fog::Utils.gpg_options} -b #{release_tmp.path}")
95
+ local_file = release_tmp.path+".asc"
96
+ remote_file = self.filename+".gpg"
97
+ yield remote_file if block_given?
98
+ raise "Unable to locate Release signature file" unless File.exists?(local_file)
99
+ fog_store(local_file, remote_file, 'application/pgp-signature; charset=us-ascii')
100
+ File.unlink(local_file)
101
+ else
102
+ raise "Signing the Release file failed."
103
+ end
104
+ else
105
+ # remove an existing Release.gpg, if it was there
106
+ fog_remove(self.filename+".gpg")
107
+ end
108
+
109
+ release_tmp.unlink
110
+ end
111
+
112
+ def update_manifest(manifest)
113
+ self.components << manifest.component unless self.components.include?(manifest.component)
114
+ self.architectures << manifest.architecture unless self.architectures.include?(manifest.architecture)
115
+ self.files.merge!(manifest.files)
116
+ end
117
+
118
+ def validate_others
119
+ to_apply = []
120
+ self.components.each do |comp|
121
+ %w(amd64 i386).each do |arch|
122
+ next if self.files.has_key?("#{comp}/binary-#{arch}/Packages")
123
+
124
+ m = Deb::Fog::Manifest.new
125
+ m.codename = self.codename
126
+ m.component = comp
127
+ m.architecture = arch
128
+ if block_given?
129
+ m.write_to_fog { |f| yield f }
130
+ else
131
+ m.write_to_fog
132
+ end
133
+ to_apply << m
134
+ end
135
+ end
136
+
137
+ to_apply.each { |m| self.update_manifest(m) }
138
+ end
139
+ end
@@ -0,0 +1,60 @@
1
+ Package: <%= name %>
2
+ Version: <%= "#{epoch}:" if epoch %><%= version %><%= "-" + iteration.to_s if iteration %>
3
+ License: <%= license %>
4
+ Vendor: <%= vendor %>
5
+ Architecture: <%= architecture %>
6
+ Maintainer: <%= maintainer %>
7
+ Installed-Size: <%= attributes[:deb_installed_size] %>
8
+ <% if !dependencies.empty? and !attributes[:no_depends?] -%>
9
+ Depends: <%= dependencies.collect { |d| fix_dependency(d) }.flatten.join(", ") %>
10
+ <% end -%>
11
+ <% if !conflicts.empty? -%>
12
+ Conflicts: <%= conflicts.join(", ") %>
13
+ <% end -%>
14
+ <% if attributes[:deb_pre_depends] -%>
15
+ Pre-Depends: <%= attributes[:deb_pre_depends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
16
+ <% end -%>
17
+ <% if !provides.empty? -%>
18
+ <%# Turn each provides from 'foo = 123' to simply 'foo' because Debian :\ -%>
19
+ <%# http://www.debian.org/doc/debian-policy/ch-relationships.html -%>
20
+ Provides: <%= provides.map {|p| p.split(" ").first}.join ", " %>
21
+ <% end -%>
22
+ <% if !replaces.empty? -%>
23
+ Replaces: <%= replaces.join(", ") %>
24
+ <% end -%>
25
+ <% if attributes[:deb_recommends] -%>
26
+ Recommends: <%= attributes[:deb_recommends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
27
+ <% end -%>
28
+ <% if attributes[:deb_suggests] -%>
29
+ Suggests: <%= attributes[:deb_suggests].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
30
+ <% end -%>
31
+ Section: <%= category %>
32
+ <% if attributes[:deb_origin] -%>
33
+ Origin: <%= attributes[:deb_origin] %>
34
+ <% end -%>
35
+ Priority: <%= attributes[:deb_priority] %>
36
+ Homepage: <%= url or "http://nourlgiven.example.com/" %>
37
+ Filename: <%= url_filename_encoded %>
38
+ <% if size -%>
39
+ Size: <%= size %>
40
+ <% end -%>
41
+ <% if sha1 -%>
42
+ SHA1: <%= sha1 %>
43
+ <% end -%>
44
+ <% if sha256 -%>
45
+ SHA256: <%= sha256 %>
46
+ <% end -%>
47
+ <% if md5 -%>
48
+ MD5sum: <%= md5 %>
49
+ <% end -%>
50
+ <% lines = (description or "no description given").split("\n") -%>
51
+ <% firstline, *remainder = lines -%>
52
+ Description: <%= firstline %>
53
+ <% if remainder.any? -%>
54
+ <%= remainder.collect { |l| l =~ /^ *$/ ? " ." : " #{l}" }.join("\n") %>
55
+ <% end -%>
56
+ <% if attributes[:deb_field] -%>
57
+ <% attributes[:deb_field].each do |field, value| -%>
58
+ <%= field %>: <%= value %>
59
+ <% end -%>
60
+ <% end -%>
@@ -0,0 +1,16 @@
1
+ Codename: <%= codename %>
2
+ Date: <%= Time.now.utc.strftime("%a, %d %b %Y %T %Z") %>
3
+ Architectures: <%= architectures.join(" ") %>
4
+ Components: <%= components.join(" ") %>
5
+ MD5Sum:
6
+ <% files.each do |f,p| -%>
7
+ <%= p[:md5] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
8
+ <% end -%>
9
+ SHA1:
10
+ <% files.each do |f,p| -%>
11
+ <%= p[:sha1] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
12
+ <% end -%>
13
+ SHA256:
14
+ <% files.each do |f,p| -%>
15
+ <%= p[:sha256] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
16
+ <% end -%>
@@ -0,0 +1,88 @@
1
+ require "base64"
2
+ require "digest/md5"
3
+ require "erb"
4
+ require "tmpdir"
5
+
6
+ module Deb::Fog::Utils
7
+ module_function
8
+ def fog; @fog end
9
+ def fog= v; @fog = v end
10
+ def bucket; @bucket end
11
+ def bucket= v; @bucket = v end
12
+ def is_public; @is_public end
13
+ def is_public= v; @is_public = v end
14
+ def signing_key; @signing_key end
15
+ def signing_key= v; @signing_key = v end
16
+ def gpg_options; @gpg_options end
17
+ def gpg_options= v; @gpg_options = v end
18
+ def prefix; @prefix end
19
+ def prefix= v; @prefix = v end
20
+
21
+ class SafeSystemError < RuntimeError; end
22
+
23
+ def safesystem(*args)
24
+ success = system(*args)
25
+ if !success
26
+ raise SafeSystemError, "'system(#{args.inspect})' failed with error code: #{$?.exitstatus}"
27
+ end
28
+ return success
29
+ end
30
+
31
+ def debianize_op(op)
32
+ # Operators in debian packaging are <<, <=, =, >= and >>
33
+ # So any operator like < or > must be replaced
34
+ {:< => "<<", :> => ">>"}[op.to_sym] or op
35
+ end
36
+
37
+ def template(path)
38
+ template_file = File.join(File.dirname(__FILE__), "templates", path)
39
+ template_code = File.read(template_file)
40
+ ERB.new(template_code, nil, "-")
41
+ end
42
+
43
+ def fog_path(path)
44
+ File.join(*[Deb::Fog::Utils.prefix, path].compact)
45
+ end
46
+
47
+ # from fog, Fog::AWS.escape
48
+ def fog_escape(string)
49
+ string.gsub(/([^a-zA-Z0-9_.\-~]+)/) {
50
+ "%" + $1.unpack("H2" * $1.bytesize).join("%").upcase
51
+ }
52
+ end
53
+
54
+ def fog_exists?(path)
55
+ return true if Deb::Fog::Utils.bucket.files.head(File.basename(path))
56
+ return false
57
+ end
58
+
59
+ def fog_read(path)
60
+ #puts "blerg: #{Deb::Fog::Utils.bucket.files}"
61
+ return nil unless fog_exists?(path)
62
+ Deb::Fog::Utils.bucket.files[fog_path(path)].read
63
+ end
64
+
65
+ def fog_store(path, filename=nil, content_type='application/octet-stream; charset=binary')
66
+ filename = File.basename(path) unless filename
67
+ obj = Deb::Fog::Utils.bucket.files.head(filename)
68
+ # check if the object already exists
69
+ unless obj.nil?
70
+ file_md5 = Digest::MD5.file(path)
71
+ # puts "#{filename} - #{file_md5} vs #{obj.etag.gsub('"', '')}"
72
+ return if file_md5.to_s == obj.etag.gsub('"', '')
73
+ end
74
+
75
+ # upload the file
76
+ file = Deb::Fog::Utils.bucket.files.create(
77
+ :key => fog_path(filename),
78
+ :body => File.open(path),
79
+ :public => Deb::Fog::Utils.is_public,
80
+ :content_type => content_type
81
+ )
82
+ # obj.write(Pathname.new(path), :acl => Deb::Fog::Utils.access_policy, :content_type => content_type)
83
+ end
84
+
85
+ def fog_remove(path)
86
+ Deb::Fog::Utils.bucket.files[fog_path(path)].destroy if fog_exists?(path)
87
+ end
88
+ end
metadata ADDED
@@ -0,0 +1,81 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: deb-fog
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Paul Czarkowski
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2014-04-09 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: thor
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ~>
18
+ - !ruby/object:Gem::Version
19
+ version: 0.18.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ~>
25
+ - !ruby/object:Gem::Version
26
+ version: 0.18.0
27
+ - !ruby/object:Gem::Dependency
28
+ name: fog
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ~>
32
+ - !ruby/object:Gem::Version
33
+ version: '1.21'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ~>
39
+ - !ruby/object:Gem::Version
40
+ version: '1.21'
41
+ description: Easily create and manage an APT repository with Fog.
42
+ email: paul.czarkowski@rackspace.com
43
+ executables:
44
+ - deb-fog
45
+ extensions: []
46
+ extra_rdoc_files: []
47
+ files:
48
+ - bin/deb-fog
49
+ - lib/deb/fog/cli.rb
50
+ - lib/deb/fog/manifest.rb
51
+ - lib/deb/fog/package.rb
52
+ - lib/deb/fog/release.rb
53
+ - lib/deb/fog/templates/package.erb
54
+ - lib/deb/fog/templates/release.erb
55
+ - lib/deb/fog/utils.rb
56
+ - lib/deb/fog.rb
57
+ - README.md
58
+ homepage: http://rackspace.com
59
+ licenses: []
60
+ metadata: {}
61
+ post_install_message:
62
+ rdoc_options: []
63
+ require_paths:
64
+ - lib
65
+ required_ruby_version: !ruby/object:Gem::Requirement
66
+ requirements:
67
+ - - ! '>='
68
+ - !ruby/object:Gem::Version
69
+ version: '0'
70
+ required_rubygems_version: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ! '>='
73
+ - !ruby/object:Gem::Version
74
+ version: '0'
75
+ requirements: []
76
+ rubyforge_project:
77
+ rubygems_version: 2.1.10
78
+ signing_key:
79
+ specification_version: 4
80
+ summary: Easily create and manage an APT repository with Fog.
81
+ test_files: []