deb-s3 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +73 -0
- data/bin/deb-s3 +9 -0
- data/lib/deb/s3.rb +5 -0
- data/lib/deb/s3/cli.rb +136 -0
- data/lib/deb/s3/manifest.rb +100 -0
- data/lib/deb/s3/package.rb +272 -0
- data/lib/deb/s3/release.rb +139 -0
- data/lib/deb/s3/templates/package.erb +57 -0
- data/lib/deb/s3/templates/release.erb +16 -0
- data/lib/deb/s3/utils.rb +64 -0
- metadata +103 -0
data/README.md
ADDED
@@ -0,0 +1,73 @@
|
|
1
|
+
# deb-s3
|
2
|
+
|
3
|
+
`deb-s3` is a simple utility to make creating and managing APT repositories on
|
4
|
+
S3.
|
5
|
+
|
6
|
+
Most existing existing guides on using S3 to host an APT repository have you
|
7
|
+
using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
|
8
|
+
the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
|
9
|
+
sync the files to S3.
|
10
|
+
|
11
|
+
The annoying thing about this process is it requires you to maintain a local
|
12
|
+
copy of the file tree for regenerating and syncing the next time. Personally,
|
13
|
+
my process is to use one-off virtual machines with [Vagrant](http://vagrantup.com),
|
14
|
+
script out the build process, and then would prefer to just upload the final
|
15
|
+
`.deb` from my Mac.
|
16
|
+
|
17
|
+
With `deb-s3`, there is no need for this. `deb-s3` features:
|
18
|
+
|
19
|
+
* Downloads the existing package manifest and parses it.
|
20
|
+
* Updates it with the new package, replacing the existing entry if already there.
|
21
|
+
* Uploads the package itself, the Packages manifest, and the Packages.gz manifest.
|
22
|
+
* Updates the Release file with the new hashes and file sizes.
|
23
|
+
|
24
|
+
## Getting Started
|
25
|
+
|
26
|
+
Simply run Bundler to ensure all dependencies are installed:
|
27
|
+
|
28
|
+
```console
|
29
|
+
$ bundle install
|
30
|
+
```
|
31
|
+
|
32
|
+
Now to upload a package, simply use:
|
33
|
+
|
34
|
+
```console
|
35
|
+
$ deb-s3 upload my-deb-package-1.0.0_amd64.deb --bucket my-bucket
|
36
|
+
>> Examining package file my-deb-package-1.0.0_amd64.deb
|
37
|
+
>> Retrieving existing package manifest
|
38
|
+
>> Uploading package and new manifests to S3
|
39
|
+
-- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
|
40
|
+
-- Transferring dists/stable/main/binary-amd64/Packages
|
41
|
+
-- Transferring dists/stable/main/binary-amd64/Packages.gz
|
42
|
+
-- Transferring dists/stable/Release
|
43
|
+
>> Update complete.
|
44
|
+
```
|
45
|
+
|
46
|
+
```
|
47
|
+
Usage:
|
48
|
+
deb-s3 upload FILE -b, --bucket=BUCKET
|
49
|
+
|
50
|
+
Options:
|
51
|
+
-b, --bucket=BUCKET # The name of the S3 bucket to upload to.
|
52
|
+
-c, [--codename=CODENAME] # The codename of the APT repository.
|
53
|
+
# Default: stable
|
54
|
+
-s, [--section=SECTION] # The section of the APT repository.
|
55
|
+
# Default: main
|
56
|
+
-a, [--arch=ARCH] # The architecture of the package in the APT repository.
|
57
|
+
-v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
|
58
|
+
# Default: public
|
59
|
+
[--access-key=ACCESS_KEY] # The access key for connecting to S3.
|
60
|
+
# Default: $AMAZON_ACCESS_KEY_ID
|
61
|
+
[--secret-key=SECRET_KEY] # The secret key for connecting to S3.
|
62
|
+
# Default: $AMAZON_SECRET_ACCESS_KEY
|
63
|
+
[--sign=SIGN] # Sign the Release file. Use --sign with your key ID to use a specific key.
|
64
|
+
|
65
|
+
Uploads the given FILE to a S3 bucket as an APT repository.
|
66
|
+
```
|
67
|
+
|
68
|
+
## TODO
|
69
|
+
|
70
|
+
This is still experimental. These are several things to be done:
|
71
|
+
|
72
|
+
* Don't re-upload a package if it already exists and has the same hashes.
|
73
|
+
* Clean up the code some more.
|
data/bin/deb-s3
ADDED
data/lib/deb/s3.rb
ADDED
data/lib/deb/s3/cli.rb
ADDED
@@ -0,0 +1,136 @@
|
|
1
|
+
require 'aws/s3'
|
2
|
+
require "thor"
|
3
|
+
|
4
|
+
require "deb/s3"
|
5
|
+
require "deb/s3/utils"
|
6
|
+
require "deb/s3/manifest"
|
7
|
+
require "deb/s3/package"
|
8
|
+
require "deb/s3/release"
|
9
|
+
|
10
|
+
class Deb::S3::CLI < Thor
|
11
|
+
|
12
|
+
option :bucket,
|
13
|
+
:required => true,
|
14
|
+
:type => :string,
|
15
|
+
:aliases => "-b",
|
16
|
+
:desc => "The name of the S3 bucket to upload to."
|
17
|
+
|
18
|
+
option :codename,
|
19
|
+
:default => "stable",
|
20
|
+
:type => :string,
|
21
|
+
:aliases => "-c",
|
22
|
+
:desc => "The codename of the APT repository."
|
23
|
+
|
24
|
+
option :section,
|
25
|
+
:default => "main",
|
26
|
+
:type => :string,
|
27
|
+
:aliases => "-s",
|
28
|
+
:desc => "The section of the APT repository."
|
29
|
+
|
30
|
+
option :arch,
|
31
|
+
:type => :string,
|
32
|
+
:aliases => "-a",
|
33
|
+
:desc => "The architecture of the package in the APT repository."
|
34
|
+
|
35
|
+
option :visibility,
|
36
|
+
:default => "public",
|
37
|
+
:type => :string,
|
38
|
+
:aliases => "-v",
|
39
|
+
:desc => "The access policy for the uploaded files. " +
|
40
|
+
"Can be public, private, or authenticated."
|
41
|
+
|
42
|
+
option :access_key,
|
43
|
+
:default => "$AMAZON_ACCESS_KEY_ID",
|
44
|
+
:type => :string,
|
45
|
+
:desc => "The access key for connecting to S3."
|
46
|
+
|
47
|
+
option :secret_key,
|
48
|
+
:default => "$AMAZON_SECRET_ACCESS_KEY",
|
49
|
+
:type => :string,
|
50
|
+
:desc => "The secret key for connecting to S3."
|
51
|
+
|
52
|
+
option :sign,
|
53
|
+
:default => "",
|
54
|
+
:type => :string,
|
55
|
+
:desc => "Sign the Release file. Use --sign with your key ID to use " +
|
56
|
+
"a specific key."
|
57
|
+
|
58
|
+
desc "upload FILE",
|
59
|
+
"Uploads the given FILE to a S3 bucket as an APT repository."
|
60
|
+
def upload(file)
|
61
|
+
# make sure the file exists
|
62
|
+
error("File doesn't exist") unless File.exists?(file)
|
63
|
+
Deb::S3::Utils.signing_key = options[:sign]
|
64
|
+
|
65
|
+
# make sure we have a valid visibility setting
|
66
|
+
Deb::S3::Utils.access_policy = case options[:visibility]
|
67
|
+
when "public"
|
68
|
+
:public_read
|
69
|
+
when "private"
|
70
|
+
:private
|
71
|
+
when "authenticated"
|
72
|
+
:authenticated_read
|
73
|
+
else
|
74
|
+
error("Invalid visibility setting given. Can be public, private, or authenticated.")
|
75
|
+
end
|
76
|
+
Deb::S3::Utils.bucket = options[:bucket]
|
77
|
+
|
78
|
+
log("Examining package file #{File.basename(file)}")
|
79
|
+
pkg = Deb::S3::Package.parse_file(file)
|
80
|
+
|
81
|
+
# copy over some options if they weren't given
|
82
|
+
arch = options[:arch] || pkg.architecture
|
83
|
+
|
84
|
+
# validate we have them
|
85
|
+
error("No architcture given and unable to determine one from the file. " +
|
86
|
+
"Please specify one with --arch [i386,amd64].") unless arch
|
87
|
+
|
88
|
+
# configure AWS::S3
|
89
|
+
access_key = if options[:access_key] == "$AMAZON_ACCESS_KEY_ID"
|
90
|
+
ENV["AMAZON_ACCESS_KEY_ID"]
|
91
|
+
else
|
92
|
+
options[:access_key]
|
93
|
+
end
|
94
|
+
secret_key = if options[:secret_key] == "$AMAZON_SECRET_ACCESS_KEY"
|
95
|
+
ENV["AMAZON_SECRET_ACCESS_KEY"]
|
96
|
+
else
|
97
|
+
options[:secret_key]
|
98
|
+
end
|
99
|
+
error("No access key given for S3. Please specify one.") unless access_key
|
100
|
+
error("No secret access key given for S3. Please specify one.") unless secret_key
|
101
|
+
AWS::S3::Base.establish_connection!(
|
102
|
+
:access_key_id => access_key,
|
103
|
+
:secret_access_key => secret_key
|
104
|
+
)
|
105
|
+
|
106
|
+
log("Retrieving existing manifests")
|
107
|
+
release = Deb::S3::Release.retrieve(options[:codename])
|
108
|
+
manifest = Deb::S3::Manifest.retrieve(options[:codename], options[:section], arch)
|
109
|
+
|
110
|
+
# add in the package
|
111
|
+
manifest.add(pkg)
|
112
|
+
|
113
|
+
log("Uploading package and new manifests to S3")
|
114
|
+
manifest.write_to_s3 { |f| sublog("Transferring #{f}") }
|
115
|
+
release.update_manifest(manifest)
|
116
|
+
release.write_to_s3 { |f| sublog("Transferring #{f}") }
|
117
|
+
|
118
|
+
log("Update complete.")
|
119
|
+
end
|
120
|
+
|
121
|
+
private
|
122
|
+
|
123
|
+
def log(message)
|
124
|
+
puts ">> #{message}"
|
125
|
+
end
|
126
|
+
|
127
|
+
def sublog(message)
|
128
|
+
puts " -- #{message}"
|
129
|
+
end
|
130
|
+
|
131
|
+
def error(message)
|
132
|
+
puts "!! #{message}"
|
133
|
+
exit 1
|
134
|
+
end
|
135
|
+
|
136
|
+
end
|
@@ -0,0 +1,100 @@
|
|
1
|
+
require "tempfile"
|
2
|
+
require "zlib"
|
3
|
+
|
4
|
+
class Deb::S3::Manifest
|
5
|
+
include Deb::S3::Utils
|
6
|
+
|
7
|
+
attr_accessor :codename
|
8
|
+
attr_accessor :component
|
9
|
+
attr_accessor :architecture
|
10
|
+
|
11
|
+
attr_accessor :files
|
12
|
+
|
13
|
+
def initialize
|
14
|
+
@packages = []
|
15
|
+
@component = nil
|
16
|
+
@architecture = nil
|
17
|
+
@files = {}
|
18
|
+
end
|
19
|
+
|
20
|
+
class << self
|
21
|
+
def retrieve(codename, component, architecture)
|
22
|
+
m = if s = Deb::S3::Utils.s3_read("dists/#{codename}/#{component}/binary-#{architecture}/Packages")
|
23
|
+
self.parse_packages(s)
|
24
|
+
else
|
25
|
+
self.new
|
26
|
+
end
|
27
|
+
|
28
|
+
m.codename = codename
|
29
|
+
m.component = component
|
30
|
+
m.architecture = architecture
|
31
|
+
m
|
32
|
+
end
|
33
|
+
|
34
|
+
def parse_packages(str)
|
35
|
+
m = self.new
|
36
|
+
str.split("\n\n").each do |s|
|
37
|
+
next if s.chomp.empty?
|
38
|
+
m.packages << Deb::S3::Package.parse_string(s)
|
39
|
+
end
|
40
|
+
m
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def packages
|
45
|
+
@packages
|
46
|
+
end
|
47
|
+
|
48
|
+
def add(pkg)
|
49
|
+
@packages.delete_if { |p| p.name == pkg.name }
|
50
|
+
@packages << pkg
|
51
|
+
pkg
|
52
|
+
end
|
53
|
+
|
54
|
+
def generate
|
55
|
+
@packages.collect { |pkg| pkg.generate }.join("\n")
|
56
|
+
end
|
57
|
+
|
58
|
+
def write_to_s3
|
59
|
+
manifest = self.generate
|
60
|
+
|
61
|
+
# store any packages that need to be stored
|
62
|
+
@packages.each do |pkg|
|
63
|
+
if pkg.needs_uploading?
|
64
|
+
yield pkg.url_filename if block_given?
|
65
|
+
s3_store(pkg.filename, pkg.url_filename_encoded)
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
# generate the Packages file
|
70
|
+
pkgs_temp = Tempfile.new("Packages")
|
71
|
+
pkgs_temp.write manifest
|
72
|
+
pkgs_temp.close
|
73
|
+
f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages"
|
74
|
+
yield f if block_given?
|
75
|
+
s3_store(pkgs_temp.path, f)
|
76
|
+
@files["#{@component}/binary-#{@architecture}/Packages"] = hashfile(pkgs_temp.path)
|
77
|
+
pkgs_temp.unlink
|
78
|
+
|
79
|
+
# generate the Packages.gz file
|
80
|
+
gztemp = Tempfile.new("Packages.gz")
|
81
|
+
gztemp.close
|
82
|
+
Zlib::GzipWriter.open(gztemp.path) { |gz| gz.write manifest }
|
83
|
+
f = "dists/#{@codename}/#{@component}/binary-#{@architecture}/Packages.gz"
|
84
|
+
yield f if block_given?
|
85
|
+
s3_store(gztemp.path, f)
|
86
|
+
@files["#{@component}/binary-#{@architecture}/Packages.gz"] = hashfile(gztemp.path)
|
87
|
+
gztemp.unlink
|
88
|
+
|
89
|
+
nil
|
90
|
+
end
|
91
|
+
|
92
|
+
def hashfile(path)
|
93
|
+
{
|
94
|
+
:size => File.size(path),
|
95
|
+
:sha1 => Digest::SHA1.file(path).hexdigest,
|
96
|
+
:sha256 => Digest::SHA2.file(path).hexdigest,
|
97
|
+
:md5 => Digest::MD5.file(path).hexdigest
|
98
|
+
}
|
99
|
+
end
|
100
|
+
end
|
@@ -0,0 +1,272 @@
|
|
1
|
+
require "digest/sha1"
|
2
|
+
require "digest/sha2"
|
3
|
+
require "digest/md5"
|
4
|
+
require "socket"
|
5
|
+
require "tmpdir"
|
6
|
+
|
7
|
+
class Deb::S3::Package
|
8
|
+
include Deb::S3::Utils
|
9
|
+
|
10
|
+
attr_accessor :name
|
11
|
+
attr_accessor :version
|
12
|
+
attr_accessor :epoch
|
13
|
+
attr_accessor :iteration
|
14
|
+
attr_accessor :maintainer
|
15
|
+
attr_accessor :vendor
|
16
|
+
attr_accessor :url
|
17
|
+
attr_accessor :category
|
18
|
+
attr_accessor :license
|
19
|
+
attr_accessor :architecture
|
20
|
+
attr_accessor :description
|
21
|
+
|
22
|
+
attr_accessor :dependencies
|
23
|
+
attr_accessor :provides
|
24
|
+
attr_accessor :conflicts
|
25
|
+
attr_accessor :replaces
|
26
|
+
attr_accessor :excludes
|
27
|
+
|
28
|
+
|
29
|
+
# Any other attributes specific to this package.
|
30
|
+
# This is where you'd put rpm, deb, or other specific attributes.
|
31
|
+
attr_accessor :attributes
|
32
|
+
|
33
|
+
# hashes
|
34
|
+
attr_accessor :url_filename
|
35
|
+
attr_accessor :sha1
|
36
|
+
attr_accessor :sha256
|
37
|
+
attr_accessor :md5
|
38
|
+
attr_accessor :size
|
39
|
+
|
40
|
+
attr_accessor :filename
|
41
|
+
|
42
|
+
class << self
|
43
|
+
include Deb::S3::Utils
|
44
|
+
|
45
|
+
def parse_file(package)
|
46
|
+
p = self.new
|
47
|
+
p.extract_info(extract_control(package))
|
48
|
+
p.apply_file_info(package)
|
49
|
+
p.filename = package
|
50
|
+
p
|
51
|
+
end
|
52
|
+
|
53
|
+
def parse_string(s)
|
54
|
+
p = self.new
|
55
|
+
p.extract_info(s)
|
56
|
+
p
|
57
|
+
end
|
58
|
+
|
59
|
+
def extract_control(package)
|
60
|
+
if system("which dpkg 2>&1 >/dev/null")
|
61
|
+
`dpkg -f #{package}`
|
62
|
+
else
|
63
|
+
Dir.mktmpdir do |path|
|
64
|
+
safesystem("ar p #{package} control.tar.gz | tar -zxf - -C #{path}")
|
65
|
+
File.read(File.join(path, "control"))
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
def initialize
|
72
|
+
@attributes = {}
|
73
|
+
|
74
|
+
# Reference
|
75
|
+
# http://www.debian.org/doc/manuals/maint-guide/first.en.html
|
76
|
+
# http://wiki.debian.org/DeveloperConfiguration
|
77
|
+
# https://github.com/jordansissel/fpm/issues/37
|
78
|
+
if ENV.include?("DEBEMAIL") and ENV.include?("DEBFULLNAME")
|
79
|
+
# Use DEBEMAIL and DEBFULLNAME as the default maintainer if available.
|
80
|
+
@maintainer = "#{ENV["DEBFULLNAME"]} <#{ENV["DEBEMAIL"]}>"
|
81
|
+
else
|
82
|
+
# TODO(sissel): Maybe support using 'git config' for a default as well?
|
83
|
+
# git config --get user.name, etc can be useful.
|
84
|
+
#
|
85
|
+
# Otherwise default to user@currenthost
|
86
|
+
@maintainer = "<#{ENV["USER"]}@#{Socket.gethostname}>"
|
87
|
+
end
|
88
|
+
|
89
|
+
@name = nil
|
90
|
+
@architecture = "native"
|
91
|
+
@description = "no description given"
|
92
|
+
@version = nil
|
93
|
+
@epoch = nil
|
94
|
+
@iteration = nil
|
95
|
+
@url = nil
|
96
|
+
@category = "default"
|
97
|
+
@license = "unknown"
|
98
|
+
@vendor = "none"
|
99
|
+
@sha1 = nil
|
100
|
+
@sha256 = nil
|
101
|
+
@md5 = nil
|
102
|
+
@size = nil
|
103
|
+
@filename = nil
|
104
|
+
@url_filename = nil
|
105
|
+
|
106
|
+
@provides = []
|
107
|
+
@conflicts = []
|
108
|
+
@replaces = []
|
109
|
+
@dependencies = []
|
110
|
+
|
111
|
+
@needs_uploading = false
|
112
|
+
end
|
113
|
+
|
114
|
+
def filename=(f)
|
115
|
+
@filename = f
|
116
|
+
@needs_uploading = true
|
117
|
+
@filename
|
118
|
+
end
|
119
|
+
|
120
|
+
def url_filename
|
121
|
+
@url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{File.basename(self.filename)}"
|
122
|
+
end
|
123
|
+
|
124
|
+
def url_filename_encoded
|
125
|
+
@url_filename || "pool/#{self.name[0]}/#{self.name[0..1]}/#{s3_escape(File.basename(self.filename))}"
|
126
|
+
end
|
127
|
+
|
128
|
+
def needs_uploading?
|
129
|
+
@needs_uploading
|
130
|
+
end
|
131
|
+
|
132
|
+
def generate
|
133
|
+
template("package.erb").result(binding)
|
134
|
+
end
|
135
|
+
|
136
|
+
# from fpm
|
137
|
+
def parse_depends(data)
|
138
|
+
return [] if data.nil? or data.empty?
|
139
|
+
# parse dependencies. Debian dependencies come in one of two forms:
|
140
|
+
# * name
|
141
|
+
# * name (op version)
|
142
|
+
# They are all on one line, separated by ", "
|
143
|
+
|
144
|
+
dep_re = /^([^ ]+)(?: \(([>=<]+) ([^)]+)\))?$/
|
145
|
+
return data.split(/, */).collect do |dep|
|
146
|
+
m = dep_re.match(dep)
|
147
|
+
if m
|
148
|
+
name, op, version = m.captures
|
149
|
+
# deb uses ">>" and "<<" for greater and less than respectively.
|
150
|
+
# fpm wants just ">" and "<"
|
151
|
+
op = "<" if op == "<<"
|
152
|
+
op = ">" if op == ">>"
|
153
|
+
# this is the proper form of dependency
|
154
|
+
"#{name} #{op} #{version}"
|
155
|
+
else
|
156
|
+
# Assume normal form dependency, "name op version".
|
157
|
+
dep
|
158
|
+
end
|
159
|
+
end
|
160
|
+
end # def parse_depends
|
161
|
+
|
162
|
+
# from fpm
|
163
|
+
def fix_dependency(dep)
|
164
|
+
# Deb dependencies are: NAME (OP VERSION), like "zsh (> 3.0)"
|
165
|
+
# Convert anything that looks like 'NAME OP VERSION' to this format.
|
166
|
+
if dep =~ /[\(,\|]/
|
167
|
+
# Don't "fix" ones that could appear well formed already.
|
168
|
+
else
|
169
|
+
# Convert ones that appear to be 'name op version'
|
170
|
+
name, op, version = dep.split(/ +/)
|
171
|
+
if !version.nil?
|
172
|
+
# Convert strings 'foo >= bar' to 'foo (>= bar)'
|
173
|
+
dep = "#{name} (#{debianize_op(op)} #{version})"
|
174
|
+
end
|
175
|
+
end
|
176
|
+
|
177
|
+
name_re = /^[^ \(]+/
|
178
|
+
name = dep[name_re]
|
179
|
+
if name =~ /[A-Z]/
|
180
|
+
dep = dep.gsub(name_re) { |n| n.downcase }
|
181
|
+
end
|
182
|
+
|
183
|
+
if dep.include?("_")
|
184
|
+
dep = dep.gsub("_", "-")
|
185
|
+
end
|
186
|
+
|
187
|
+
# Convert gem ~> X.Y.Z to '>= X.Y.Z' and << X.Y+1.0
|
188
|
+
if dep =~ /\(~>/
|
189
|
+
name, version = dep.gsub(/[()~>]/, "").split(/ +/)[0..1]
|
190
|
+
nextversion = version.split(".").collect { |v| v.to_i }
|
191
|
+
l = nextversion.length
|
192
|
+
nextversion[l-2] += 1
|
193
|
+
nextversion[l-1] = 0
|
194
|
+
nextversion = nextversion.join(".")
|
195
|
+
return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
|
196
|
+
elsif (m = dep.match(/(\S+)\s+\(!= (.+)\)/))
|
197
|
+
# Append this to conflicts
|
198
|
+
self.conflicts += [dep.gsub(/!=/,"=")]
|
199
|
+
return []
|
200
|
+
elsif (m = dep.match(/(\S+)\s+\(= (.+)\)/)) and
|
201
|
+
self.attributes[:deb_ignore_iteration_in_dependencies?]
|
202
|
+
# Convert 'foo (= x)' to 'foo (>= x)' and 'foo (<< x+1)'
|
203
|
+
# but only when flag --ignore-iteration-in-dependencies is passed.
|
204
|
+
name, version = m[1..2]
|
205
|
+
nextversion = version.split('.').collect { |v| v.to_i }
|
206
|
+
nextversion[-1] += 1
|
207
|
+
nextversion = nextversion.join(".")
|
208
|
+
return ["#{name} (>= #{version})", "#{name} (<< #{nextversion})"]
|
209
|
+
else
|
210
|
+
# otherwise the dep is probably fine
|
211
|
+
return dep.rstrip
|
212
|
+
end
|
213
|
+
end # def fix_dependency
|
214
|
+
|
215
|
+
# from fpm
|
216
|
+
def extract_info(control)
|
217
|
+
parse = lambda do |field|
|
218
|
+
value = control[/^#{field}: .*/]
|
219
|
+
if value.nil?
|
220
|
+
return nil
|
221
|
+
else
|
222
|
+
return value.split(": ",2).last
|
223
|
+
end
|
224
|
+
end
|
225
|
+
|
226
|
+
# Parse 'epoch:version-iteration' in the version string
|
227
|
+
version_re = /^(?:([0-9]+):)?(.+?)(?:-(.*))?$/
|
228
|
+
m = version_re.match(parse.call("Version"))
|
229
|
+
if !m
|
230
|
+
raise "Unsupported version string '#{parse.call("Version")}'"
|
231
|
+
end
|
232
|
+
self.epoch, self.version, self.iteration = m.captures
|
233
|
+
|
234
|
+
self.architecture = parse.call("Architecture")
|
235
|
+
self.category = parse.call("Section")
|
236
|
+
self.license = parse.call("License") || self.license
|
237
|
+
self.maintainer = parse.call("Maintainer")
|
238
|
+
self.name = parse.call("Package")
|
239
|
+
self.url = parse.call("Homepage")
|
240
|
+
self.vendor = parse.call("Vendor") || self.vendor
|
241
|
+
self.attributes[:deb_priority] = parse.call("Priority")
|
242
|
+
|
243
|
+
# Packages manifest fields
|
244
|
+
self.url_filename = parse.call("Filename")
|
245
|
+
self.sha1 = parse.call("SHA1")
|
246
|
+
self.sha256 = parse.call("SHA256")
|
247
|
+
self.md5 = parse.call("MD5sum")
|
248
|
+
self.size = parse.call("Size")
|
249
|
+
|
250
|
+
# The description field is a special flower, parse it that way.
|
251
|
+
# The description is the first line as a normal Description field, but also continues
|
252
|
+
# on future lines indented by one space, until the end of the file. Blank
|
253
|
+
# lines are marked as ' .'
|
254
|
+
description = control[/^Description: .*[^\Z]/m]
|
255
|
+
description = description.gsub(/^[^(Description|\s)].*$/, "").split(": ", 2).last
|
256
|
+
self.description = description.gsub(/^ /, "").gsub(/^\.$/, "")
|
257
|
+
|
258
|
+
#self.config_files = config_files
|
259
|
+
|
260
|
+
self.dependencies += Array(parse_depends(parse.call("Depends")))
|
261
|
+
self.conflicts += Array(parse_depends(parse.call("Conflicts")))
|
262
|
+
self.provides += Array(parse_depends(parse.call("Provides")))
|
263
|
+
self.replaces += Array(parse_depends(parse.call("Replaces")))
|
264
|
+
end # def extract_info
|
265
|
+
|
266
|
+
def apply_file_info(file)
|
267
|
+
self.size = File.size(file)
|
268
|
+
self.sha1 = Digest::SHA1.file(file).hexdigest
|
269
|
+
self.sha256 = Digest::SHA2.file(file).hexdigest
|
270
|
+
self.md5 = Digest::MD5.file(file).hexdigest
|
271
|
+
end
|
272
|
+
end
|
@@ -0,0 +1,139 @@
|
|
1
|
+
require "tempfile"
|
2
|
+
|
3
|
+
class Deb::S3::Release
|
4
|
+
include Deb::S3::Utils
|
5
|
+
|
6
|
+
attr_accessor :codename
|
7
|
+
attr_accessor :architectures
|
8
|
+
attr_accessor :components
|
9
|
+
|
10
|
+
attr_accessor :files
|
11
|
+
attr_accessor :policy
|
12
|
+
|
13
|
+
def initialize
|
14
|
+
@codename = nil
|
15
|
+
@architectures = []
|
16
|
+
@components = []
|
17
|
+
@files = {}
|
18
|
+
@policy = :public_read
|
19
|
+
end
|
20
|
+
|
21
|
+
class << self
|
22
|
+
def retrieve(codename)
|
23
|
+
if s = Deb::S3::Utils.s3_read("dists/#{codename}/Release")
|
24
|
+
self.parse_release(s)
|
25
|
+
else
|
26
|
+
rel = self.new
|
27
|
+
rel.codename = codename
|
28
|
+
rel
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
def parse_release(str)
|
33
|
+
rel = self.new
|
34
|
+
rel.parse(str)
|
35
|
+
rel
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
def filename
|
40
|
+
"dists/#{@codename}/Release"
|
41
|
+
end
|
42
|
+
|
43
|
+
def parse(str)
|
44
|
+
parse = lambda do |field|
|
45
|
+
value = str[/^#{field}: .*/]
|
46
|
+
if value.nil?
|
47
|
+
return nil
|
48
|
+
else
|
49
|
+
return value.split(": ",2).last
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
# grab basic fields
|
54
|
+
self.codename = parse.call("Codename")
|
55
|
+
self.architectures = (parse.call("Architectures") || "").split(/\s+/)
|
56
|
+
self.components = (parse.call("Components") || "").split(/\s+/)
|
57
|
+
|
58
|
+
# find all the hashes
|
59
|
+
str.scan(/^\s+([^\s]+)\s+(\d+)\s+(.+)$/).each do |(hash,size,name)|
|
60
|
+
self.files[name] ||= { :size => size.to_i }
|
61
|
+
case hash.length
|
62
|
+
when 32
|
63
|
+
self.files[name][:md5] = hash
|
64
|
+
when 40
|
65
|
+
self.files[name][:sha1] = hash
|
66
|
+
when 64
|
67
|
+
self.files[name][:sha256] = hash
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
def generate
|
73
|
+
template("release.erb").result(binding)
|
74
|
+
end
|
75
|
+
|
76
|
+
def write_to_s3
|
77
|
+
# validate some other files are present
|
78
|
+
if block_given?
|
79
|
+
self.validate_others { |f| yield f }
|
80
|
+
else
|
81
|
+
self.validate_others
|
82
|
+
end
|
83
|
+
|
84
|
+
# generate the Release file
|
85
|
+
release_tmp = Tempfile.new("Release")
|
86
|
+
release_tmp.puts self.generate
|
87
|
+
release_tmp.close
|
88
|
+
yield self.filename if block_given?
|
89
|
+
s3_store(release_tmp.path, self.filename)
|
90
|
+
|
91
|
+
# sign the file, if necessary
|
92
|
+
if Deb::S3::Utils.signing_key
|
93
|
+
key_param = Deb::S3::Utils.signing_key != "" ? "--default-key=#{Deb::S3::Utils.signing_key}" : ""
|
94
|
+
if system("gpg -a #{key_param} -b #{release_tmp.path}")
|
95
|
+
local_file = release_tmp.path+".asc"
|
96
|
+
remote_file = self.filename+".gpg"
|
97
|
+
yield remote_file if block_given?
|
98
|
+
raise "Unable to locate Release signature file" unless File.exists?(local_file)
|
99
|
+
s3_store(local_file, remote_file)
|
100
|
+
File.unlink(local_file)
|
101
|
+
else
|
102
|
+
raise "Signing the Release file failed."
|
103
|
+
end
|
104
|
+
else
|
105
|
+
# remove an existing Release.gpg, if it was there
|
106
|
+
s3_remove(self.filename+".gpg")
|
107
|
+
end
|
108
|
+
|
109
|
+
release_tmp.unlink
|
110
|
+
end
|
111
|
+
|
112
|
+
def update_manifest(manifest)
|
113
|
+
self.components << manifest.component unless self.components.include?(manifest.component)
|
114
|
+
self.architectures << manifest.architecture unless self.architectures.include?(manifest.architecture)
|
115
|
+
self.files.merge!(manifest.files)
|
116
|
+
end
|
117
|
+
|
118
|
+
def validate_others
|
119
|
+
to_apply = []
|
120
|
+
self.components.each do |comp|
|
121
|
+
%w(amd64 i386).each do |arch|
|
122
|
+
next if self.files.has_key?("#{comp}/binary-#{arch}/Packages")
|
123
|
+
|
124
|
+
m = Deb::S3::Manifest.new
|
125
|
+
m.codename = self.codename
|
126
|
+
m.component = comp
|
127
|
+
m.architecture = arch
|
128
|
+
if block_given?
|
129
|
+
m.write_to_s3 { |f| yield f }
|
130
|
+
else
|
131
|
+
m.write_to_s3
|
132
|
+
end
|
133
|
+
to_apply << m
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
137
|
+
to_apply.each { |m| self.update_manifest(m) }
|
138
|
+
end
|
139
|
+
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
Package: <%= name %>
|
2
|
+
Version: <%= "#{epoch}:" if epoch %><%= version %><%= "-" + iteration.to_s if iteration %>
|
3
|
+
License: <%= license %>
|
4
|
+
Vendor: <%= vendor %>
|
5
|
+
Architecture: <%= architecture %>
|
6
|
+
Maintainer: <%= maintainer %>
|
7
|
+
Installed-Size: <%= attributes[:deb_installed_size] %>
|
8
|
+
<% if !dependencies.empty? and !attributes[:no_depends?] -%>
|
9
|
+
Depends: <%= dependencies.collect { |d| fix_dependency(d) }.flatten.join(", ") %>
|
10
|
+
<% end -%>
|
11
|
+
<% if !conflicts.empty? -%>
|
12
|
+
Conflicts: <%= conflicts.join(", ") %>
|
13
|
+
<% end -%>
|
14
|
+
<% if attributes[:deb_pre_depends_given?] -%>
|
15
|
+
Pre-Depends: <%= attributes[:deb_pre_depends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
|
16
|
+
<% end -%>
|
17
|
+
<% if !provides.empty? -%>
|
18
|
+
<%# Turn each provides from 'foo = 123' to simply 'foo' because Debian :\ -%>
|
19
|
+
<%# http://www.debian.org/doc/debian-policy/ch-relationships.html -%>
|
20
|
+
Provides: <%= provides.map {|p| p.split(" ").first}.join ", " %>
|
21
|
+
<% end -%>
|
22
|
+
<% if !replaces.empty? -%>
|
23
|
+
Replaces: <%= replaces.join(", ") %>
|
24
|
+
<% end -%>
|
25
|
+
<% if attributes[:deb_recommends_given?] -%>
|
26
|
+
Recommends: <%= attributes[:deb_recommends].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
|
27
|
+
<% end -%>
|
28
|
+
<% if attributes[:deb_suggests_given?] -%>
|
29
|
+
Suggests: <%= attributes[:deb_suggests].collect { |d| fix_dependency(d) }.flatten.join(", ") %>
|
30
|
+
<% end -%>
|
31
|
+
Section: <%= category %>
|
32
|
+
Priority: <%= attributes[:deb_priority] %>
|
33
|
+
Homepage: <%= url or "http://nourlgiven.example.com/" %>
|
34
|
+
Filename: <%= url_filename_encoded %>
|
35
|
+
<% if size -%>
|
36
|
+
Size: <%= size %>
|
37
|
+
<% end -%>
|
38
|
+
<% if sha1 -%>
|
39
|
+
SHA1: <%= sha1 %>
|
40
|
+
<% end -%>
|
41
|
+
<% if sha256 -%>
|
42
|
+
SHA256: <%= sha256 %>
|
43
|
+
<% end -%>
|
44
|
+
<% if md5 -%>
|
45
|
+
MD5sum: <%= md5 %>
|
46
|
+
<% end -%>
|
47
|
+
<% lines = (description or "no description given").split("\n") -%>
|
48
|
+
<% firstline, *remainder = lines -%>
|
49
|
+
Description: <%= firstline %>
|
50
|
+
<% if remainder.any? -%>
|
51
|
+
<%= remainder.collect { |l| l =~ /^ *$/ ? " ." : " #{l}" }.join("\n") %>
|
52
|
+
<% end -%>
|
53
|
+
<% if attributes[:deb_field_given?] -%>
|
54
|
+
<% attributes[:deb_field].each do |field, value| -%>
|
55
|
+
<%= field %>: <%= value %>
|
56
|
+
<% end -%>
|
57
|
+
<% end -%>
|
@@ -0,0 +1,16 @@
|
|
1
|
+
Codename: <%= codename %>
|
2
|
+
Date: <%= Time.now.utc.strftime("%a, %d %b %Y %T %Z") %>
|
3
|
+
Architectures: <%= architectures.join(" ") %>
|
4
|
+
Components: <%= components.join(" ") %>
|
5
|
+
MD5Sum:
|
6
|
+
<% files.each do |f,p| -%>
|
7
|
+
<%= p[:md5] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
|
8
|
+
<% end -%>
|
9
|
+
SHA1:
|
10
|
+
<% files.each do |f,p| -%>
|
11
|
+
<%= p[:sha1] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
|
12
|
+
<% end -%>
|
13
|
+
SHA256:
|
14
|
+
<% files.each do |f,p| -%>
|
15
|
+
<%= p[:sha256] %> <%= p[:size].to_s.rjust(16) %> <%= f %>
|
16
|
+
<% end -%>
|
data/lib/deb/s3/utils.rb
ADDED
@@ -0,0 +1,64 @@
|
|
1
|
+
require "erb"
|
2
|
+
require "tmpdir"
|
3
|
+
|
4
|
+
module Deb::S3::Utils
|
5
|
+
module_function
|
6
|
+
def bucket; @bucket end
|
7
|
+
def bucket= v; @bucket = v end
|
8
|
+
def access_policy; @access_policy end
|
9
|
+
def access_policy= v; @access_policy = v end
|
10
|
+
def signing_key; @signing_key end
|
11
|
+
def signing_key= v; @signing_key = v end
|
12
|
+
|
13
|
+
def safesystem(*args)
|
14
|
+
success = system(*args)
|
15
|
+
if !success
|
16
|
+
raise "'system(#{args.inspect})' failed with error code: #{$?.exitstatus}"
|
17
|
+
end
|
18
|
+
return success
|
19
|
+
end
|
20
|
+
|
21
|
+
def debianize_op(op)
|
22
|
+
# Operators in debian packaging are <<, <=, =, >= and >>
|
23
|
+
# So any operator like < or > must be replaced
|
24
|
+
{:< => "<<", :> => ">>"}[op.to_sym] or op
|
25
|
+
end
|
26
|
+
|
27
|
+
def template(path)
|
28
|
+
template_file = File.join(File.dirname(__FILE__), "templates", path)
|
29
|
+
template_code = File.read(template_file)
|
30
|
+
ERB.new(template_code, nil, "-")
|
31
|
+
end
|
32
|
+
|
33
|
+
# from fog, Fog::AWS.escape
|
34
|
+
def s3_escape(string)
|
35
|
+
string.gsub(/([^a-zA-Z0-9_.\-~]+)/) {
|
36
|
+
"%" + $1.unpack("H2" * $1.bytesize).join("%").upcase
|
37
|
+
}
|
38
|
+
end
|
39
|
+
|
40
|
+
def s3_exists?(path)
|
41
|
+
AWS::S3::S3Object.exists?(path, Deb::S3::Utils.bucket)
|
42
|
+
end
|
43
|
+
|
44
|
+
def s3_read(path)
|
45
|
+
return nil unless s3_exists?(path)
|
46
|
+
s = ""
|
47
|
+
AWS::S3::S3Object.stream(path, Deb::S3::Utils.bucket) do |chunk|
|
48
|
+
s += chunk
|
49
|
+
end
|
50
|
+
s
|
51
|
+
end
|
52
|
+
|
53
|
+
def s3_store(path, filename=nil)
|
54
|
+
filename = File.basename(path) unless filename
|
55
|
+
File.open(path) do |file|
|
56
|
+
AWS::S3::S3Object.store(filename, file,
|
57
|
+
Deb::S3::Utils.bucket, :access => Deb::S3::Utils.access_policy)
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
def s3_remove(path)
|
62
|
+
AWS::S3::S3Object.delete(Deb::S3::Utils.bucket, path) if s3_exists?(path)
|
63
|
+
end
|
64
|
+
end
|
metadata
ADDED
@@ -0,0 +1,103 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: deb-s3
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Ken Robertson
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2013-02-26 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: thor
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ~>
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: 0.17.0
|
22
|
+
type: :runtime
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.17.0
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: aws-s3
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ~>
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: 0.6.3
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ~>
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: 0.6.3
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: rake
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ! '>='
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0'
|
54
|
+
type: :development
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ! '>='
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
description: Easily create and manage an APT repository on S3.
|
63
|
+
email: ken@invalidlogic.com
|
64
|
+
executables:
|
65
|
+
- deb-s3
|
66
|
+
extensions: []
|
67
|
+
extra_rdoc_files: []
|
68
|
+
files:
|
69
|
+
- bin/deb-s3
|
70
|
+
- lib/deb/s3/cli.rb
|
71
|
+
- lib/deb/s3/manifest.rb
|
72
|
+
- lib/deb/s3/package.rb
|
73
|
+
- lib/deb/s3/release.rb
|
74
|
+
- lib/deb/s3/templates/package.erb
|
75
|
+
- lib/deb/s3/templates/release.erb
|
76
|
+
- lib/deb/s3/utils.rb
|
77
|
+
- lib/deb/s3.rb
|
78
|
+
- README.md
|
79
|
+
homepage: http://invalidlogic.com/
|
80
|
+
licenses: []
|
81
|
+
post_install_message:
|
82
|
+
rdoc_options: []
|
83
|
+
require_paths:
|
84
|
+
- lib
|
85
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
86
|
+
none: false
|
87
|
+
requirements:
|
88
|
+
- - ! '>='
|
89
|
+
- !ruby/object:Gem::Version
|
90
|
+
version: '0'
|
91
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
92
|
+
none: false
|
93
|
+
requirements:
|
94
|
+
- - ! '>='
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
requirements: []
|
98
|
+
rubyforge_project:
|
99
|
+
rubygems_version: 1.8.23
|
100
|
+
signing_key:
|
101
|
+
specification_version: 3
|
102
|
+
summary: Easily create and manage an APT repository on S3.
|
103
|
+
test_files: []
|