deb-s3-lock-fix 0.11.8.fix1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (13) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +284 -0
  3. data/bin/deb-s3 +10 -0
  4. data/lib/deb/s3/cli.rb +763 -0
  5. data/lib/deb/s3/lock.rb +125 -0
  6. data/lib/deb/s3/manifest.rb +144 -0
  7. data/lib/deb/s3/package.rb +309 -0
  8. data/lib/deb/s3/release.rb +161 -0
  9. data/lib/deb/s3/templates/package.erb +66 -0
  10. data/lib/deb/s3/templates/release.erb +20 -0
  11. data/lib/deb/s3/utils.rb +115 -0
  12. data/lib/deb/s3.rb +6 -0
  13. metadata +111 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 180e005ba32e429abe5a462bb81cd03d6897b02fb10e079a655008589fa687df
4
+ data.tar.gz: df8aeb29f17894bdfca5f82caaaf5ad6576d07de0fed3789d532de3c93486a13
5
+ SHA512:
6
+ metadata.gz: '09eaff5677bd51d239794dd5b5bcf9f9ac18275593cd51088f22a0bded0bdd1c2bc4c82cf771aaa8c32d7fd525c973919ac6a5fe4b85818923922e576fa75b13'
7
+ data.tar.gz: c463aba5d27113100687654aebd7952ca32142f2835190038dfaf401918d004cdf9bdca74e641866b4e90e5ec91cacbd94bde28c504d0ea637c798181d35e9aa
data/README.md ADDED
@@ -0,0 +1,284 @@
1
+ # deb-s3
2
+
3
+ [![Build Status](https://travis-ci.org/deb-s3/deb-s3.svg?branch=master)](https://travis-ci.org/deb-s3/deb-s3)
4
+
5
+ **This repository is a fork of [krobertson/deb-s3](https://github.com/krobertson/deb-s3).**
6
+
7
+ `deb-s3` is a simple utility to make creating and managing APT repositories on
8
+ S3.
9
+
10
+ Most existing guides on using S3 to host an APT repository have you
11
+ using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
12
+ the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
13
+ sync the files to S3.
14
+
15
+ The annoying thing about this process is it requires you to maintain a local
16
+ copy of the file tree for regenerating and syncing the next time. Personally,
17
+ my process is to use one-off virtual machines with
18
+ [Vagrant](http://vagrantup.com), script out the build process, and then would
19
+ prefer to just upload the final `.deb` from my Mac.
20
+
21
+ With `deb-s3`, there is no need for this. `deb-s3` features:
22
+
23
+ * Downloads the existing package manifest and parses it.
24
+ * Updates it with the new package, replacing the existing entry if already
25
+ there or adding a new one if not.
26
+ * Uploads the package itself, the Packages manifest, and the Packages.gz
27
+ manifest. It will skip the uploading if the package is already there.
28
+ * Updates the Release file with the new hashes and file sizes.
29
+
30
+ ## Getting Started
31
+
32
+ Install the package via gem
33
+
34
+ ```console
35
+ $ gem install deb-s3
36
+ ```
37
+
38
+ or via APT (Debian 11 "Bullseye" and Ubuntu 22.04 "Jammy Jellyfish" and newer):
39
+
40
+ ```console
41
+ # Add repository key
42
+ $ sudo wget -O /etc/apt/trusted.gpg.d/deb-s3-archive-keyring.gpg https://raw.githubusercontent.com/deb-s3/deb-s3/master/deb-s3-archive-keyring.gpg
43
+
44
+ # Add repository
45
+ $ echo "deb http://deb-s3-repo.s3.us-east-2.amazonaws.com/$(lsb_release -is | tr A-Z a-z)/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list > /dev/null
46
+
47
+ # Install package
48
+ $ sudo apt-get update
49
+ $ sudo apt-get install deb-s3
50
+ ```
51
+
52
+ To run the code directly, just check out the repo and run bundler to ensure
53
+ all dependencies are installed:
54
+
55
+ ```console
56
+ $ git clone https://github.com/deb-s3/deb-s3.git
57
+ $ cd deb-s3
58
+ $ bundle install
59
+ ```
60
+
61
+ Now to upload a package, simply use:
62
+
63
+ ```console
64
+ $ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
65
+ >> Examining package file my-deb-package-1.0.0_amd64.deb
66
+ >> Retrieving existing package manifest
67
+ >> Uploading package and new manifests to S3
68
+ -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
69
+ -- Transferring dists/stable/main/binary-amd64/Packages
70
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
71
+ -- Transferring dists/stable/Release
72
+ >> Update complete.
73
+ ```
74
+
75
+ ```
76
+ Usage:
77
+ deb-s3 upload FILES
78
+
79
+ Options:
80
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
81
+ -p, [--preserve-versions], [--no-preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
82
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
83
+ [--fail-if-exists], [--no-fail-if-exists] # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
84
+ [--skip-package-upload], [--no-skip-package-upload] # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
85
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
86
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
87
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
88
+ [--suite=SUITE] # The suite to use in the repository Release file.
89
+ -c, [--codename=CODENAME] # The codename of the APT repository.
90
+ # Default: stable
91
+ -m, [--component=COMPONENT] # The component of the APT repository.
92
+ # Default: main
93
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
94
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
95
+ [--s3-region=S3_REGION] # The region for connecting to S3.
96
+ # Default: us-east-1
97
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
98
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
99
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
100
+ # Default: public
101
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
102
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
103
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
104
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
105
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
106
+
107
+ Uploads the given files to a S3 bucket as an APT repository.
108
+ ```
109
+
110
+ You can also delete packages from the APT repository. Please keep in mind that
111
+ this does NOT delete the .deb file itself (the `clean` command does that), it
112
+ only removes it from the list of packages in the specified component, codename
113
+ and architecture.
114
+
115
+ Now to delete the package:
116
+ ```console
117
+ $ deb-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
118
+ >> Retrieving existing manifests
119
+ -- Deleting my-deb-package version 1.0.0
120
+ >> Uploading new manifests to S3
121
+ -- Transferring dists/stable/main/binary-amd64/Packages
122
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
123
+ -- Transferring dists/stable/Release
124
+ >> Update complete.
125
+ ```
126
+
127
+ ```
128
+ Usage:
129
+ deb-s3 delete PACKAGE
130
+
131
+ Options:
132
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
133
+ [--versions=one two three] # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
134
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
135
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
136
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
137
+ [--suite=SUITE] # The suite to use in the repository Release file.
138
+ -c, [--codename=CODENAME] # The codename of the APT repository.
139
+ # Default: stable
140
+ -m, [--component=COMPONENT] # The component of the APT repository.
141
+ # Default: main
142
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
143
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
144
+ [--s3-region=S3_REGION] # The region for connecting to S3.
145
+ # Default: us-east-1
146
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
147
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
148
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
149
+ # Default: public
150
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
151
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
152
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
153
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
154
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
155
+
156
+ Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
157
+ ```
158
+
159
+ Dangling `.deb` files left by the `delete` command (or uploading new versions) can be removed using the `clean` command:
160
+
161
+ ```console
162
+ $ deb-s3 clean --bucket my-bucket
163
+ >> Retrieving existing manifests
164
+ >> Searching for unreferenced packages
165
+ -- pool/m/my/my-deb-package-1.0.0_amd64.deb
166
+ ```
167
+
168
+ ```
169
+ Usage:
170
+ deb-s3 clean
171
+
172
+ Options:
173
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
174
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
175
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
176
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
177
+ [--suite=SUITE] # The suite to use in the repository Release file.
178
+ -c, [--codename=CODENAME] # The codename of the APT repository.
179
+ # Default: stable
180
+ -m, [--component=COMPONENT] # The component of the APT repository.
181
+ # Default: main
182
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
183
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
184
+ [--session-token=SESSION_TOKEN] # The (optional) session token for connecting to S3.
185
+ [--endpoint=ENDPOINT] # The URL endpoint to the S3 API.
186
+ [--s3-region=S3_REGION] # The region for connecting to S3.
187
+ # Default: us-east-1
188
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
189
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
190
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
191
+ # Default: public
192
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
193
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
194
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
195
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
196
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
197
+
198
+ Delete packages from the pool which are no longer referenced
199
+ ```
200
+
201
+ You can also verify an existing APT repository on S3 using the `verify` command:
202
+
203
+ ```console
204
+ deb-s3 verify -b my-bucket
205
+ >> Retrieving existing manifests
206
+ >> Checking for missing packages in: stable/main i386
207
+ >> Checking for missing packages in: stable/main amd64
208
+ >> Checking for missing packages in: stable/main all
209
+ ```
210
+
211
+ ```
212
+ Usage:
213
+ deb-s3 verify
214
+
215
+ Options:
216
+ -f, [--fix-manifests], [--no-fix-manifests] # Whether to fix problems in manifests when verifying.
217
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
218
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
219
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
220
+ [--suite=SUITE] # The suite to use in the repository Release file.
221
+ -c, [--codename=CODENAME] # The codename of the APT repository.
222
+ # Default: stable
223
+ -m, [--component=COMPONENT] # The component of the APT repository.
224
+ # Default: main
225
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
226
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
227
+ [--s3-region=S3_REGION] # The region for connecting to S3.
228
+ # Default: us-east-1
229
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
230
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
231
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
232
+ # Default: public
233
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
234
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
235
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
236
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
237
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
238
+
239
+ Verifies that the files in the package manifests exist
240
+ ```
241
+
242
+ #### Example S3 IAM Policy
243
+
244
+ ```
245
+ {
246
+ "Version": "2012-10-17",
247
+ "Statement": [
248
+ {
249
+ "Effect": "Allow",
250
+ "Action": [
251
+ "s3:ListBucket"
252
+ ],
253
+ "Resource": [
254
+ "arn:aws:s3:::BUCKETNAME",
255
+ ]
256
+ },
257
+ {
258
+ "Effect": "Allow",
259
+ "Action": [
260
+ "s3:PutObject",
261
+ "s3:GetObject",
262
+ "s3:DeleteObject",
263
+ "s3:DeleteObjectVersion",
264
+ "s3:GetObjectAcl",
265
+ "s3:GetObjectTagging",
266
+ "s3:GetObjectTorrent",
267
+ "s3:GetObjectVersion",
268
+ "s3:GetObjectVersionAcl",
269
+ "s3:GetObjectVersionTagging",
270
+ "s3:GetObjectVersionTorrent",
271
+ "s3:PutObjectAcl",
272
+ "s3:PutObjectTagging",
273
+ "s3:PutObjectVersionAcl",
274
+ "s3:PutObjectVersionTagging",
275
+ "s3:ReplicateObject",
276
+ "s3:RestoreObject"
277
+ ],
278
+ "Resource": [
279
+ "arn:aws:s3:::BUCKETNAME/*"
280
+ ]
281
+ }
282
+ ]
283
+ }
284
+ ```
data/bin/deb-s3 ADDED
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'pathname'
4
+ $:.unshift File.join(Pathname.new(__FILE__).realpath,'../../lib')
5
+
6
+ require 'rubygems'
7
+ require 'deb/s3/cli'
8
+
9
+ Deb::S3::CLI.start
10
+