deb-s3-lock-fix 0.11.8.fix1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +284 -0
  3. data/bin/deb-s3 +10 -0
  4. data/lib/deb/s3/cli.rb +763 -0
  5. data/lib/deb/s3/lock.rb +125 -0
  6. data/lib/deb/s3/manifest.rb +144 -0
  7. data/lib/deb/s3/package.rb +309 -0
  8. data/lib/deb/s3/release.rb +161 -0
  9. data/lib/deb/s3/templates/package.erb +66 -0
  10. data/lib/deb/s3/templates/release.erb +20 -0
  11. data/lib/deb/s3/utils.rb +115 -0
  12. data/lib/deb/s3.rb +6 -0
  13. metadata +111 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 180e005ba32e429abe5a462bb81cd03d6897b02fb10e079a655008589fa687df
4
+ data.tar.gz: df8aeb29f17894bdfca5f82caaaf5ad6576d07de0fed3789d532de3c93486a13
5
+ SHA512:
6
+ metadata.gz: '09eaff5677bd51d239794dd5b5bcf9f9ac18275593cd51088f22a0bded0bdd1c2bc4c82cf771aaa8c32d7fd525c973919ac6a5fe4b85818923922e576fa75b13'
7
+ data.tar.gz: c463aba5d27113100687654aebd7952ca32142f2835190038dfaf401918d004cdf9bdca74e641866b4e90e5ec91cacbd94bde28c504d0ea637c798181d35e9aa
data/README.md ADDED
@@ -0,0 +1,284 @@
1
+ # deb-s3
2
+
3
+ [![Build Status](https://travis-ci.org/deb-s3/deb-s3.svg?branch=master)](https://travis-ci.org/deb-s3/deb-s3)
4
+
5
+ **This repository is a fork of [krobertson/deb-s3](https://github.com/krobertson/deb-s3).**
6
+
7
+ `deb-s3` is a simple utility to make creating and managing APT repositories on
8
+ S3.
9
+
10
+ Most existing guides on using S3 to host an APT repository have you
11
+ using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
12
+ the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
13
+ sync the files to S3.
14
+
15
+ The annoying thing about this process is it requires you to maintain a local
16
+ copy of the file tree for regenerating and syncing the next time. Personally,
17
+ my process is to use one-off virtual machines with
18
+ [Vagrant](http://vagrantup.com), script out the build process, and then would
19
+ prefer to just upload the final `.deb` from my Mac.
20
+
21
+ With `deb-s3`, there is no need for this. `deb-s3` features:
22
+
23
+ * Downloads the existing package manifest and parses it.
24
+ * Updates it with the new package, replacing the existing entry if already
25
+ there or adding a new one if not.
26
+ * Uploads the package itself, the Packages manifest, and the Packages.gz
27
+ manifest. It will skip the uploading if the package is already there.
28
+ * Updates the Release file with the new hashes and file sizes.
29
+
30
+ ## Getting Started
31
+
32
+ Install the package via gem
33
+
34
+ ```console
35
+ $ gem install deb-s3
36
+ ```
37
+
38
+ or via APT (Debian 11 "Bullseye" and Ubuntu 22.04 "Jammy Jellyfish" and newer):
39
+
40
+ ```console
41
+ # Add repository key
42
+ $ sudo wget -O /etc/apt/trusted.gpg.d/deb-s3-archive-keyring.gpg https://raw.githubusercontent.com/deb-s3/deb-s3/master/deb-s3-archive-keyring.gpg
43
+
44
+ # Add repository
45
+ $ echo "deb http://deb-s3-repo.s3.us-east-2.amazonaws.com/$(lsb_release -is | tr A-Z a-z)/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list > /dev/null
46
+
47
+ # Install package
48
+ $ sudo apt-get update
49
+ $ sudo apt-get install deb-s3
50
+ ```
51
+
52
+ To run the code directly, just check out the repo and run bundler to ensure
53
+ all dependencies are installed:
54
+
55
+ ```console
56
+ $ git clone https://github.com/deb-s3/deb-s3.git
57
+ $ cd deb-s3
58
+ $ bundle install
59
+ ```
60
+
61
+ Now to upload a package, simply use:
62
+
63
+ ```console
64
+ $ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
65
+ >> Examining package file my-deb-package-1.0.0_amd64.deb
66
+ >> Retrieving existing package manifest
67
+ >> Uploading package and new manifests to S3
68
+ -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
69
+ -- Transferring dists/stable/main/binary-amd64/Packages
70
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
71
+ -- Transferring dists/stable/Release
72
+ >> Update complete.
73
+ ```
74
+
75
+ ```
76
+ Usage:
77
+ deb-s3 upload FILES
78
+
79
+ Options:
80
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
81
+ -p, [--preserve-versions], [--no-preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
82
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
83
+ [--fail-if-exists], [--no-fail-if-exists] # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
84
+ [--skip-package-upload], [--no-skip-package-upload] # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
85
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
86
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
87
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
88
+ [--suite=SUITE] # The suite to use in the repository Release file.
89
+ -c, [--codename=CODENAME] # The codename of the APT repository.
90
+ # Default: stable
91
+ -m, [--component=COMPONENT] # The component of the APT repository.
92
+ # Default: main
93
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
94
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
95
+ [--s3-region=S3_REGION] # The region for connecting to S3.
96
+ # Default: us-east-1
97
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
98
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
99
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
100
+ # Default: public
101
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
102
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
103
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
104
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
105
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
106
+
107
+ Uploads the given files to a S3 bucket as an APT repository.
108
+ ```
109
+
110
+ You can also delete packages from the APT repository. Please keep in mind that
111
+ this does NOT delete the .deb file itself (the `clean` command does that), it
112
+ only removes it from the list of packages in the specified component, codename
113
+ and architecture.
114
+
115
+ Now to delete the package:
116
+ ```console
117
+ $ deb-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
118
+ >> Retrieving existing manifests
119
+ -- Deleting my-deb-package version 1.0.0
120
+ >> Uploading new manifests to S3
121
+ -- Transferring dists/stable/main/binary-amd64/Packages
122
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
123
+ -- Transferring dists/stable/Release
124
+ >> Update complete.
125
+ ```
126
+
127
+ ```
128
+ Usage:
129
+ deb-s3 delete PACKAGE
130
+
131
+ Options:
132
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
133
+ [--versions=one two three] # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
134
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
135
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
136
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
137
+ [--suite=SUITE] # The suite to use in the repository Release file.
138
+ -c, [--codename=CODENAME] # The codename of the APT repository.
139
+ # Default: stable
140
+ -m, [--component=COMPONENT] # The component of the APT repository.
141
+ # Default: main
142
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
143
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
144
+ [--s3-region=S3_REGION] # The region for connecting to S3.
145
+ # Default: us-east-1
146
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
147
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
148
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
149
+ # Default: public
150
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
151
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
152
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
153
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
154
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
155
+
156
+ Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
157
+ ```
158
+
159
+ Dangling `.deb` files left by the `delete` command (or uploading new versions) can be removed using the `clean` command:
160
+
161
+ ```console
162
+ $ deb-s3 clean --bucket my-bucket
163
+ >> Retrieving existing manifests
164
+ >> Searching for unreferenced packages
165
+ -- pool/m/my/my-deb-package-1.0.0_amd64.deb
166
+ ```
167
+
168
+ ```
169
+ Usage:
170
+ deb-s3 clean
171
+
172
+ Options:
173
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
174
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
175
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
176
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
177
+ [--suite=SUITE] # The suite to use in the repository Release file.
178
+ -c, [--codename=CODENAME] # The codename of the APT repository.
179
+ # Default: stable
180
+ -m, [--component=COMPONENT] # The component of the APT repository.
181
+ # Default: main
182
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
183
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
184
+ [--session-token=SESSION_TOKEN] # The (optional) session token for connecting to S3.
185
+ [--endpoint=ENDPOINT] # The URL endpoint to the S3 API.
186
+ [--s3-region=S3_REGION] # The region for connecting to S3.
187
+ # Default: us-east-1
188
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
189
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
190
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
191
+ # Default: public
192
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
193
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
194
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
195
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
196
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
197
+
198
+ Delete packages from the pool which are no longer referenced
199
+ ```
200
+
201
+ You can also verify an existing APT repository on S3 using the `verify` command:
202
+
203
+ ```console
204
+ deb-s3 verify -b my-bucket
205
+ >> Retrieving existing manifests
206
+ >> Checking for missing packages in: stable/main i386
207
+ >> Checking for missing packages in: stable/main amd64
208
+ >> Checking for missing packages in: stable/main all
209
+ ```
210
+
211
+ ```
212
+ Usage:
213
+ deb-s3 verify
214
+
215
+ Options:
216
+ -f, [--fix-manifests], [--no-fix-manifests] # Whether to fix problems in manifests when verifying.
217
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
218
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
219
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
220
+ [--suite=SUITE] # The suite to use in the repository Release file.
221
+ -c, [--codename=CODENAME] # The codename of the APT repository.
222
+ # Default: stable
223
+ -m, [--component=COMPONENT] # The component of the APT repository.
224
+ # Default: main
225
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
226
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
227
+ [--s3-region=S3_REGION] # The region for connecting to S3.
228
+ # Default: us-east-1
229
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
230
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
231
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
232
+ # Default: public
233
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
234
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
235
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
236
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
237
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
238
+
239
+ Verifies that the files in the package manifests exist
240
+ ```
241
+
242
+ #### Example S3 IAM Policy
243
+
244
+ ```
245
+ {
246
+ "Version": "2012-10-17",
247
+ "Statement": [
248
+ {
249
+ "Effect": "Allow",
250
+ "Action": [
251
+ "s3:ListBucket"
252
+ ],
253
+ "Resource": [
254
+ "arn:aws:s3:::BUCKETNAME",
255
+ ]
256
+ },
257
+ {
258
+ "Effect": "Allow",
259
+ "Action": [
260
+ "s3:PutObject",
261
+ "s3:GetObject",
262
+ "s3:DeleteObject",
263
+ "s3:DeleteObjectVersion",
264
+ "s3:GetObjectAcl",
265
+ "s3:GetObjectTagging",
266
+ "s3:GetObjectTorrent",
267
+ "s3:GetObjectVersion",
268
+ "s3:GetObjectVersionAcl",
269
+ "s3:GetObjectVersionTagging",
270
+ "s3:GetObjectVersionTorrent",
271
+ "s3:PutObjectAcl",
272
+ "s3:PutObjectTagging",
273
+ "s3:PutObjectVersionAcl",
274
+ "s3:PutObjectVersionTagging",
275
+ "s3:ReplicateObject",
276
+ "s3:RestoreObject"
277
+ ],
278
+ "Resource": [
279
+ "arn:aws:s3:::BUCKETNAME/*"
280
+ ]
281
+ }
282
+ ]
283
+ }
284
+ ```
data/bin/deb-s3 ADDED
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'pathname'
4
+ $:.unshift File.join(Pathname.new(__FILE__).realpath,'../../lib')
5
+
6
+ require 'rubygems'
7
+ require 'deb/s3/cli'
8
+
9
+ Deb::S3::CLI.start
10
+