deb-s3-lock-fix 0.11.8.fix0.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (13) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +294 -0
  3. data/bin/deb-s3 +10 -0
  4. data/lib/deb/s3/cli.rb +767 -0
  5. data/lib/deb/s3/lock.rb +96 -0
  6. data/lib/deb/s3/manifest.rb +144 -0
  7. data/lib/deb/s3/package.rb +309 -0
  8. data/lib/deb/s3/release.rb +161 -0
  9. data/lib/deb/s3/templates/package.erb +66 -0
  10. data/lib/deb/s3/templates/release.erb +20 -0
  11. data/lib/deb/s3/utils.rb +115 -0
  12. data/lib/deb/s3.rb +6 -0
  13. metadata +125 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 5b7c555d0e33412b90fb17d46b473f97d414e504e96ced3726e5cdbaa0c80c58
4
+ data.tar.gz: ed1697addac22433d6830beef8a84cf7a03c85a160004729fc75fb66a210f162
5
+ SHA512:
6
+ metadata.gz: 2ad9c7570206b4f39b777667e24e3f004f816b946a55048bcf8b8f84ad24f4cccb7e81a034d25dd5996f47e9ccc351f966338df37f45a838e073688bde56434c
7
+ data.tar.gz: 1fce1fe773fb1397f81bc513b32d3e9f7b39398bf1a0e2e7dd8183a6f9876131ef919cb9353361be9e791fdbb7d3edb9dc336bd04e05a680c1799ec968b3a5f6
data/README.md ADDED
@@ -0,0 +1,294 @@
1
+ # deb-s3-lock-fix
2
+
3
+ [![Build Status](https://travis-ci.org/deb-s3/deb-s3.svg?branch=master)](https://travis-ci.org/deb-s3/deb-s3)
4
+
5
+ **This repository is a fork of [deb-s3](https://github.com/deb-s3/deb-s3).**
6
+
7
+ Note: The locking mechanism in the original deb-s3 library does not prevent race conditions.
8
+ It relies on S3 for distributed locking, which will not work consistently due to S3's eventual
9
+ consistency model. This fork uses DynamoDB for distributed locking, since it ensures atomic
10
+ conditional put operations on the lock.
11
+
12
+ To use this library create a DynamoDB table and export the following three env variables:
13
+ `DEB_S3_ACCESS_KEY_ID`
14
+ `DEB_S3_SECRET_ACCESS_KEY`
15
+ `AWS_BUILDERS_REGION`
16
+
17
+ `deb-s3` is a simple utility to make creating and managing APT repositories on
18
+ S3.
19
+
20
+ Most existing guides on using S3 to host an APT repository have you
21
+ using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
22
+ the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
23
+ sync the files to S3.
24
+
25
+ The annoying thing about this process is it requires you to maintain a local
26
+ copy of the file tree for regenerating and syncing the next time. Personally,
27
+ my process is to use one-off virtual machines with
28
+ [Vagrant](http://vagrantup.com), script out the build process, and then would
29
+ prefer to just upload the final `.deb` from my Mac.
30
+
31
+ With `deb-s3`, there is no need for this. `deb-s3` features:
32
+
33
+ * Downloads the existing package manifest and parses it.
34
+ * Updates it with the new package, replacing the existing entry if already
35
+ there or adding a new one if not.
36
+ * Uploads the package itself, the Packages manifest, and the Packages.gz
37
+ manifest. It will skip the uploading if the package is already there.
38
+ * Updates the Release file with the new hashes and file sizes.
39
+
40
+ ## Getting Started
41
+
42
+ Install the package via gem
43
+
44
+ ```console
45
+ $ gem install deb-s3
46
+ ```
47
+
48
+ or via APT (Debian 11 "Bullseye" and Ubuntu 22.04 "Jammy Jellyfish" and newer):
49
+
50
+ ```console
51
+ # Add repository key
52
+ $ sudo wget -O /etc/apt/trusted.gpg.d/deb-s3-archive-keyring.gpg https://raw.githubusercontent.com/deb-s3/deb-s3/master/deb-s3-archive-keyring.gpg
53
+
54
+ # Add repository
55
+ $ echo "deb http://deb-s3-repo.s3.us-east-2.amazonaws.com/$(lsb_release -is | tr A-Z a-z)/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list > /dev/null
56
+
57
+ # Install package
58
+ $ sudo apt-get update
59
+ $ sudo apt-get install deb-s3
60
+ ```
61
+
62
+ To run the code directly, just check out the repo and run bundler to ensure
63
+ all dependencies are installed:
64
+
65
+ ```console
66
+ $ git clone https://github.com/deb-s3/deb-s3.git
67
+ $ cd deb-s3
68
+ $ bundle install
69
+ ```
70
+
71
+ Now to upload a package, simply use:
72
+
73
+ ```console
74
+ $ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
75
+ >> Examining package file my-deb-package-1.0.0_amd64.deb
76
+ >> Retrieving existing package manifest
77
+ >> Uploading package and new manifests to S3
78
+ -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
79
+ -- Transferring dists/stable/main/binary-amd64/Packages
80
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
81
+ -- Transferring dists/stable/Release
82
+ >> Update complete.
83
+ ```
84
+
85
+ ```
86
+ Usage:
87
+ deb-s3 upload FILES
88
+
89
+ Options:
90
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
91
+ -p, [--preserve-versions], [--no-preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
92
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
93
+ [--fail-if-exists], [--no-fail-if-exists] # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
94
+ [--skip-package-upload], [--no-skip-package-upload] # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
95
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
96
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
97
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
98
+ [--suite=SUITE] # The suite to use in the repository Release file.
99
+ -c, [--codename=CODENAME] # The codename of the APT repository.
100
+ # Default: stable
101
+ -m, [--component=COMPONENT] # The component of the APT repository.
102
+ # Default: main
103
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
104
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
105
+ [--s3-region=S3_REGION] # The region for connecting to S3.
106
+ # Default: us-east-1
107
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
108
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
109
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
110
+ # Default: public
111
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
112
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
113
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
114
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
115
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
116
+
117
+ Uploads the given files to a S3 bucket as an APT repository.
118
+ ```
119
+
120
+ You can also delete packages from the APT repository. Please keep in mind that
121
+ this does NOT delete the .deb file itself (the `clean` command does that), it
122
+ only removes it from the list of packages in the specified component, codename
123
+ and architecture.
124
+
125
+ Now to delete the package:
126
+ ```console
127
+ $ deb-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
128
+ >> Retrieving existing manifests
129
+ -- Deleting my-deb-package version 1.0.0
130
+ >> Uploading new manifests to S3
131
+ -- Transferring dists/stable/main/binary-amd64/Packages
132
+ -- Transferring dists/stable/main/binary-amd64/Packages.gz
133
+ -- Transferring dists/stable/Release
134
+ >> Update complete.
135
+ ```
136
+
137
+ ```
138
+ Usage:
139
+ deb-s3 delete PACKAGE
140
+
141
+ Options:
142
+ -a, [--arch=ARCH] # The architecture of the package in the APT repository.
143
+ [--versions=one two three] # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
144
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
145
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
146
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
147
+ [--suite=SUITE] # The suite to use in the repository Release file.
148
+ -c, [--codename=CODENAME] # The codename of the APT repository.
149
+ # Default: stable
150
+ -m, [--component=COMPONENT] # The component of the APT repository.
151
+ # Default: main
152
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
153
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
154
+ [--s3-region=S3_REGION] # The region for connecting to S3.
155
+ # Default: us-east-1
156
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
157
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
158
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
159
+ # Default: public
160
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
161
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
162
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
163
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
164
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
165
+
166
+ Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
167
+ ```
168
+
169
+ Dangling `.deb` files left by the `delete` command (or uploading new versions) can be removed using the `clean` command:
170
+
171
+ ```console
172
+ $ deb-s3 clean --bucket my-bucket
173
+ >> Retrieving existing manifests
174
+ >> Searching for unreferenced packages
175
+ -- pool/m/my/my-deb-package-1.0.0_amd64.deb
176
+ ```
177
+
178
+ ```
179
+ Usage:
180
+ deb-s3 clean
181
+
182
+ Options:
183
+ -l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
184
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
185
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
186
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
187
+ [--suite=SUITE] # The suite to use in the repository Release file.
188
+ -c, [--codename=CODENAME] # The codename of the APT repository.
189
+ # Default: stable
190
+ -m, [--component=COMPONENT] # The component of the APT repository.
191
+ # Default: main
192
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
193
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
194
+ [--session-token=SESSION_TOKEN] # The (optional) session token for connecting to S3.
195
+ [--endpoint=ENDPOINT] # The URL endpoint to the S3 API.
196
+ [--s3-region=S3_REGION] # The region for connecting to S3.
197
+ # Default: us-east-1
198
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
199
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
200
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
201
+ # Default: public
202
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
203
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
204
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
205
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
206
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
207
+
208
+ Delete packages from the pool which are no longer referenced
209
+ ```
210
+
211
+ You can also verify an existing APT repository on S3 using the `verify` command:
212
+
213
+ ```console
214
+ deb-s3 verify -b my-bucket
215
+ >> Retrieving existing manifests
216
+ >> Checking for missing packages in: stable/main i386
217
+ >> Checking for missing packages in: stable/main amd64
218
+ >> Checking for missing packages in: stable/main all
219
+ ```
220
+
221
+ ```
222
+ Usage:
223
+ deb-s3 verify
224
+
225
+ Options:
226
+ -f, [--fix-manifests], [--no-fix-manifests] # Whether to fix problems in manifests when verifying.
227
+ -b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
228
+ [--prefix=PREFIX] # The path prefix to use when storing on S3.
229
+ -o, [--origin=ORIGIN] # The origin to use in the repository Release file.
230
+ [--suite=SUITE] # The suite to use in the repository Release file.
231
+ -c, [--codename=CODENAME] # The codename of the APT repository.
232
+ # Default: stable
233
+ -m, [--component=COMPONENT] # The component of the APT repository.
234
+ # Default: main
235
+ [--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
236
+ [--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
237
+ [--s3-region=S3_REGION] # The region for connecting to S3.
238
+ # Default: us-east-1
239
+ [--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
240
+ [--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
241
+ -v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
242
+ # Default: public
243
+ [--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
244
+ [--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
245
+ -e, [--encryption], [--no-encryption] # Use S3 server side encryption.
246
+ -q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
247
+ -C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
248
+
249
+ Verifies that the files in the package manifests exist
250
+ ```
251
+
252
+ #### Example S3 IAM Policy
253
+
254
+ ```
255
+ {
256
+ "Version": "2012-10-17",
257
+ "Statement": [
258
+ {
259
+ "Effect": "Allow",
260
+ "Action": [
261
+ "s3:ListBucket"
262
+ ],
263
+ "Resource": [
264
+ "arn:aws:s3:::BUCKETNAME",
265
+ ]
266
+ },
267
+ {
268
+ "Effect": "Allow",
269
+ "Action": [
270
+ "s3:PutObject",
271
+ "s3:GetObject",
272
+ "s3:DeleteObject",
273
+ "s3:DeleteObjectVersion",
274
+ "s3:GetObjectAcl",
275
+ "s3:GetObjectTagging",
276
+ "s3:GetObjectTorrent",
277
+ "s3:GetObjectVersion",
278
+ "s3:GetObjectVersionAcl",
279
+ "s3:GetObjectVersionTagging",
280
+ "s3:GetObjectVersionTorrent",
281
+ "s3:PutObjectAcl",
282
+ "s3:PutObjectTagging",
283
+ "s3:PutObjectVersionAcl",
284
+ "s3:PutObjectVersionTagging",
285
+ "s3:ReplicateObject",
286
+ "s3:RestoreObject"
287
+ ],
288
+ "Resource": [
289
+ "arn:aws:s3:::BUCKETNAME/*"
290
+ ]
291
+ }
292
+ ]
293
+ }
294
+ ```
data/bin/deb-s3 ADDED
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'pathname'
4
+ $:.unshift File.join(Pathname.new(__FILE__).realpath,'../../lib')
5
+
6
+ require 'rubygems'
7
+ require 'deb/s3/cli'
8
+
9
+ Deb::S3::CLI.start
10
+