deb-s3-lock-fix 0.11.8.fix0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5b7c555d0e33412b90fb17d46b473f97d414e504e96ced3726e5cdbaa0c80c58
+  data.tar.gz: ed1697addac22433d6830beef8a84cf7a03c85a160004729fc75fb66a210f162
+SHA512:
+  metadata.gz: 2ad9c7570206b4f39b777667e24e3f004f816b946a55048bcf8b8f84ad24f4cccb7e81a034d25dd5996f47e9ccc351f966338df37f45a838e073688bde56434c
+  data.tar.gz: 1fce1fe773fb1397f81bc513b32d3e9f7b39398bf1a0e2e7dd8183a6f9876131ef919cb9353361be9e791fdbb7d3edb9dc336bd04e05a680c1799ec968b3a5f6

data/README.md

@@ -0,0 +1,294 @@
+# deb-s3-lock-fix
+
+[![Build Status](https://travis-ci.org/deb-s3/deb-s3.svg?branch=master)](https://travis-ci.org/deb-s3/deb-s3)
+
+**This repository is a fork of [deb-s3](https://github.com/deb-s3/deb-s3).**
+
+Note: The locking mechanism in the original deb-s3 library does not prevent race conditions.
+It relies on S3 for distributed locking, which will not work consistently due to S3's eventual
+consistency model. This fork uses DynamoDB for distributed locking, since it ensures atomic
+conditional put operations on the lock.
+
+To use this library create a DynamoDB table and export the following three env variables:
+`DEB_S3_ACCESS_KEY_ID`
+`DEB_S3_SECRET_ACCESS_KEY`
+`AWS_BUILDERS_REGION`
+
+`deb-s3` is a simple utility to make creating and managing APT repositories on
+S3.
+
+Most existing guides on using S3 to host an APT repository have you
+using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
+the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
+sync the files to S3.
+
+The annoying thing about this process is it requires you to maintain a local
+copy of the file tree for regenerating and syncing the next time. Personally,
+my process is to use one-off virtual machines with
+[Vagrant](http://vagrantup.com), script out the build process, and then would
+prefer to just upload the final `.deb` from my Mac.
+
+With `deb-s3`, there is no need for this. `deb-s3` features:
+
+* Downloads the existing package manifest and parses it.
+* Updates it with the new package, replacing the existing entry if already
+  there or adding a new one if not.
+* Uploads the package itself, the Packages manifest, and the Packages.gz
+  manifest. It will skip the uploading if the package is already there.
+* Updates the Release file with the new hashes and file sizes.
+
+## Getting Started
+
+Install the package via gem
+
+```console
+$ gem install deb-s3
+```
+
+or via APT (Debian 11 "Bullseye" and Ubuntu 22.04 "Jammy Jellyfish" and newer):
+
+```console
+# Add repository key
+$ sudo wget -O /etc/apt/trusted.gpg.d/deb-s3-archive-keyring.gpg https://raw.githubusercontent.com/deb-s3/deb-s3/master/deb-s3-archive-keyring.gpg
+
+# Add repository
+$ echo "deb http://deb-s3-repo.s3.us-east-2.amazonaws.com/$(lsb_release -is | tr A-Z a-z)/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list > /dev/null
+
+# Install package
+$ sudo apt-get update
+$ sudo apt-get install deb-s3
+```
+
+To run the code directly, just check out the repo and run bundler to ensure
+all dependencies are installed:
+
+```console
+$ git clone https://github.com/deb-s3/deb-s3.git
+$ cd deb-s3
+$ bundle install
+```
+
+Now to upload a package, simply use:
+
+```console
+$ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
+>> Examining package file my-deb-package-1.0.0_amd64.deb
+>> Retrieving existing package manifest
+>> Uploading package and new manifests to S3
+   -- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
+   -- Transferring dists/stable/main/binary-amd64/Packages
+   -- Transferring dists/stable/main/binary-amd64/Packages.gz
+   -- Transferring dists/stable/Release
+>> Update complete.
+```
+
+```
+Usage:
+  deb-s3 upload FILES
+
+Options:
+  -a, [--arch=ARCH]                                        # The architecture of the package in the APT repository.
+  -p, [--preserve-versions], [--no-preserve-versions]      # Whether to preserve other versions of a package in the repository when uploading one.
+  -l, [--lock], [--no-lock]                                # Whether to check for an existing lock on the repository to prevent simultaneous updates
+      [--fail-if-exists], [--no-fail-if-exists]            # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
+      [--skip-package-upload], [--no-skip-package-upload]  # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
+  -b, [--bucket=BUCKET]                                    # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                                    # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                                    # The origin to use in the repository Release file.
+      [--suite=SUITE]                                      # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                                # The codename of the APT repository.
+                                                           # Default: stable
+  -m, [--component=COMPONENT]                              # The component of the APT repository.
+                                                           # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                      # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]              # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                              # The region for connecting to S3.
+                                                           # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]        # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                              # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                            # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                           # Default: public
+      [--sign=SIGN]                                        # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                          # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]                    # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                              # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                      # Add cache-control headers to S3 objects.
+
+Uploads the given files to a S3 bucket as an APT repository.
+```
+
+You can also delete packages from the APT repository. Please keep in mind that
+this does NOT delete the .deb file itself (the `clean` command does that), it
+only removes it from the list of packages in the specified component, codename
+and architecture.
+
+Now to delete the package:
+```console
+$ deb-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
+>> Retrieving existing manifests
+   -- Deleting my-deb-package version 1.0.0
+>> Uploading new manifests to S3
+   -- Transferring dists/stable/main/binary-amd64/Packages
+   -- Transferring dists/stable/main/binary-amd64/Packages.gz
+   -- Transferring dists/stable/Release
+>> Update complete.
+```
+
+```
+Usage:
+  deb-s3 delete PACKAGE
+
+Options:
+  -a, [--arch=ARCH]                                  # The architecture of the package in the APT repository.
+      [--versions=one two three]                     # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
+  -b, [--bucket=BUCKET]                              # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                              # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                              # The origin to use in the repository Release file.
+      [--suite=SUITE]                                # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                          # The codename of the APT repository.
+                                                     # Default: stable
+  -m, [--component=COMPONENT]                        # The component of the APT repository.
+                                                     # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]        # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                        # The region for connecting to S3.
+                                                     # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]  # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                        # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                      # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                     # Default: public
+      [--sign=SIGN]                                  # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                    # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]              # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                        # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                # Add cache-control headers to S3 objects.
+
+Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
+```
+
+Dangling `.deb` files left by the `delete` command (or uploading new versions) can be removed using the `clean` command:
+
+```console
+$ deb-s3 clean --bucket my-bucket
+>> Retrieving existing manifests
+>> Searching for unreferenced packages
+   -- pool/m/my/my-deb-package-1.0.0_amd64.deb
+```
+
+```
+Usage:
+  deb-s3 clean
+
+Options:
+  -l, [--lock], [--no-lock]                          # Whether to check for an existing lock on the repository to prevent simultaneous updates
+  -b, [--bucket=BUCKET]                              # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                              # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                              # The origin to use in the repository Release file.
+      [--suite=SUITE]                                # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                          # The codename of the APT repository.
+                                                     # Default: stable
+  -m, [--component=COMPONENT]                        # The component of the APT repository.
+                                                     # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]        # The secret key for connecting to S3.
+      [--session-token=SESSION_TOKEN]                # The (optional) session token for connecting to S3.
+      [--endpoint=ENDPOINT]                          # The URL endpoint to the S3 API.
+      [--s3-region=S3_REGION]                        # The region for connecting to S3.
+                                                     # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]  # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                        # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                      # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                     # Default: public
+      [--sign=SIGN]                                  # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                    # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]              # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                        # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                # Add cache-control headers to S3 objects.
+
+Delete packages from the pool which are no longer referenced
+```
+
+You can also verify an existing APT repository on S3 using the `verify` command:
+
+```console
+deb-s3 verify -b my-bucket
+>> Retrieving existing manifests
+>> Checking for missing packages in: stable/main i386
+>> Checking for missing packages in: stable/main amd64
+>> Checking for missing packages in: stable/main all
+```
+
+```
+Usage:
+  deb-s3 verify
+
+Options:
+  -f, [--fix-manifests], [--no-fix-manifests]        # Whether to fix problems in manifests when verifying.
+  -b, [--bucket=BUCKET]                              # The name of the S3 bucket to upload to.
+      [--prefix=PREFIX]                              # The path prefix to use when storing on S3.
+  -o, [--origin=ORIGIN]                              # The origin to use in the repository Release file.
+      [--suite=SUITE]                                # The suite to use in the repository Release file.
+  -c, [--codename=CODENAME]                          # The codename of the APT repository.
+                                                     # Default: stable
+  -m, [--component=COMPONENT]                        # The component of the APT repository.
+                                                     # Default: main
+      [--access-key-id=ACCESS_KEY_ID]                # The access key for connecting to S3.
+      [--secret-access-key=SECRET_ACCESS_KEY]        # The secret key for connecting to S3.
+      [--s3-region=S3_REGION]                        # The region for connecting to S3.
+                                                     # Default: us-east-1
+      [--force-path-style], [--no-force-path-style]  # Use S3 path style instead of subdomains.
+      [--proxy-uri=PROXY_URI]                        # The URI of the proxy to send service requests through.
+  -v, [--visibility=VISIBILITY]                      # The access policy for the uploaded files. Can be public, private, or authenticated.
+                                                     # Default: public
+      [--sign=SIGN]                                  # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
+      [--gpg-options=GPG_OPTIONS]                    # Additional command line options to pass to GPG when signing.
+  -e, [--encryption], [--no-encryption]              # Use S3 server side encryption.
+  -q, [--quiet], [--no-quiet]                        # Doesn't output information, just returns status appropriately.
+  -C, [--cache-control=CACHE_CONTROL]                # Add cache-control headers to S3 objects.
+
+Verifies that the files in the package manifests exist
+```
+
+#### Example S3 IAM Policy
+
+```
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "s3:ListBucket"
+            ],
+            "Resource": [
+                "arn:aws:s3:::BUCKETNAME",
+            ]
+        },
+        {
+            "Effect": "Allow",
+            "Action": [
+                "s3:PutObject",
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:DeleteObjectVersion",
+                "s3:GetObjectAcl",
+                "s3:GetObjectTagging",
+                "s3:GetObjectTorrent",
+                "s3:GetObjectVersion",
+                "s3:GetObjectVersionAcl",
+                "s3:GetObjectVersionTagging",
+                "s3:GetObjectVersionTorrent",
+                "s3:PutObjectAcl",
+                "s3:PutObjectTagging",
+                "s3:PutObjectVersionAcl",
+                "s3:PutObjectVersionTagging",
+                "s3:ReplicateObject",
+                "s3:RestoreObject"
+            ],
+            "Resource": [
+                "arn:aws:s3:::BUCKETNAME/*"
+            ]
+        }
+    ]
+}
+```

data/bin/deb-s3

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+require 'pathname'
+$:.unshift File.join(Pathname.new(__FILE__).realpath,'../../lib')
+
+require 'rubygems'
+require 'deb/s3/cli'
+
+Deb::S3::CLI.start
+