cloudflare-r2-cli 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +61 -2
- data/README.md +47 -4
- data/docs/ARCHITECTURE.md +65 -10
- data/docs/FEATURES.md +81 -3
- data/docs/SECURITY.md +4 -0
- data/lib/r2/cli.rb +111 -2
- data/lib/r2/content_type.rb +70 -0
- data/lib/r2/errors.rb +9 -0
- data/lib/r2/retry.rb +105 -0
- data/lib/r2/storage.rb +125 -19
- data/lib/r2/version.rb +1 -1
- data/lib/r2.rb +2 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3195ca7161952b3ef9a5e7cc6e6d1afb4af82c631364a6786f4b1d4aeaace272
|
|
4
|
+
data.tar.gz: ca9017f232ee7a6a8a15048dfc6dd895e490f9ac18eb442dfb7e518deb89830d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7166d020132b6cabf2835a41a8cca93531eb58e4b65102ca8f9d8b9a16b9055eb460578f346a4b38cf15e757d3f5581e14da22eaf2b3b893497deabfb4de53d8
|
|
7
|
+
data.tar.gz: 331b10be069ca8262aceb8ebffebddaedc99c4cf806a4ff35eae5679eeed7304ef7b64079f7d4e66d4c4370767d6859f8e9cce72b776cf1f745bc38c6c88ff78
|
data/CHANGELOG.md
CHANGED
|
@@ -3,7 +3,65 @@
|
|
|
3
3
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
4
4
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
5
5
|
|
|
6
|
-
## [
|
|
6
|
+
## [1.2.0] - 2026-09-21
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
|
|
10
|
+
* `r2 exists` — checks whether an object exists in the configured bucket,
|
|
11
|
+
exiting with status code `1` when it does not.
|
|
12
|
+
* `r2 list --prefix` — lists only the objects whose keys start with the given
|
|
13
|
+
prefix.
|
|
14
|
+
* `r2 delete --force` — deletes an object without asking for confirmation.
|
|
15
|
+
* Automatic `Content-Type` definition on upload, based on the object key.
|
|
16
|
+
* Automatic retries with exponential backoff for transient network failures
|
|
17
|
+
in every storage operation.
|
|
18
|
+
* `R2::ContentType` — maps the object key extension to a content type.
|
|
19
|
+
* `R2::Retry` — retry policy with exponential backoff shared by the storage
|
|
20
|
+
operations.
|
|
21
|
+
* `R2::Errors::ConfirmationRequiredError` and `R2::Errors::AbortedError`.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
* `r2 delete` now asks for confirmation before deleting. In non-interactive
|
|
26
|
+
executions (scripts and pipelines) the deletion requires `--force`.
|
|
27
|
+
* `r2 list` now follows the pagination of the bucket, listing every object
|
|
28
|
+
instead of only the first page.
|
|
29
|
+
* `Storage#list` accepts a `prefix` and lists every page of the bucket.
|
|
30
|
+
* `Storage#upload` accepts an explicit `content_type` and rewinds the streamed
|
|
31
|
+
content before every attempt.
|
|
32
|
+
* `Storage#exists?` — checks the existence of an object using its metadata.
|
|
33
|
+
* Restructured the documentation to cover the new commands, options and
|
|
34
|
+
reliability behavior.
|
|
35
|
+
|
|
36
|
+
## [1.2.0] - 2026-09-21
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
|
|
40
|
+
* `r2 exists` — checks whether an object exists in the configured bucket,
|
|
41
|
+
exiting with status code `1` when it does not.
|
|
42
|
+
* `r2 list --prefix` — lists only the objects whose keys start with the given
|
|
43
|
+
prefix.
|
|
44
|
+
* `r2 delete --force` — deletes an object without asking for confirmation.
|
|
45
|
+
* Automatic `Content-Type` definition on upload, based on the object key.
|
|
46
|
+
* Automatic retries with exponential backoff for transient network failures
|
|
47
|
+
in every storage operation.
|
|
48
|
+
* `R2::ContentType` — maps the object key extension to a content type.
|
|
49
|
+
* `R2::Retry` — retry policy with exponential backoff shared by the storage
|
|
50
|
+
operations.
|
|
51
|
+
* `R2::Errors::ConfirmationRequiredError` and `R2::Errors::AbortedError`.
|
|
52
|
+
|
|
53
|
+
### Changed
|
|
54
|
+
|
|
55
|
+
* `r2 delete` now asks for confirmation before deleting. In non-interactive
|
|
56
|
+
executions (scripts and pipelines) the deletion requires `--force`.
|
|
57
|
+
* `r2 list` now follows the pagination of the bucket, listing every object
|
|
58
|
+
instead of only the first page.
|
|
59
|
+
* `Storage#list` accepts a `prefix` and lists every page of the bucket.
|
|
60
|
+
* `Storage#upload` accepts an explicit `content_type` and rewinds the streamed
|
|
61
|
+
content before every attempt.
|
|
62
|
+
* `Storage#exists?` — checks the existence of an object using its metadata.
|
|
63
|
+
* Restructured the documentation to cover the new commands, options and
|
|
64
|
+
reliability behavior.
|
|
7
65
|
|
|
8
66
|
## [1.1.0] - 2026-09-09
|
|
9
67
|
|
|
@@ -45,6 +103,7 @@ First stable release of `cloudflare-r2-cli`.
|
|
|
45
103
|
* Unit, integration and E2E test suites.
|
|
46
104
|
* Continuous integration via GitHub Actions.
|
|
47
105
|
|
|
48
|
-
[Unreleased]: https://github.com/rpzerosixcode/cloudflare-r2-cli/compare/v1.
|
|
106
|
+
[Unreleased]: https://github.com/rpzerosixcode/cloudflare-r2-cli/compare/v1.2.0...HEAD
|
|
107
|
+
[1.2.0]: https://github.com/rpzerosixcode/cloudflare-r2-cli/compare/v1.1.0...v1.2.0
|
|
49
108
|
[1.1.0]: https://github.com/rpzerosixcode/cloudflare-r2-cli/compare/v1.0.0...v1.1.0
|
|
50
109
|
[1.0.0]: https://github.com/rpzerosixcode/cloudflare-r2-cli/releases/tag/v1.0.0
|
data/README.md
CHANGED
|
@@ -12,7 +12,9 @@ Ruby CLI to manage objects on Cloudflare R2 from the terminal.
|
|
|
12
12
|
* [Download](#download)
|
|
13
13
|
* [Delete](#delete)
|
|
14
14
|
* [List](#list)
|
|
15
|
+
* [Exists](#exists)
|
|
15
16
|
* [Global Options](#global-options)
|
|
17
|
+
* [Reliability](#reliability)
|
|
16
18
|
* [Testing](#testing)
|
|
17
19
|
* [Changelog](#changelog)
|
|
18
20
|
* [License](#license)
|
|
@@ -37,7 +39,7 @@ $ git clone https://github.com/rpzerosixcode/cloudflare-r2-cli.git
|
|
|
37
39
|
$ cd cloudflare-r2-cli
|
|
38
40
|
$ bundle install
|
|
39
41
|
$ bundle exec rake build
|
|
40
|
-
$ gem install pkg/cloudflare-r2-cli-1.
|
|
42
|
+
$ gem install pkg/cloudflare-r2-cli-1.2.0.gem
|
|
41
43
|
```
|
|
42
44
|
|
|
43
45
|
## Configuration
|
|
@@ -68,7 +70,7 @@ $ r2 upload ./images/photo.png
|
|
|
68
70
|
$ r2 upload ./images/photo.png --key uploads/photo.png
|
|
69
71
|
```
|
|
70
72
|
|
|
71
|
-
By default, the object key in the bucket is the name of the given file. Use `--key` to store the object under a custom key. On success, a confirmation message is displayed.
|
|
73
|
+
By default, the object key in the bucket is the name of the given file. Use `--key` to store the object under a custom key. The `Content-Type` of the object is defined automatically from its key extension (`application/octet-stream` when the extension is unknown). On success, a confirmation message is displayed.
|
|
72
74
|
|
|
73
75
|
### Download
|
|
74
76
|
|
|
@@ -87,9 +89,17 @@ Deletes a file stored in the configured bucket:
|
|
|
87
89
|
|
|
88
90
|
```console
|
|
89
91
|
$ r2 delete image.jpg
|
|
92
|
+
$ r2 delete image.jpg --force
|
|
90
93
|
```
|
|
91
94
|
|
|
92
|
-
|
|
95
|
+
The deletion is confirmed before the request is sent:
|
|
96
|
+
|
|
97
|
+
```console
|
|
98
|
+
$ r2 delete image.jpg
|
|
99
|
+
Delete "image.jpg" from the bucket? [y/N]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
The operation only proceeds with an affirmative answer (`y` or `yes`). Use `--force` to delete without confirmation, which is required in non-interactive executions, such as scripts and pipelines, where the confirmation cannot be requested. On success, a confirmation message is displayed.
|
|
93
103
|
|
|
94
104
|
### List
|
|
95
105
|
|
|
@@ -97,6 +107,26 @@ Lists the files stored in the configured bucket:
|
|
|
97
107
|
|
|
98
108
|
```console
|
|
99
109
|
$ r2 list
|
|
110
|
+
$ r2 list --prefix uploads/
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Every object is listed, following the pagination of the bucket. Use `--prefix` to list only the objects whose keys start with the given prefix.
|
|
114
|
+
|
|
115
|
+
### Exists
|
|
116
|
+
|
|
117
|
+
Checks whether an object exists in the configured bucket:
|
|
118
|
+
|
|
119
|
+
```console
|
|
120
|
+
$ r2 exists image.jpg
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
The exit status is `0` when the object exists and `1` when it does not, which allows the command to be used in scripts:
|
|
124
|
+
|
|
125
|
+
```console
|
|
126
|
+
$ r2 exists image.jpg
|
|
127
|
+
Object exists: image.jpg
|
|
128
|
+
$ r2 exists missing.jpg
|
|
129
|
+
Object not found: missing.jpg
|
|
100
130
|
```
|
|
101
131
|
|
|
102
132
|
### Global Options
|
|
@@ -107,11 +137,24 @@ Every command accepts the global `--verbose` flag, which writes detailed diagnos
|
|
|
107
137
|
$ r2 list --verbose
|
|
108
138
|
$ r2 upload image.jpg --verbose
|
|
109
139
|
$ r2 download image.jpg --verbose
|
|
110
|
-
$ r2 delete image.jpg --verbose
|
|
140
|
+
$ r2 delete image.jpg --force --verbose
|
|
141
|
+
$ r2 exists image.jpg --verbose
|
|
111
142
|
```
|
|
112
143
|
|
|
113
144
|
On any error, the CLI displays the corresponding message on the error output and exits with status code `1`.
|
|
114
145
|
|
|
146
|
+
## Reliability
|
|
147
|
+
|
|
148
|
+
### Automatic retries
|
|
149
|
+
|
|
150
|
+
Requests that fail due to transient network failures are retried automatically with exponential backoff: 3 attempts in total, waiting 0.5s and then 1s between them, capped at 5s. Failures that are not transient, such as missing credentials or an invalid bucket, are reported immediately.
|
|
151
|
+
|
|
152
|
+
The retries are applied to every storage operation and restart from the beginning of the data: uploaded content is rewound and downloaded content is written from the start again, so a retried request never uploads an empty body or leaves a partially written file.
|
|
153
|
+
|
|
154
|
+
### Pagination
|
|
155
|
+
|
|
156
|
+
`r2 list` follows the pagination of the bucket, requesting every page until the last one, so all objects are listed regardless of the amount of keys in the bucket.
|
|
157
|
+
|
|
115
158
|
## Testing
|
|
116
159
|
|
|
117
160
|
Run the full suite (unit, integration and E2E):
|
data/docs/ARCHITECTURE.md
CHANGED
|
@@ -15,6 +15,8 @@ Built with `thor`. Acts as a minimal orchestrator:
|
|
|
15
15
|
|
|
16
16
|
The CLI contains no business rules and does not access files or `ENV` directly.
|
|
17
17
|
|
|
18
|
+
Commands that delete data ask for confirmation before delegating the operation. The confirmation is requested only when the standard input is a terminal, and `--force` skips it, so non-interactive executions must opt out explicitly.
|
|
19
|
+
|
|
18
20
|
### Configuration
|
|
19
21
|
|
|
20
22
|
`lib/r2/configuration.rb`
|
|
@@ -36,10 +38,11 @@ Wraps `aws-sdk-s3` to communicate with the S3-compatible Cloudflare R2 endpoint.
|
|
|
36
38
|
|
|
37
39
|
Responsible only for:
|
|
38
40
|
|
|
39
|
-
* Uploading objects.
|
|
41
|
+
* Uploading objects, defining the content type from the object key.
|
|
40
42
|
* Downloading objects.
|
|
41
43
|
* Deleting objects.
|
|
42
|
-
* Listing objects.
|
|
44
|
+
* Listing objects, following the pagination of the bucket and filtering by prefix.
|
|
45
|
+
* Checking whether an object exists.
|
|
43
46
|
|
|
44
47
|
`Storage` does not:
|
|
45
48
|
|
|
@@ -50,6 +53,31 @@ Responsible only for:
|
|
|
50
53
|
|
|
51
54
|
The bucket is provided by `Configuration` rather than passed to individual operations.
|
|
52
55
|
|
|
56
|
+
Every request is wrapped in the retry support (see `Retry`), so transient network failures are retried before the failure reaches the caller.
|
|
57
|
+
|
|
58
|
+
### Content Type
|
|
59
|
+
|
|
60
|
+
`lib/r2/content_type.rb`
|
|
61
|
+
|
|
62
|
+
Defines the content type of an object from the extension of its key:
|
|
63
|
+
|
|
64
|
+
* `ContentType.for(key)` returns the mapped type, or `application/octet-stream` when the extension is not mapped.
|
|
65
|
+
* The comparison is case insensitive, and keys inside directories are supported.
|
|
66
|
+
|
|
67
|
+
`Storage#upload` uses this value automatically and accepts an explicit `content_type` when the caller needs to override it.
|
|
68
|
+
|
|
69
|
+
### Retry
|
|
70
|
+
|
|
71
|
+
`lib/r2/retry.rb`
|
|
72
|
+
|
|
73
|
+
Concentrates the automatic retries of the application:
|
|
74
|
+
|
|
75
|
+
* `Policy` holds the retry settings: total attempts, base delay and maximum delay.
|
|
76
|
+
* `Retry.call` runs the block and retries only the error classes informed as transient, waiting between the attempts with exponential backoff.
|
|
77
|
+
* The waiting strategy and the logger are injected, keeping the behavior testable without real waiting.
|
|
78
|
+
|
|
79
|
+
`Storage` uses a shared policy for every operation and passes `Seahorse::Client::NetworkingError` as the transient error, which is also the error mapped to `NetworkError`.
|
|
80
|
+
|
|
53
81
|
### Errors
|
|
54
82
|
|
|
55
83
|
`lib/r2/errors.rb`
|
|
@@ -64,6 +92,11 @@ All errors inherit from `R2::Errors::Error`. Exceptions raised by `aws-sdk-s3` a
|
|
|
64
92
|
* `NetworkError`
|
|
65
93
|
* `StorageError`
|
|
66
94
|
|
|
95
|
+
Errors of the CLI are also part of the domain:
|
|
96
|
+
|
|
97
|
+
* `ConfirmationRequiredError` when a destructive operation cannot be confirmed in a non-interactive execution.
|
|
98
|
+
* `AbortedError` when the user does not confirm a destructive operation.
|
|
99
|
+
|
|
67
100
|
This prevents callers from depending on AWS SDK-specific exceptions.
|
|
68
101
|
|
|
69
102
|
The CLI rescues the base `Error`, prints the message to `stderr`, and exits with a non-zero status.
|
|
@@ -88,26 +121,36 @@ CLI
|
|
|
88
121
|
│
|
|
89
122
|
├── Logging ←── --verbose
|
|
90
123
|
│
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
124
|
+
├── $stdin ←── confirmation of destructive commands
|
|
125
|
+
│
|
|
126
|
+
└── Storage
|
|
127
|
+
│
|
|
128
|
+
├── ContentType ──→ Content-Type of the object
|
|
129
|
+
│
|
|
130
|
+
└── Retry ──→ aws-sdk-s3 ──→ Cloudflare R2
|
|
131
|
+
│
|
|
132
|
+
└── failure → R2::Errors::*
|
|
133
|
+
│
|
|
134
|
+
└── CLI → stderr + exit 1
|
|
96
135
|
```
|
|
97
136
|
|
|
137
|
+
Transient network failures are retried by `Retry` before the error mapping, so a brief instability does not interrupt the operation.
|
|
138
|
+
|
|
98
139
|
## Testing
|
|
99
140
|
|
|
100
141
|
### Unit Tests
|
|
101
142
|
|
|
102
143
|
`spec/unit/`
|
|
103
144
|
|
|
104
|
-
Tests individual components in isolation.
|
|
145
|
+
Tests individual components in isolation: CLI, configuration, content type, errors, logging, retry and storage.
|
|
105
146
|
|
|
106
147
|
### Integration Tests
|
|
107
148
|
|
|
108
149
|
`spec/integration/`
|
|
109
150
|
|
|
110
|
-
Tests the interaction between the CLI and `Storage` using a fake S3 client.
|
|
151
|
+
Tests the interaction between the CLI and `Storage` using a fake S3 client, which reproduces the bucket behavior, including the pagination of the listing and the metadata requests.
|
|
152
|
+
|
|
153
|
+
The confirmation prompts are validated with a controlled standard input, so the suite never waits for a real user and the result does not depend on whether the runner has a terminal attached.
|
|
111
154
|
|
|
112
155
|
### End-to-End Tests
|
|
113
156
|
|
|
@@ -121,7 +164,19 @@ These tests are skipped when the required credentials are unavailable.
|
|
|
121
164
|
|
|
122
165
|
### Direct Constructor Injection
|
|
123
166
|
|
|
124
|
-
No dependency-injection container is used. Dependencies are passed directly through constructors, keeping the architecture simple and explicit.
|
|
167
|
+
No dependency-injection container is used. Dependencies are passed directly through constructors, keeping the architecture simple and explicit. `Storage`, for instance, receives its logger, retry policy and waiting strategy, which makes the retries testable without real waiting.
|
|
168
|
+
|
|
169
|
+
### Deletion Confirmation
|
|
170
|
+
|
|
171
|
+
Destructive commands confirm the operation before sending it, since a deletion cannot be undone. The confirmation depends on an interactive input, so scripts and pipelines must use `--force`: the alternative, deleting silently when no terminal is attached, would make accidental deletions possible in automated executions.
|
|
172
|
+
|
|
173
|
+
### Retries with Backoff
|
|
174
|
+
|
|
175
|
+
Transient network failures are retried with exponential backoff by the application, on top of the retries already performed by `aws-sdk-s3`. Keeping the behavior in the project makes it explicit, configurable and covered by the tests, instead of being an undocumented side effect of the SDK.
|
|
176
|
+
|
|
177
|
+
### Automatic Content Type
|
|
178
|
+
|
|
179
|
+
The content type is derived from the object key instead of the local file, because the key is what defines the object in the bucket and can differ from the name of the uploaded file (`--key`). Unmapped extensions fall back to `application/octet-stream`, the default of S3-compatible services.
|
|
125
180
|
|
|
126
181
|
### R2 Region
|
|
127
182
|
|
data/docs/FEATURES.md
CHANGED
|
@@ -23,6 +23,7 @@ $ r2 upload ./images/photo.png --key uploads/photo.png
|
|
|
23
23
|
* Validates that the given file exists.
|
|
24
24
|
* Validates that the path is not a directory.
|
|
25
25
|
* Opens the file in binary read mode.
|
|
26
|
+
* Defines the `Content-Type` of the object from the extension of its key, falling back to `application/octet-stream` when the extension is not mapped.
|
|
26
27
|
* Uploads the content to the configured Cloudflare R2 bucket.
|
|
27
28
|
* Uses the file name as the object key in the bucket, or the custom key given via `--key`.
|
|
28
29
|
* Displays a success message after the upload.
|
|
@@ -62,23 +63,40 @@ Deletes a file stored in the configured Cloudflare R2 bucket.
|
|
|
62
63
|
**Usage:**
|
|
63
64
|
|
|
64
65
|
```console
|
|
65
|
-
$ r2 delete <file>
|
|
66
|
+
$ r2 delete <file> [--force]
|
|
66
67
|
```
|
|
67
68
|
|
|
68
|
-
**
|
|
69
|
+
**Examples:**
|
|
69
70
|
|
|
70
71
|
```console
|
|
71
72
|
$ r2 delete image.jpg
|
|
73
|
+
$ r2 delete image.jpg --force
|
|
72
74
|
```
|
|
73
75
|
|
|
74
76
|
**Behavior:**
|
|
75
77
|
|
|
76
78
|
* Receives the name of the file to delete.
|
|
79
|
+
* Asks for the confirmation of the deletion before sending the request, unless `--force` is given.
|
|
77
80
|
* Uses the file name as the object key.
|
|
78
81
|
* Requests the object deletion from Cloudflare R2.
|
|
79
82
|
* Considers the operation successful when the storage completes the request without errors.
|
|
80
83
|
* Displays a success message after the operation.
|
|
81
84
|
|
|
85
|
+
**Confirmation:**
|
|
86
|
+
|
|
87
|
+
The prompt is written to the standard output and the answer is read from the standard input:
|
|
88
|
+
|
|
89
|
+
```console
|
|
90
|
+
$ r2 delete image.jpg
|
|
91
|
+
Delete "image.jpg" from the bucket? [y/N]
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The deletion only proceeds with an affirmative answer (`y` or `yes`); any other answer, including an empty one, aborts the operation. In non-interactive executions, such as scripts and pipelines, the confirmation cannot be requested, so the CLI aborts with an explanatory message and status code `1`. Use `--force` to delete without confirmation:
|
|
95
|
+
|
|
96
|
+
```console
|
|
97
|
+
$ r2 delete image.jpg --force
|
|
98
|
+
```
|
|
99
|
+
|
|
82
100
|
The feature does not perform a follow-up query to check that the object no longer exists. The success confirmation is based on the result of the deletion operation provided by the storage layer.
|
|
83
101
|
|
|
84
102
|
On error, the corresponding message is displayed and the CLI exits with a non-zero status code.
|
|
@@ -89,18 +107,75 @@ Lists the files stored in the configured Cloudflare R2 bucket.
|
|
|
89
107
|
|
|
90
108
|
**Usage:**
|
|
91
109
|
|
|
110
|
+
```console
|
|
111
|
+
$ r2 list [--prefix <prefix>]
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
**Examples:**
|
|
115
|
+
|
|
92
116
|
```console
|
|
93
117
|
$ r2 list
|
|
118
|
+
$ r2 list --prefix uploads/
|
|
94
119
|
```
|
|
95
120
|
|
|
96
121
|
**Behavior:**
|
|
97
122
|
|
|
98
123
|
* Queries the objects stored in the configured bucket.
|
|
124
|
+
* Follows the pagination of the bucket, requesting every page until the last one, so all objects are returned.
|
|
125
|
+
* Lists only the objects whose keys start with the given prefix, when `--prefix` is used.
|
|
99
126
|
* Displays the files found.
|
|
100
|
-
* Returns all objects without pagination or control over the amount of returned objects.
|
|
101
127
|
|
|
102
128
|
On error, the corresponding message is displayed and the CLI exits with a non-zero status code.
|
|
103
129
|
|
|
130
|
+
## Exists
|
|
131
|
+
|
|
132
|
+
Checks whether an object exists in the configured Cloudflare R2 bucket.
|
|
133
|
+
|
|
134
|
+
**Usage:**
|
|
135
|
+
|
|
136
|
+
```console
|
|
137
|
+
$ r2 exists <key>
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**Examples:**
|
|
141
|
+
|
|
142
|
+
```console
|
|
143
|
+
$ r2 exists image.jpg
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
**Behavior:**
|
|
147
|
+
|
|
148
|
+
* Requests the metadata of the object, instead of its content, so the check does not depend on the object size.
|
|
149
|
+
* Displays `Object exists: <key>` and exits with status code `0` when the object exists.
|
|
150
|
+
* Displays `Object not found: <key>` and exits with status code `1` when the object does not exist, allowing the command to be used in scripts.
|
|
151
|
+
|
|
152
|
+
On error, the corresponding message is displayed and the CLI exits with a non-zero status code.
|
|
153
|
+
|
|
154
|
+
## Reliability
|
|
155
|
+
|
|
156
|
+
### Automatic retries
|
|
157
|
+
|
|
158
|
+
Transient failures while communicating with Cloudflare R2, such as brief network instabilities, are retried automatically with exponential backoff:
|
|
159
|
+
|
|
160
|
+
| Attempt | Wait before the attempt |
|
|
161
|
+
| ------- | ----------------------- |
|
|
162
|
+
| 1 | — |
|
|
163
|
+
| 2 | 0.5s |
|
|
164
|
+
| 3 | 1s |
|
|
165
|
+
|
|
166
|
+
The wait is capped at 5s, and the operation fails after the last attempt, reporting the mapped domain error. Failures that are not transient, such as missing credentials, an invalid bucket or a missing object, are reported immediately, without retries.
|
|
167
|
+
|
|
168
|
+
The retries are applied to every storage operation and restart from the beginning of the data: the uploaded content is rewound and the downloaded content is written from the start again, so a retried request never sends an incomplete body or leaves a partially written file.
|
|
169
|
+
|
|
170
|
+
### Diagnostics
|
|
171
|
+
|
|
172
|
+
With `--verbose`, every retry is reported on the error output with the operation, the failure cause and the wait applied:
|
|
173
|
+
|
|
174
|
+
```console
|
|
175
|
+
$ r2 upload image.jpg --verbose
|
|
176
|
+
Retrying upload of image.jpg in 0.5s (attempt 2 of 3) after a transient failure: Seahorse::Client::NetworkingError: ...
|
|
177
|
+
```
|
|
178
|
+
|
|
104
179
|
## Global Options
|
|
105
180
|
|
|
106
181
|
Every command accepts the global `--verbose` flag, which writes detailed diagnostic information to the error output without changing the standard output:
|
|
@@ -108,6 +183,9 @@ Every command accepts the global `--verbose` flag, which writes detailed diagnos
|
|
|
108
183
|
```console
|
|
109
184
|
$ r2 list --verbose
|
|
110
185
|
$ r2 upload image.jpg --verbose
|
|
186
|
+
$ r2 download image.jpg --verbose
|
|
187
|
+
$ r2 delete image.jpg --force --verbose
|
|
188
|
+
$ r2 exists image.jpg --verbose
|
|
111
189
|
```
|
|
112
190
|
|
|
113
191
|
Without the flag, diagnostics are discarded by a null logger and only the standard output is produced.
|
data/docs/SECURITY.md
CHANGED
|
@@ -9,6 +9,10 @@ The project follows the following security practices:
|
|
|
9
9
|
- **Isolated test environment** — E2E tests use a dedicated bucket
|
|
10
10
|
(`R2_TEST_BUCKET`) separated from the default application bucket.
|
|
11
11
|
|
|
12
|
+
- **Confirmation of destructive operations** — `r2 delete` asks for
|
|
13
|
+
confirmation before deleting and requires `--force` in non-interactive
|
|
14
|
+
executions, avoiding accidental deletions in scripts and pipelines.
|
|
15
|
+
|
|
12
16
|
- **Ignored environment files** — `.env` files are excluded from version
|
|
13
17
|
control. Only the `.env.example` template is versioned.
|
|
14
18
|
|
data/lib/r2/cli.rb
CHANGED
|
@@ -124,13 +124,23 @@ module R2
|
|
|
124
124
|
end
|
|
125
125
|
|
|
126
126
|
desc "delete FILE", "Deletes a file from R2"
|
|
127
|
+
method_option :force,
|
|
128
|
+
type: :boolean,
|
|
129
|
+
default: false,
|
|
130
|
+
desc: "Deletes without asking for confirmation"
|
|
127
131
|
|
|
128
132
|
long_desc <<~LONGDESC
|
|
129
133
|
Deletes a file from the configured Cloudflare R2 bucket.
|
|
130
134
|
|
|
135
|
+
The deletion is confirmed before the request is sent. Since a
|
|
136
|
+
non-interactive execution cannot ask the user, it requires the
|
|
137
|
+
--force option to proceed.
|
|
138
|
+
|
|
131
139
|
Examples:
|
|
132
140
|
|
|
133
141
|
$ r2 delete image.jpg
|
|
142
|
+
|
|
143
|
+
$ r2 delete image.jpg --force
|
|
134
144
|
LONGDESC
|
|
135
145
|
|
|
136
146
|
# Deletes a file from the configured Cloudflare R2 bucket.
|
|
@@ -141,6 +151,7 @@ module R2
|
|
|
141
151
|
# @param file [String] name of the file to delete
|
|
142
152
|
def delete(file)
|
|
143
153
|
logger.debug("Starting delete: #{file.inspect}.")
|
|
154
|
+
confirm_deletion(file)
|
|
144
155
|
storage.delete(key: file)
|
|
145
156
|
puts "Deleted successfully: #{file}"
|
|
146
157
|
logger.debug("Finished delete: #{file.inspect}.")
|
|
@@ -150,19 +161,28 @@ module R2
|
|
|
150
161
|
end
|
|
151
162
|
|
|
152
163
|
desc "list", "Lists the files stored in R2"
|
|
164
|
+
method_option :prefix,
|
|
165
|
+
type: :string,
|
|
166
|
+
desc: "Lists only the objects whose keys start with the given prefix"
|
|
153
167
|
|
|
154
168
|
long_desc <<~LONGDESC
|
|
155
169
|
Lists the files stored in the configured Cloudflare R2 bucket.
|
|
156
170
|
|
|
171
|
+
Every object is listed, following the pagination of the bucket.
|
|
172
|
+
Use --prefix to list only the objects whose keys start with the
|
|
173
|
+
given prefix.
|
|
174
|
+
|
|
157
175
|
Examples:
|
|
158
176
|
|
|
159
177
|
$ r2 list
|
|
178
|
+
|
|
179
|
+
$ r2 list --prefix uploads/
|
|
160
180
|
LONGDESC
|
|
161
181
|
|
|
162
182
|
# Lists the files stored in the configured Cloudflare R2 bucket.
|
|
163
183
|
def list
|
|
164
|
-
logger.debug("Starting list.")
|
|
165
|
-
files = storage.list
|
|
184
|
+
logger.debug("Starting list: prefix=#{options[:prefix].inspect}.")
|
|
185
|
+
files = storage.list(prefix: options[:prefix])
|
|
166
186
|
|
|
167
187
|
files.each do |file|
|
|
168
188
|
puts file
|
|
@@ -173,6 +193,38 @@ module R2
|
|
|
173
193
|
exit 1
|
|
174
194
|
end
|
|
175
195
|
|
|
196
|
+
desc "exists KEY", "Checks whether an object exists in R2"
|
|
197
|
+
|
|
198
|
+
long_desc <<~LONGDESC
|
|
199
|
+
Checks whether an object exists in the configured Cloudflare R2
|
|
200
|
+
bucket. The exit status is 0 when the object exists and 1 when it
|
|
201
|
+
does not, which allows the command to be used in scripts.
|
|
202
|
+
|
|
203
|
+
Examples:
|
|
204
|
+
|
|
205
|
+
$ r2 exists image.jpg
|
|
206
|
+
LONGDESC
|
|
207
|
+
|
|
208
|
+
# Checks whether an object exists in the configured Cloudflare R2
|
|
209
|
+
# bucket.
|
|
210
|
+
#
|
|
211
|
+
# @param key [String] object key in the bucket
|
|
212
|
+
def exists(key)
|
|
213
|
+
logger.debug("Starting existence check: #{key.inspect}.")
|
|
214
|
+
|
|
215
|
+
if storage.exists?(key: key)
|
|
216
|
+
puts "Object exists: #{key}"
|
|
217
|
+
logger.debug("Finished existence check: #{key.inspect} exists.")
|
|
218
|
+
else
|
|
219
|
+
puts "Object not found: #{key}"
|
|
220
|
+
logger.debug("Finished existence check: #{key.inspect} does not exist.")
|
|
221
|
+
exit 1
|
|
222
|
+
end
|
|
223
|
+
rescue Errors::Error => e
|
|
224
|
+
warn "Error: #{e.message}"
|
|
225
|
+
exit 1
|
|
226
|
+
end
|
|
227
|
+
|
|
176
228
|
private
|
|
177
229
|
|
|
178
230
|
# Opens the given file for reading.
|
|
@@ -269,5 +321,62 @@ module R2
|
|
|
269
321
|
|
|
270
322
|
raise Errors::PermissionError, "Permission denied to write the file: #{destination}"
|
|
271
323
|
end
|
|
324
|
+
|
|
325
|
+
# Asks the user to confirm the deletion of an object.
|
|
326
|
+
#
|
|
327
|
+
# The confirmation is skipped when `--force` is given. Without it the
|
|
328
|
+
# deletion only happens after an affirmative answer, which requires an
|
|
329
|
+
# interactive input: scripts and pipelines cannot be asked, so they
|
|
330
|
+
# must opt out explicitly, avoiding accidental deletions.
|
|
331
|
+
#
|
|
332
|
+
# @param key [String] object key in the bucket
|
|
333
|
+
# @raise [Errors::ConfirmationRequiredError] when the input is not
|
|
334
|
+
# interactive and `--force` was not given
|
|
335
|
+
# @raise [Errors::AbortedError] when the deletion is not confirmed
|
|
336
|
+
def confirm_deletion(key)
|
|
337
|
+
if options[:force]
|
|
338
|
+
logger.debug("Skipping the deletion confirmation: --force was given.")
|
|
339
|
+
return
|
|
340
|
+
end
|
|
341
|
+
|
|
342
|
+
raise Errors::ConfirmationRequiredError, confirmation_required_message(key) unless interactive_input?
|
|
343
|
+
|
|
344
|
+
return if affirmative?(ask_deletion_confirmation(key))
|
|
345
|
+
|
|
346
|
+
raise Errors::AbortedError, "Deletion aborted: #{key}"
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
# Writes the deletion confirmation prompt and reads the answer.
|
|
350
|
+
#
|
|
351
|
+
# @param key [String] object key in the bucket
|
|
352
|
+
# @return [String, nil] answer given by the user
|
|
353
|
+
def ask_deletion_confirmation(key)
|
|
354
|
+
$stdout.print("Delete #{key.inspect} from the bucket? [y/N] ")
|
|
355
|
+
$stdin.gets
|
|
356
|
+
end
|
|
357
|
+
|
|
358
|
+
# Indicates whether the standard input can be used to ask the user.
|
|
359
|
+
#
|
|
360
|
+
# @return [Boolean] true when the input is a terminal
|
|
361
|
+
def interactive_input?
|
|
362
|
+
$stdin.respond_to?(:tty?) && $stdin.tty?
|
|
363
|
+
end
|
|
364
|
+
|
|
365
|
+
# Indicates whether the given answer confirms the operation.
|
|
366
|
+
#
|
|
367
|
+
# @param answer [String, nil] answer given by the user
|
|
368
|
+
# @return [Boolean] true when the answer is affirmative
|
|
369
|
+
def affirmative?(answer)
|
|
370
|
+
answer.to_s.strip.match?(/\Ay(es)?\z/i)
|
|
371
|
+
end
|
|
372
|
+
|
|
373
|
+
# Builds the message shown when the confirmation cannot be requested.
|
|
374
|
+
#
|
|
375
|
+
# @param key [String] object key in the bucket
|
|
376
|
+
# @return [String] message explaining how to proceed
|
|
377
|
+
def confirmation_required_message(key)
|
|
378
|
+
"Deletion of #{key} requires confirmation. Run the command in an " \
|
|
379
|
+
"interactive terminal or use --force to delete it without confirmation."
|
|
380
|
+
end
|
|
272
381
|
end
|
|
273
382
|
end
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module R2
|
|
4
|
+
# Determines the content type of an object from its key.
|
|
5
|
+
#
|
|
6
|
+
# The mapping covers the formats commonly handled by the CLI. Keys whose
|
|
7
|
+
# extension is not mapped fall back to the generic binary type, which is
|
|
8
|
+
# the same behavior offered by S3-compatible services.
|
|
9
|
+
#
|
|
10
|
+
# Reference: https://developer.mozilla.org/docs/Web/HTTP/Basics_of_HTTP/MIME_types
|
|
11
|
+
module ContentType
|
|
12
|
+
# Content type used when the object extension is not mapped.
|
|
13
|
+
DEFAULT = "application/octet-stream"
|
|
14
|
+
|
|
15
|
+
# Content types by file extension.
|
|
16
|
+
TYPES = {
|
|
17
|
+
".avif" => "image/avif",
|
|
18
|
+
".bmp" => "image/bmp",
|
|
19
|
+
".csv" => "text/csv",
|
|
20
|
+
".doc" => "application/msword",
|
|
21
|
+
".docx" => "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
|
22
|
+
".gif" => "image/gif",
|
|
23
|
+
".gz" => "application/gzip",
|
|
24
|
+
".htm" => "text/html",
|
|
25
|
+
".html" => "text/html",
|
|
26
|
+
".ico" => "image/vnd.microsoft.icon",
|
|
27
|
+
".jpeg" => "image/jpeg",
|
|
28
|
+
".jpg" => "image/jpeg",
|
|
29
|
+
".js" => "text/javascript",
|
|
30
|
+
".json" => "application/json",
|
|
31
|
+
".md" => "text/markdown",
|
|
32
|
+
".mp3" => "audio/mpeg",
|
|
33
|
+
".mp4" => "video/mp4",
|
|
34
|
+
".pdf" => "application/pdf",
|
|
35
|
+
".png" => "image/png",
|
|
36
|
+
".svg" => "image/svg+xml",
|
|
37
|
+
".tar" => "application/x-tar",
|
|
38
|
+
".tif" => "image/tiff",
|
|
39
|
+
".tiff" => "image/tiff",
|
|
40
|
+
".txt" => "text/plain",
|
|
41
|
+
".wav" => "audio/wav",
|
|
42
|
+
".webp" => "image/webp",
|
|
43
|
+
".xls" => "application/vnd.ms-excel",
|
|
44
|
+
".xlsx" => "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
|
45
|
+
".xml" => "application/xml",
|
|
46
|
+
".yaml" => "application/yaml",
|
|
47
|
+
".yml" => "application/yaml",
|
|
48
|
+
".zip" => "application/zip"
|
|
49
|
+
}.freeze
|
|
50
|
+
|
|
51
|
+
# Determines the content type of the given object key.
|
|
52
|
+
#
|
|
53
|
+
# The extension is compared in lowercase, so keys stored with
|
|
54
|
+
# uppercase names are also recognized.
|
|
55
|
+
#
|
|
56
|
+
# @param key [String] object key in the bucket
|
|
57
|
+
# @return [String] content type corresponding to the key extension
|
|
58
|
+
def self.for(key)
|
|
59
|
+
TYPES.fetch(extension(key), DEFAULT)
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# Extracts the extension of the given object key.
|
|
63
|
+
#
|
|
64
|
+
# @param key [String] object key in the bucket
|
|
65
|
+
# @return [String] lowercase extension, including the leading dot
|
|
66
|
+
def self.extension(key)
|
|
67
|
+
File.extname(key.to_s).downcase
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
data/lib/r2/errors.rb
CHANGED
|
@@ -42,5 +42,14 @@ module R2
|
|
|
42
42
|
# Unclassified failure in the storage layer.
|
|
43
43
|
class StorageError < Error
|
|
44
44
|
end
|
|
45
|
+
|
|
46
|
+
# The confirmation required before a destructive operation could not
|
|
47
|
+
# be requested, because the execution is not interactive.
|
|
48
|
+
class ConfirmationRequiredError < Error
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# The user did not confirm a destructive operation.
|
|
52
|
+
class AbortedError < Error
|
|
53
|
+
end
|
|
45
54
|
end
|
|
46
55
|
end
|
data/lib/r2/retry.rb
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module R2
|
|
4
|
+
# Automatic retries with exponential backoff.
|
|
5
|
+
#
|
|
6
|
+
# Transient failures, such as brief network instabilities, are retried a
|
|
7
|
+
# limited number of times. The wait between the attempts grows
|
|
8
|
+
# exponentially and is capped, so the remote service has time to recover
|
|
9
|
+
# without making the user wait indefinitely.
|
|
10
|
+
#
|
|
11
|
+
# Only the error classes informed as transient are retried; every other
|
|
12
|
+
# failure is raised immediately, preserving the domain error mapping.
|
|
13
|
+
module Retry
|
|
14
|
+
# Total number of attempts: the first execution plus the retries.
|
|
15
|
+
DEFAULT_MAX_ATTEMPTS = 3
|
|
16
|
+
|
|
17
|
+
# Wait applied after the first failed attempt, in seconds.
|
|
18
|
+
DEFAULT_BASE_DELAY = 0.5
|
|
19
|
+
|
|
20
|
+
# Upper limit of the wait between attempts, in seconds.
|
|
21
|
+
DEFAULT_MAX_DELAY = 5.0
|
|
22
|
+
|
|
23
|
+
# Waiting strategy used when nothing else is provided.
|
|
24
|
+
DEFAULT_SLEEPER = ->(seconds) { sleep(seconds) }
|
|
25
|
+
|
|
26
|
+
# Description used in the diagnostics when the caller gives none.
|
|
27
|
+
DEFAULT_DESCRIPTION = "operation"
|
|
28
|
+
|
|
29
|
+
# Retry settings shared by the operations of a client.
|
|
30
|
+
class Policy
|
|
31
|
+
attr_reader :max_attempts, :base_delay, :max_delay
|
|
32
|
+
|
|
33
|
+
# @param max_attempts [Integer] total number of attempts
|
|
34
|
+
# @param base_delay [Float] wait after the first failure, in seconds
|
|
35
|
+
# @param max_delay [Float] upper limit of the wait, in seconds
|
|
36
|
+
def initialize(
|
|
37
|
+
max_attempts: DEFAULT_MAX_ATTEMPTS,
|
|
38
|
+
base_delay: DEFAULT_BASE_DELAY,
|
|
39
|
+
max_delay: DEFAULT_MAX_DELAY
|
|
40
|
+
)
|
|
41
|
+
@max_attempts = max_attempts
|
|
42
|
+
@base_delay = base_delay
|
|
43
|
+
@max_delay = max_delay
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# Wait applied after the given failed attempt.
|
|
47
|
+
#
|
|
48
|
+
# @param attempt [Integer] number of the attempt that just failed
|
|
49
|
+
# @return [Float] seconds to wait before the next attempt
|
|
50
|
+
def delay_for(attempt)
|
|
51
|
+
[@base_delay * (2**(attempt - 1)), @max_delay].min
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# Runs the block, retrying the transient failures it raises.
|
|
56
|
+
#
|
|
57
|
+
# @param policy [Policy] retry settings
|
|
58
|
+
# @param retry_on [Array<Class>] error classes considered transient
|
|
59
|
+
# @param logger [#debug, nil] logger used for diagnostics
|
|
60
|
+
# @param sleeper [#call, nil] waiting strategy used between attempts
|
|
61
|
+
# @param description [String, nil] operation description used in the diagnostics
|
|
62
|
+
# @yield the operation to run
|
|
63
|
+
# @return [Object] result of the block
|
|
64
|
+
# @raise [StandardError] error of the last attempt when the retries are exhausted
|
|
65
|
+
def self.call(policy: Policy.new, retry_on: [], logger: nil, sleeper: nil, description: nil)
|
|
66
|
+
attempt = 0
|
|
67
|
+
description ||= DEFAULT_DESCRIPTION
|
|
68
|
+
|
|
69
|
+
begin
|
|
70
|
+
attempt += 1
|
|
71
|
+
yield
|
|
72
|
+
rescue StandardError => e
|
|
73
|
+
raise unless transient?(e, retry_on) && attempt < policy.max_attempts
|
|
74
|
+
|
|
75
|
+
delay = policy.delay_for(attempt)
|
|
76
|
+
logger&.debug(retry_message(attempt, policy, description, e, delay))
|
|
77
|
+
(sleeper || DEFAULT_SLEEPER).call(delay)
|
|
78
|
+
retry
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Indicates whether an error is considered transient.
|
|
83
|
+
#
|
|
84
|
+
# @param error [StandardError] error raised by the operation
|
|
85
|
+
# @param retry_on [Array<Class>] error classes considered transient
|
|
86
|
+
# @return [Boolean] true when the failure may be retried
|
|
87
|
+
def self.transient?(error, retry_on)
|
|
88
|
+
retry_on.any? { |error_class| error.is_a?(error_class) }
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
# Builds the diagnostic message of a retry.
|
|
92
|
+
#
|
|
93
|
+
# @param attempt [Integer] number of the attempt that just failed
|
|
94
|
+
# @param policy [Policy] retry settings
|
|
95
|
+
# @param description [String] description of the operation
|
|
96
|
+
# @param error [StandardError] error raised by the attempt
|
|
97
|
+
# @param delay [Float] seconds to wait before the next attempt
|
|
98
|
+
# @return [String] message describing the retry
|
|
99
|
+
def self.retry_message(attempt, policy, description, error, delay)
|
|
100
|
+
"Retrying #{description} in #{delay}s (attempt #{attempt + 1} of " \
|
|
101
|
+
"#{policy.max_attempts}) after a transient failure: " \
|
|
102
|
+
"#{error.class}: #{error.message}"
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
end
|
data/lib/r2/storage.rb
CHANGED
|
@@ -1,22 +1,34 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "aws-sdk-s3"
|
|
4
|
+
require_relative "content_type"
|
|
4
5
|
require_relative "errors"
|
|
5
6
|
require_relative "logging"
|
|
7
|
+
require_relative "retry"
|
|
6
8
|
|
|
7
9
|
module R2
|
|
8
10
|
# Storage layer responsible for communicating with Cloudflare R2.
|
|
9
11
|
#
|
|
10
12
|
# Uses the `aws-sdk-s3` gem with the S3-compatible endpoint provided by
|
|
11
13
|
# the application configuration.
|
|
14
|
+
#
|
|
15
|
+
# Transient network failures are retried with exponential backoff
|
|
16
|
+
# (see `R2::Retry`), so brief instabilities do not fail the operation.
|
|
12
17
|
class Storage
|
|
18
|
+
# Error classes considered transient and therefore retried.
|
|
19
|
+
RETRYABLE_ERRORS = [Seahorse::Client::NetworkingError].freeze
|
|
20
|
+
|
|
13
21
|
# Initializes the storage with the configured credentials and bucket.
|
|
14
22
|
#
|
|
15
23
|
# @param config [Configuration] application configuration
|
|
16
24
|
# @param logger [#debug, nil] logger used for diagnostics
|
|
17
|
-
|
|
25
|
+
# @param retry_policy [Retry::Policy, nil] retry settings, defaults to the project policy
|
|
26
|
+
# @param sleeper [#call, nil] waiting strategy used between attempts
|
|
27
|
+
def initialize(config, logger: nil, retry_policy: nil, sleeper: nil)
|
|
18
28
|
@bucket = config.bucket
|
|
19
29
|
@logger = logger || R2::Logging::NullLogger.new
|
|
30
|
+
@retry_policy = retry_policy || Retry::Policy.new
|
|
31
|
+
@sleeper = sleeper
|
|
20
32
|
@s3 = Aws::S3::Client.new(
|
|
21
33
|
region: config.region,
|
|
22
34
|
access_key_id: config.access_key_id,
|
|
@@ -31,16 +43,29 @@ module R2
|
|
|
31
43
|
# Receives content already prepared by the layer that uses the storage
|
|
32
44
|
# and delivers it to Cloudflare R2.
|
|
33
45
|
#
|
|
46
|
+
# The content type is determined from the object key, unless an
|
|
47
|
+
# explicit value is given, so the stored object is served with the
|
|
48
|
+
# correct type instead of the generic binary type.
|
|
49
|
+
#
|
|
50
|
+
# When the content comes from a stream, it is rewound before every
|
|
51
|
+
# attempt, since a retry must read the content from the beginning.
|
|
52
|
+
#
|
|
34
53
|
# @param key [String] object key in the bucket
|
|
35
54
|
# @param body [IO, String] content of the object to upload
|
|
55
|
+
# @param content_type [String, nil] content type stored in the object metadata
|
|
36
56
|
# @raise [Errors::Error] if the operation fails
|
|
37
|
-
def upload(key:, body:)
|
|
38
|
-
|
|
39
|
-
@
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
57
|
+
def upload(key:, body:, content_type: nil)
|
|
58
|
+
type = content_type || ContentType.for(key)
|
|
59
|
+
@logger.debug("Uploading object #{key.inspect} as #{type.inspect} to bucket #{@bucket.inspect}.")
|
|
60
|
+
with_retries(operation: "upload", key: key) do
|
|
61
|
+
body.rewind if body.respond_to?(:rewind)
|
|
62
|
+
@s3.put_object(
|
|
63
|
+
bucket: @bucket,
|
|
64
|
+
key: key,
|
|
65
|
+
body: body,
|
|
66
|
+
content_type: type
|
|
67
|
+
)
|
|
68
|
+
end
|
|
44
69
|
@logger.debug("Upload of object #{key.inspect} completed.")
|
|
45
70
|
nil
|
|
46
71
|
rescue StandardError => e
|
|
@@ -56,10 +81,12 @@ module R2
|
|
|
56
81
|
# @raise [Errors::Error] if the operation fails
|
|
57
82
|
def delete(key:)
|
|
58
83
|
@logger.debug("Deleting object #{key.inspect} from bucket #{@bucket.inspect}.")
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
84
|
+
with_retries(operation: "delete", key: key) do
|
|
85
|
+
@s3.delete_object(
|
|
86
|
+
bucket: @bucket,
|
|
87
|
+
key: key
|
|
88
|
+
)
|
|
89
|
+
end
|
|
63
90
|
@logger.debug("Deletion of object #{key.inspect} completed.")
|
|
64
91
|
nil
|
|
65
92
|
rescue StandardError => e
|
|
@@ -68,12 +95,24 @@ module R2
|
|
|
68
95
|
|
|
69
96
|
# Lists the objects stored in the configured bucket.
|
|
70
97
|
#
|
|
98
|
+
# Every page of the listing is requested, so all the stored objects
|
|
99
|
+
# are returned regardless of the amount of keys in the bucket.
|
|
100
|
+
#
|
|
101
|
+
# @param prefix [String, nil] lists only the objects whose keys start with the prefix
|
|
71
102
|
# @return [Array<String>] keys of the stored objects
|
|
72
103
|
# @raise [Errors::Error] if the operation fails
|
|
73
|
-
def list
|
|
74
|
-
@logger.debug("Listing objects in bucket #{@bucket.inspect}.")
|
|
75
|
-
|
|
76
|
-
|
|
104
|
+
def list(prefix: nil)
|
|
105
|
+
@logger.debug("Listing objects in bucket #{@bucket.inspect} with prefix #{prefix.inspect}.")
|
|
106
|
+
keys = []
|
|
107
|
+
token = nil
|
|
108
|
+
|
|
109
|
+
loop do
|
|
110
|
+
response = list_page(token, prefix)
|
|
111
|
+
keys.concat(response.contents.map(&:key))
|
|
112
|
+
token = response.next_continuation_token
|
|
113
|
+
break if token.nil? || token.empty?
|
|
114
|
+
end
|
|
115
|
+
|
|
77
116
|
@logger.debug("Found #{keys.size} object(s) in bucket #{@bucket.inspect}.")
|
|
78
117
|
keys
|
|
79
118
|
rescue StandardError => e
|
|
@@ -83,7 +122,9 @@ module R2
|
|
|
83
122
|
# Downloads an object from the configured bucket.
|
|
84
123
|
#
|
|
85
124
|
# The content is streamed directly to the destination file, so large
|
|
86
|
-
# objects do not need to be fully loaded into memory.
|
|
125
|
+
# objects do not need to be fully loaded into memory. Every attempt
|
|
126
|
+
# writes the file from the beginning, avoiding a partially written
|
|
127
|
+
# content when a retry is needed.
|
|
87
128
|
#
|
|
88
129
|
# @param key [String] object key in the bucket
|
|
89
130
|
# @param destination [String] local path where the content is written
|
|
@@ -91,8 +132,10 @@ module R2
|
|
|
91
132
|
# @raise [Errors::Error] if the operation fails
|
|
92
133
|
def download(key:, destination:)
|
|
93
134
|
@logger.debug("Downloading object #{key.inspect} to #{destination.inspect}.")
|
|
94
|
-
|
|
95
|
-
|
|
135
|
+
with_retries(operation: "download", key: key) do
|
|
136
|
+
File.open(destination, "wb") do |file|
|
|
137
|
+
@s3.get_object(bucket: @bucket, key: key, response_target: file)
|
|
138
|
+
end
|
|
96
139
|
end
|
|
97
140
|
@logger.debug("Download of object #{key.inspect} completed.")
|
|
98
141
|
destination
|
|
@@ -100,8 +143,71 @@ module R2
|
|
|
100
143
|
raise_storage_error(e, operation: "download", key: key, destination: destination)
|
|
101
144
|
end
|
|
102
145
|
|
|
146
|
+
# Checks whether an object exists in the configured bucket.
|
|
147
|
+
#
|
|
148
|
+
# The object metadata is requested instead of the content, so the
|
|
149
|
+
# check is cheap regardless of the object size.
|
|
150
|
+
#
|
|
151
|
+
# @param key [String] object key in the bucket
|
|
152
|
+
# @return [Boolean] true when the object exists
|
|
153
|
+
# @raise [Errors::Error] if the operation fails
|
|
154
|
+
def exists?(key:)
|
|
155
|
+
@logger.debug("Checking whether object #{key.inspect} exists in bucket #{@bucket.inspect}.")
|
|
156
|
+
with_retries(operation: "exists", key: key) do
|
|
157
|
+
@s3.head_object(bucket: @bucket, key: key)
|
|
158
|
+
end
|
|
159
|
+
@logger.debug("Object #{key.inspect} was found in bucket #{@bucket.inspect}.")
|
|
160
|
+
true
|
|
161
|
+
rescue Aws::S3::Errors::NoSuchKey, Aws::S3::Errors::NotFound
|
|
162
|
+
@logger.debug("Object #{key.inspect} was not found in bucket #{@bucket.inspect}.")
|
|
163
|
+
false
|
|
164
|
+
rescue StandardError => e
|
|
165
|
+
raise_storage_error(e, operation: "exists", key: key)
|
|
166
|
+
end
|
|
167
|
+
|
|
103
168
|
private
|
|
104
169
|
|
|
170
|
+
# Requests a single page of the object listing.
|
|
171
|
+
#
|
|
172
|
+
# @param token [String, nil] continuation token of the page
|
|
173
|
+
# @param prefix [String, nil] lists only the objects whose keys start with the prefix
|
|
174
|
+
# @return [Object] response of the listing operation
|
|
175
|
+
def list_page(token, prefix)
|
|
176
|
+
params = { bucket: @bucket }
|
|
177
|
+
params[:prefix] = prefix unless prefix.nil?
|
|
178
|
+
params[:continuation_token] = token unless token.nil?
|
|
179
|
+
|
|
180
|
+
with_retries(operation: "list") { @s3.list_objects_v2(**params) }
|
|
181
|
+
end
|
|
182
|
+
|
|
183
|
+
# Runs the given block retrying transient network failures.
|
|
184
|
+
#
|
|
185
|
+
# @param operation [String] operation being performed
|
|
186
|
+
# @param key [String, nil] object key involved in the operation
|
|
187
|
+
# @yield the request to run against the storage
|
|
188
|
+
# @return [Object] result of the block
|
|
189
|
+
def with_retries(operation:, key: nil, &)
|
|
190
|
+
R2::Retry.call(
|
|
191
|
+
policy: @retry_policy,
|
|
192
|
+
retry_on: RETRYABLE_ERRORS,
|
|
193
|
+
logger: @logger,
|
|
194
|
+
sleeper: @sleeper,
|
|
195
|
+
description: describe_operation(operation, key),
|
|
196
|
+
&
|
|
197
|
+
)
|
|
198
|
+
end
|
|
199
|
+
|
|
200
|
+
# Builds the description of an operation used in the diagnostics.
|
|
201
|
+
#
|
|
202
|
+
# @param operation [String] operation being performed
|
|
203
|
+
# @param key [String, nil] object key involved in the operation
|
|
204
|
+
# @return [String] operation description
|
|
205
|
+
def describe_operation(operation, key)
|
|
206
|
+
return "#{operation} on bucket #{@bucket}" if key.nil?
|
|
207
|
+
|
|
208
|
+
"#{operation} of #{key}"
|
|
209
|
+
end
|
|
210
|
+
|
|
105
211
|
# Converts storage layer errors into project domain errors, avoiding
|
|
106
212
|
# exposing internal details of the implementations.
|
|
107
213
|
#
|
data/lib/r2/version.rb
CHANGED
data/lib/r2.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: cloudflare-r2-cli
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- rpzerosixcode
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-09-
|
|
11
|
+
date: 2026-09-21 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: aws-sdk-s3
|
|
@@ -127,8 +127,10 @@ files:
|
|
|
127
127
|
- lib/r2.rb
|
|
128
128
|
- lib/r2/cli.rb
|
|
129
129
|
- lib/r2/configuration.rb
|
|
130
|
+
- lib/r2/content_type.rb
|
|
130
131
|
- lib/r2/errors.rb
|
|
131
132
|
- lib/r2/logging.rb
|
|
133
|
+
- lib/r2/retry.rb
|
|
132
134
|
- lib/r2/storage.rb
|
|
133
135
|
- lib/r2/version.rb
|
|
134
136
|
homepage: https://github.com/rpzerosixcode/cloudflare-r2-cli
|