s3p 3.4.3 → 3.4.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -11,7 +11,7 @@ jobs:
11
11
  - name: Use Node.js
12
12
  uses: actions/setup-node@v1
13
13
  with:
14
- node-version: 12.x
14
+ node-version: 16.x
15
15
  - name: Install NPM dependencies
16
16
  run: npm ci
17
17
  - name: Run tests
package/README.md CHANGED
@@ -23,18 +23,19 @@ Read more about [S3P on Medium](https://medium.com/@shanebdavis/s3p-massively-pa
23
23
  1. [NodeJS](https://nodejs.org/en/download/)
24
24
  2. [AWS-CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
25
25
 
26
- The `aws-cli` is required for copying large files. By default, files larger than 100 megabytes are copied with `aws-cli`. This is a good compromise for performance. However, you can change that threshold to 5 gigabytes with the `--large-copy-threshold` option.
27
- > Why? The `aws-sdk` does not support coping files larger than 5 gigabytes without a much more complicated solution.
26
+ The `aws-cli` is required for copying large files. By default, files larger than 100 megabytes are copied with `aws-cli`. This is a good compromise for performance. However, you can change that threshold to 5 gigabytes with the `--large-copy-threshold` option.
27
+
28
+ > Why? The `aws-sdk` does not support coping files larger than 5 gigabytes without a much more complicated solution.
28
29
 
29
30
  3. Key names must use a limited character set:
30
- ```
31
- <space>
32
- !"#$%&'()*+,-./
33
- 0123456789:;<=>?@
34
- ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`
35
- abcdefghijklmnopqrstuvwxyz{|}~
36
- ```
37
- > Why? Since Aws-S3 doesn't support listing Keys in descending order, S3P uses a character-range-based divide-and-conquer algorithm.
31
+ ```
32
+ <space>
33
+ !"#$%&'()*+,-./
34
+ 0123456789:;<=>?@
35
+ ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`
36
+ abcdefghijklmnopqrstuvwxyz{|}~
37
+ ```
38
+ > Why? Since Aws-S3 doesn't support listing Keys in descending order, S3P uses a character-range-based divide-and-conquer algorithm.
38
39
 
39
40
  # AWS Credentials
40
41
 
@@ -61,6 +62,7 @@ npx s3p cp --help
61
62
  # Install NPM Package
62
63
 
63
64
  You can also install s3p locally which will allow it to run faster.
65
+
64
66
  ```shell
65
67
  # install s3p on your current machine
66
68
  npm install s3p -g
@@ -89,15 +91,14 @@ S3-bucket-copying performance can exceed 8 gigabytes per second.
89
91
 
90
92
  The average file-size has a big impact on s3p's overall bytes-per-second:
91
93
 
92
- |location | command | aws-cli | s3p | speedup | average size |
93
- | - |- |- |- |- | - |
94
- |local | ls | 2000 items/s | 20000 items/s | 10x | n/a|
95
- |local | cp | 30 mB/s | 150 mB/s | 5x | 512 kB |
96
- |ec2 | cp | 150 mB/s | 8 gB/s | 54x | 100 mB |
94
+ | location | command | aws-cli | s3p | speedup | average size |
95
+ | -------- | ------- | ------------ | ------------- | ------- | ------------ |
96
+ | local | ls | 2000 items/s | 20000 items/s | 10x | n/a |
97
+ | local | cp | 30 mB/s | 150 mB/s | 5x | 512 kB |
98
+ | ec2 | cp | 150 mB/s | 8 gB/s | 54x | 100 mB |
97
99
 
98
100
  > S3P was developed to operate on buckets with millions of items and 100s of terabytes. Currently, S3P is still only a single-core NODE application. There are opportunities for even more massively parallel S3 operations by forking workers or even distributing the work across instances with something like Elastic-Queue. If someone needs solutions that are 100-1000x faster than aws-cli, let us know. We'd love to work with you.<br>-
99
- shane@genui.com
100
-
101
+ > shane@genui.com
101
102
 
102
103
  # Documentation
103
104
 
@@ -116,39 +117,51 @@ npx s3p cp --help
116
117
 
117
118
  All the capabilities of the CLI are also available as an API. To learn the API, first learn the CLI options, and then, to learn the API call for a specific CLI command, run that command on the command-line with the `--api-example` option. This will output example JavaScript code for invoking that command programmatically.
118
119
 
119
- > NOTE: When you use `--api-example` on the command-line, your command won't actually run. S3P will *only* output the JavaScript equivalent of the CLI command to the console and then quit.
120
+ > NOTE: When you use `--api-example` on the command-line, your command won't actually run. S3P will _only_ output the JavaScript equivalent of the CLI command to the console and then quit.
120
121
 
121
122
  ### Example
123
+
122
124
  Run:
123
125
 
124
126
  ```shell
125
127
  > npx s3p ls --bucket foo --quiet --api-example
126
128
  ```
129
+
127
130
  Output:
131
+
128
132
  ```javascript
129
- require('s3p').ls({
130
- bucket: "foo",
131
- quiet: true
132
- })
133
+ require("s3p").ls({
134
+ bucket: "foo",
135
+ quiet: true,
136
+ });
133
137
  // > Promise
134
138
  ```
135
139
 
136
140
  Test run:
141
+
137
142
  ```shell
138
143
  > node
139
144
  ```
145
+
140
146
  Paste:
147
+
141
148
  ```javascript
142
- require('s3p').listBuckets({
143
- bucket: "foo",
144
- quiet: true
145
- }).then((out) => console.log(out));
149
+ require("s3p")
150
+ .ls({
151
+ bucket: "foo",
152
+ quiet: true,
153
+ })
154
+ .then(out => console.log(out));
146
155
  ```
156
+
147
157
  Output:
158
+
148
159
  ```javascript
149
- {
150
- "bucket-a-name": creationDateA,
151
- "bucket-b-name": creationDateB
160
+ [
161
+ 'item1',
162
+ 'item2',
163
+ 'item3',
164
+ ... 8463 more items
152
165
  }
153
166
  ```
154
167
 
@@ -156,4 +169,4 @@ Output:
156
169
 
157
170
  S3P was originally developed by [GenUI.com](https://www.genui.com/) in conjunction with [Resolution Bioscience, Inc.](http://www.resolutionbio.com/)
158
171
 
159
- GenUI is a technology commercialization software consultancy based in Seattle. We accelerate software roadmaps. Please feel free to [contact GenUI](https://www.genui.com/contact) and tell us about your project. We'd love to hear from you.
172
+ GenUI is a technology commercialization software consultancy based in Seattle. We accelerate software roadmaps. Please feel free to [contact GenUI](https://www.genui.com/contact) and tell us about your project. We'd love to hear from you.
package/package.json CHANGED
@@ -50,7 +50,7 @@
50
50
  "test": "npm run nn\njest",
51
51
  "watch": "npm-watch"
52
52
  },
53
- "version": "3.4.3",
53
+ "version": "3.4.5",
54
54
  "watch": {
55
55
  "build": {
56
56
  "extensions": [