s3p 3.5.1 → 3.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@ You can use it as a command-line tool for common operations, or you can use it a
|
|
|
8
8
|
|
|
9
9
|
# Why is S3P so fast?
|
|
10
10
|
|
|
11
|
-
S3's API is structured around listing items in serial - request 1000 items, wait, then request the next 1000. This is how nearly all S3 tools work. S3P, however, can list items in parallel. It leverages S3's ability to request the first 1000 items equal-to or after a given key. Then, with the help of algorithmic bisection and some intelligent heuristics, S3P can scan the contents of a bucket with an arbitrary degree of parallism. In practice, S3P can list buckets up to **
|
|
11
|
+
S3's API is structured around listing items in serial - request 1000 items, wait, then request the next 1000. This is how nearly all S3 tools work. S3P, however, can list items in parallel. It leverages S3's ability to request the first 1000 items equal-to or after a given key. Then, with the help of algorithmic bisection and some intelligent heuristics, S3P can scan the contents of a bucket with an arbitrary degree of parallism. In practice, S3P can list buckets up to **20x faster** than conventional methods.
|
|
12
12
|
|
|
13
13
|
S3P is really just a fancy, really fast, S3 listing tool. Summarizing, copying and synching are all boosted by S3P's core ability to list objects radically faster.
|
|
14
14
|
|
|
@@ -83,7 +83,7 @@ In addition to performance, S3P provides flexible options for custom list, copyi
|
|
|
83
83
|
|
|
84
84
|
Surprisingly, you don't even need to run S3P in the cloud to see much of its benefits. You can run it on your local machine and, since S3 copying never goes directly through S3P, it doesn't use up any AWS bandwidth.
|
|
85
85
|
|
|
86
|
-
S3-bucket-listing performance can hit almost 20,000 items per second.
|
|
86
|
+
S3-bucket-listing performance can hit almost ~~20,000~~ 50,000 items per second (as-of S3Pv3.5).
|
|
87
87
|
|
|
88
88
|
S3-bucket-copying performance can exceed 8 gigabytes per second.
|
|
89
89
|
|
|
@@ -93,11 +93,11 @@ The average file-size has a big impact on s3p's overall bytes-per-second:
|
|
|
93
93
|
|
|
94
94
|
| location | command | aws-cli | s3p | speedup | average size |
|
|
95
95
|
| -------- | ------- | ------------ | ------------- | ------- | ------------ |
|
|
96
|
-
| local | ls |
|
|
96
|
+
| local | ls | 2500 items/s | 50000 items/s | 20x | n/a |
|
|
97
97
|
| local | cp | 30 mB/s | 150 mB/s | 5x | 512 kB |
|
|
98
98
|
| ec2 | cp | 150 mB/s | 8 gB/s | 54x | 100 mB |
|
|
99
99
|
|
|
100
|
-
> S3P was developed to operate on buckets with millions of items and 100s of terabytes. Currently, S3P is still only a single-core
|
|
100
|
+
> S3P was developed to operate on buckets with millions of items and 100s of terabytes. Currently, S3P is still only a single-core NodeJS application. There are opportunities for even more massively parallel S3 operations by forking workers or even distributing the work across instances with something like Elastic-Queue. If someone needs solutions that are 100-1000x faster than aws-cli, let us know. We'd love to work with you.<br>-
|
|
101
101
|
> shane@genui.com
|
|
102
102
|
|
|
103
103
|
# Documentation
|
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
"use strict"
|
|
2
2
|
let Caf = require('caffeine-script-runtime');
|
|
3
|
-
Caf.defMod(module, () => {return Caf.importInvoke(["BaseClass", "shellEscape", "S3Client", "Promise", "ListBucketsCommand", "ListObjectsV2Command", "GetObjectCommand", "CopyObjectCommand", "DeleteObjectCommand", "HeadObjectCommand", "lowerCamelCase", "currentSecond", "log", "Array", "Error", "
|
|
3
|
+
Caf.defMod(module, () => {return Caf.importInvoke(["BaseClass", "shellEscape", "S3Client", "merge", "process", "Promise", "ListBucketsCommand", "ListObjectsV2Command", "GetObjectCommand", "CopyObjectCommand", "DeleteObjectCommand", "HeadObjectCommand", "lowerCamelCase", "currentSecond", "log", "Array", "Error", "encodeURIComponent", "path", "timeout", "createParentDirs", "createWriteStreamSafe", "compactFlatten", "createS3Url", "shellExec", "dashCase", "isFunction", "present"], [global, require('../StandardImport'), require('./FsEasy'), require('./LibMisc'), require('@aws-sdk/client-s3'), {shellEscape: require('shell-escape'), promisify: require('util').promisify, fs: require('fs'), path: require('path')}], (BaseClass, shellEscape, S3Client, merge, process, Promise, ListBucketsCommand, ListObjectsV2Command, GetObjectCommand, CopyObjectCommand, DeleteObjectCommand, HeadObjectCommand, lowerCamelCase, currentSecond, log, Array, Error, encodeURIComponent, path, timeout, createParentDirs, createWriteStreamSafe, compactFlatten, createS3Url, shellExec, dashCase, isFunction, present) => {let escape, S3; escape = function(single) {return shellEscape([single]);}; return S3 = Caf.defClass(class S3 extends BaseClass {}, function(S3, classSuper, instanceSuper) {this.awsSdkS3 = new S3Client(merge({region: process.env.AWS_REGION})); this.classGetter({s3: function() {return {listBuckets: (options = {}) => Promise.resolve(this.awsSdkS3.send(new ListBucketsCommand(options))), listObjectsV2: (options = {}) => Promise.resolve(this.awsSdkS3.send(new ListObjectsV2Command(options))), getObject: (options = {}) => Promise.resolve(this.awsSdkS3.send(new GetObjectCommand(options))), copyObject: (options = {}) => Promise.resolve(this.awsSdkS3.send(new CopyObjectCommand(options))), deleteObject: (options = {}) => Promise.resolve(this.awsSdkS3.send(new DeleteObjectCommand(options))), headObject: (options = {}) => Promise.resolve(this.awsSdkS3.send(new HeadObjectCommand(options)))};}}); this.commonCopyOptionsSdkKeys = ["ACL", "CacheControl", "ContentDisposition", "ContentEncoding", "ContentLanguage", "ContentType", "Expires", "RequestPayer", "StorageClass"]; this.commonSdkCopyOptionsByLowerCamelCaseKeys = Caf.object(this.commonCopyOptionsSdkKeys, null, null, null, (k) => lowerCamelCase(k)); this.listBuckets = () => this.s3.listBuckets().then(({Buckets}) => Caf.object(Buckets, ({Name, CreationDate}) => CreationDate, null, null, ({Name, CreationDate}) => Name)); this.list = ({bucket, prefix, limit = 1000, fetchOwner, startAfter}) => {let startTime; startTime = currentSecond(); return this.s3.listObjectsV2({Bucket: bucket, Prefix: prefix, MaxKeys: limit, StartAfter: startAfter, FetchOwner: fetchOwner}).tapCatch((error) => log.error({"S3.list-error": {bucket, prefix, startAfter, limit, error}})).then((results) => {let duration; duration = currentSecond() - startTime; if (!(results.Contents != null)) {results.Contents = [];}; if (!(Caf.is(results.Contents, Array))) {log.warn({"S3.list-no-Contents": {bucket, prefix, startAfter, limit, duration, results}}); throw new Error("S3.list: Contents is not an array");} else {if (duration > 60) {log.warn({"S3.list-slow": {bucket, prefix, startAfter, limit, duration, results: merge(results, {Contents: `Array ${Caf.toString(results.Contents.length)}`})}});};}; return results.Contents;});}; this.copy = (options) => {let largeCopyThreshold, scratchState, copyScratchState, fromBucket, toBucket, fromKey, toKey, size, toFolder, pretend, verbose, copyOptions, toLocalFile, temp, temp1; if ((largeCopyThreshold = options.largeCopyThreshold, scratchState = options.scratchState)) {copyScratchState = ((temp = scratchState.copyScratchState) != null ? temp : scratchState.copyScratchState = {});}; temp1 = this._normalizeCopyOptions(options); fromBucket = temp1.fromBucket; toBucket = temp1.toBucket; fromKey = temp1.fromKey; toKey = temp1.toKey; size = temp1.size; toFolder = temp1.toFolder; pretend = temp1.pretend; verbose = temp1.verbose; return (size >= largeCopyThreshold) ? this.largeCopy({commonCopyOptions: this.extractCommonCopyOptions(options), fromBucket, toBucket, toFolder, fromKey, toKey, pretend, verbose}) : (copyOptions = merge(toFolder ? {Bucket: fromBucket, Key: fromKey} : {CopySource: encodeURIComponent(`${Caf.toString(fromBucket)}/${Caf.toString(fromKey)}`), Bucket: toBucket, Key: toKey}, this._getCommonCopyOptionsForSdk(options)), toLocalFile = toFolder ? path.join(toFolder, toKey) : undefined, verbose ? log.unquoted(merge({copyObject: copyOptions, toLocalFile})) : undefined, pretend ? timeout(1, () => {return {pretend: true};}) : toFolder ? (/\/$/.test(toKey) && size === 0) ? createParentDirs(toLocalFile + "ignored-file-name") : createWriteStreamSafe(toLocalFile, copyScratchState).then((writeStream) => new Promise((resolve, reject) => this.s3.getObject(copyOptions).then(({Body}) => Body.pipe(writeStream).on("finish", resolve).on("error", reject)))) : this.s3.copyObject(copyOptions));}; this.delete = (options) => this.s3.deleteObject({Bucket: options.bucket, Key: options.key}); this.largeCopy = (options) => {let commonCopyOptions, fromBucket, fromFolder, toBucket, toFolder, fromKey, toKey, pretend, verbose, command, temp; commonCopyOptions = options.commonCopyOptions; temp = this._normalizeCopyOptions(options); fromBucket = temp.fromBucket; fromFolder = temp.fromFolder; toBucket = temp.toBucket; toFolder = temp.toFolder; fromKey = temp.fromKey; toKey = temp.toKey; pretend = temp.pretend; verbose = temp.verbose; command = compactFlatten([`aws s3 cp ${Caf.toString(escape(createS3Url(fromBucket, fromFolder, fromKey)))} ${Caf.toString(escape(createS3Url(toBucket, toFolder, toKey)))}`, Caf.array(commonCopyOptions, (v, key) => `--${Caf.toString(dashCase(key))} ${Caf.toString((() => {switch (v) {case true: return ""; default: return escape(v);};})())}`, (v, key) => v)]).join(" "); if (verbose) {log.unquoted({largeCopy: command});}; return pretend ? timeout(1, () => {return {pretend: true};}) : shellExec(command);}; this.headObject = ({bucket, key}) => this.s3.headObject({Bucket: bucket, Key: key}); this._normalizeCopyOptions = function(options) {let bucket, key, fromBucket, toBucket, fromKey, toKey, size, temp, temp1, temp2, temp3; bucket = options.bucket; key = options.key; fromBucket = (undefined !== (temp = options.fromBucket)) ? temp : bucket; toBucket = (undefined !== (temp1 = options.toBucket)) ? temp1 : bucket; fromKey = (undefined !== (temp2 = options.fromKey)) ? temp2 : key; toKey = (undefined !== (temp3 = options.toKey)) ? temp3 : key; size = options.size; if (isFunction(toKey)) {toKey = toKey(fromKey, fromBucket, toBucket, options.size);}; if (!(present(fromBucket) && present(toBucket) && present(fromKey) && present(toKey))) {throw new Error("Missing one of: fromBucket, toBucket, fromKey, toKey or bucket or key as a default");}; return merge(options, {fromBucket, fromKey, toBucket, toKey});}; this._getCommonCopyOptionsForSdk = (options) => Caf.object(this.commonSdkCopyOptionsByLowerCamelCaseKeys, (sdkKey, lowerCamelCasedKey) => options[lowerCamelCasedKey], null, null, (sdkKey, lowerCamelCasedKey) => sdkKey); this.extractCommonCopyOptions = (options) => Caf.object(this.commonSdkCopyOptionsByLowerCamelCaseKeys, (v, k) => options[k]); this.shouldSyncObjects = function(options) {let fromBucket, toBucket, fromKey, toKey, temp; temp = this._normalizeCopyOptions(options); fromBucket = temp.fromBucket; toBucket = temp.toBucket; fromKey = temp.fromKey; toKey = temp.toKey; return Promise.then(() => (options.size < Caf.pow(1024, 2) || !options.size) ? true : this.headObject(options).then(({ContentLength}) => ContentLength !== options.size));}; this.syncObject = (options) => this.shouldSyncObjects(options).then((shouldSync) => shouldSync ? this.copyObject(options).then((result) => merge(result, {copied: true})) : {copied: false});});});});
|
|
4
4
|
//# sourceMappingURL=S3.js.map
|
|
@@ -6,5 +6,5 @@
|
|
|
6
6
|
"source/S3Parallel/Lib/S3.caf"
|
|
7
7
|
],
|
|
8
8
|
"names": [],
|
|
9
|
-
"mappings": "AAAA;;AAAA,
|
|
9
|
+
"mappings": "AAAA;;AAAA,4eAAO,8BAAiB,qBAAS,sBAAU,gCAAwB,sCAAc,sCAAiB,mBAAK,qaACvG,SAAM,QAAG,CAAC,SAAW,oBAAe,sDAInB,uDAEf,gBAAY,IAAI,SAAS,MAAK,CAAC,QAAQ,2BAGvC,iBAAY,CACV,gBAAM,QACJ,aAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,mBAAmB,YACxF,eAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,qBAAqB,YAC1F,WAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,iBAAiB,YACtF,YAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,kBAAkB,YACvF,cAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,oBAAoB,YACzF,YAAgB,CAAC,iBAAiB,gBAAgB,mBAAe,IAAI,kBAAkB,iBAE3F,iCACE,OACA,gBACA,sBACA,mBACA,mBACA,eACA,WACA,gBACA,iBAEF,2DAA0D,wDAAmC,eAAe,wBAE5G,MACE,2BACM,CAAC,CAAC,wBAC2B,mCAA2B,wEAEzD,CAAC,CAAC,QAAQ,QAAQ,cAAY,YAAY,gCAC/C,YAAY,wBACZ,uBACE,QAAY,QACZ,QAAY,QACZ,SAAY,OACZ,YAAY,YACZ,YAAY,sBACJ,CAAC,UACT,WAAU,kBAAkB,QAAQ,QAAQ,YAAY,OAAO,eAE3D,CAAC,2BACL,WAAW,kBAAkB,WAC7B,MAAI,4BAA8B,yBAClC,KAAO,0BAAoB,UACzB,UAAS,wBAAwB,QAAQ,QAAQ,YAAY,OAAO,UAAU,YAC9E,MAAM,IAAI,MAAM,6CAEb,IAAG,WAAW,KACjB,SAAQ,CAAC,iBAAiB,QAAQ,QAAQ,YAAY,OAAO,UAAU,SAAS,MAAM,UAAS,gCAAmB,4CAEpH,mCAsBG,CAAC,oLACN,0BAAG,mEACD,4BAA+B,gDAAiB,+CAElD,2BAAuB,yNAMpB,SAAQ,sBACT,eAAU,CACR,mBAAmB,8BAA0B,UAC7C,YAAY,UAAU,UAAU,SAAS,OAAO,SAAS,YAEzD,CACF,cAAc,MACT,WAAQ,CACT,QAAY,YACZ,KAAY,WAEV,CACF,YAAY,mCAAwB,4BAAc,aAClD,QAAY,UACZ,KAAY,QAEd,iCAA6B,WAE/B,cAAiB,WAAc,UAAU,UAAU,oBAEhD,UACD,aAAa,OACX,YAAY,aACZ,4BAED,UACD,QAAQ,UAAK,iBAAS,WAEhB,WACH,YAAW,UAAU,SAAQ,KAC9B,iBAAiB,cAAc,uBAE/B,sBACE,aACA,uBAEI,CAAC,gBAAgB,IAAI,QAAQ,CAAC,SAAS,WAC3C,kBAAc,kBACR,CAAC,CAAC,UACN,UACM,gBACF,UAAQ,YACR,SAAQ,aAGlB,mBAAe,+BAEZ,CAAC,YACR,qBAAgB,CACd,QAAQ,gBACR,KAAQ,gCAEA,CAAC,iIAAW,oBACtB,kCACA,2BAAuB,sNAMvB,UACE,gBAAc,0BAGN,OAAO,YAAY,YAAY,YAAY,2BAC3C,OAAO,YAAY,UAAU,UAAU,sBAC3B,+BAAwB,kBACjC,SAAS,sBACd,gBAAO,IAAC,KACH,MAAU,oBACV,cAAO,oCAEZ,MAER,IAAG,UACD,cAAa,WAAW,oBAEvB,UACD,QAAQ,UAAK,iBAAS,WAEtB,UAAU,8BAGD,CAAC,CAAC,QAAQ,SACrB,oBACE,QAAQ,QACR,KAAK,oCAET,QAAwB,CAAC,kGAAW,SAClC,qGAEe,wEACF,sEACD,+DACF,0BAGV,IAAG,WAAW,SACZ,QAAQ,MAAM,SAAS,YAAY,UAAU,iBAE/C,KAAO,CAAmB,uBAAI,QAAQ,aAAa,QAAQ,YAAY,QAAQ,UAC7E,MAAM,IAAI,MAAQ,gGACpB,MAAM,UAAY,YAAY,SAAS,UAAU,8CAErB,CAAC,uBACU,+EACrC,QAAQ,0GAEe,CAAC,uBACT,yDACf,QAAQ,8BAEZ,QAAoB,CAAC,0DAAW,OAC9B,2BAAuB,qHACvB,aAAY,MACP,CAAY,uBAAG,MAAM,OAAM,gBAC5B,OAEA,gBAAY,cACN,CAAC,CAAC,mBAAmB,kBAAiB,oCASrC,CAAC,YACZ,uBAAmB,cACb,CAAC,eACF,aACD,gBAAY,cACN,CAAC,WAAW,MAAM,SAAQ,QAAQ,UAErC,CAAG,QAAQ"
|
|
10
10
|
}
|
package/package.json
CHANGED
|
@@ -5,7 +5,7 @@ escape = (single) -> shellEscape [] single
|
|
|
5
5
|
|
|
6
6
|
class S3 extends BaseClass
|
|
7
7
|
|
|
8
|
-
@awsSdkS3 = new S3Client
|
|
8
|
+
@awsSdkS3 = new S3Client merge region: process.env.AWS_REGION # useAccelerateEndpoint: true
|
|
9
9
|
# https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html
|
|
10
10
|
|
|
11
11
|
@classGetter
|
|
@@ -117,17 +117,20 @@ class S3 extends BaseClass
|
|
|
117
117
|
timeout 1 -> pretend: true
|
|
118
118
|
|
|
119
119
|
else if toFolder
|
|
120
|
-
|
|
121
|
-
toLocalFile
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
.
|
|
129
|
-
.
|
|
130
|
-
|
|
120
|
+
if /\/$/.test(toKey) && size == 0 # this is a folder
|
|
121
|
+
createParentDirs toLocalFile + "ignored-file-name"
|
|
122
|
+
else
|
|
123
|
+
createWriteStreamSafe
|
|
124
|
+
toLocalFile
|
|
125
|
+
copyScratchState
|
|
126
|
+
|
|
127
|
+
.then (writeStream) -> new Promise (resolve, reject) ->
|
|
128
|
+
@s3.getObject copyOptions
|
|
129
|
+
.then ({Body}) ->
|
|
130
|
+
Body
|
|
131
|
+
.pipe(writeStream);
|
|
132
|
+
.on :finish resolve
|
|
133
|
+
.on :error reject
|
|
131
134
|
|
|
132
135
|
else
|
|
133
136
|
@s3.copyObject copyOptions
|