opendal 0.1.6.pre.rc.1

data/core/src/docs/rfcs/0423_command_line_interface.md

@@ -0,0 +1,268 @@
+- Proposal Name: `command_line_interface`
+- Start Date: 2022-07-08
+- RFC PR: [apache/opendal#423](https://github.com/apache/opendal/pull/423)
+- Tracking Issue: [apache/opendal#422](https://github.com/apache/opendal/issues/422)
+
+# Summary
+
+Add command line interface for OpenDAL.
+
+# Motivation
+
+> **Q**: There are so many cli out there, why we still need a cli for OpenDAL?
+> 
+> **A**: Because there are so many cli out there.
+
+To manipulate our date store in different could service, we need to install different clis:
+
+- [`aws-cli`]/[`s3cmd`]/... for AWS (S3)
+- [`azcopy`] for Azure Storage Service
+- [`gcloud`] for Google Cloud
+
+Those clis provide native and seamless experiences for their own products but also lock us and our data. 
+
+However, for 80% cases, we just want to do simple jobs like `cp`, `mv` and `rm`. It's boring to figure out how to use them:
+
+- `aws --endpoint-url http://127.0.0.1:9900/ s3 cp data s3://testbucket/data  --recursive`
+- `azcopy copy 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive`
+- `gsutil cp data gs://testbucket/`
+
+Can we use them in the same way? Can we let the data flow freely? 
+
+Let's look back OpenDAL's slogan:
+
+**Open Data Access Layer that connect the whole world together**
+
+This is a natural extension for OpenDAL: providing a command line interface!
+
+# Guide-level explanation
+
+OpenDAL will provide a new cli called: `oli`. It's a shortcut of `OpenDAL Command Line Interface`.
+
+Users can install this cli via:
+
+```shell
+cargo install oli
+```
+
+Or using they favourite package management:
+
+```shell
+# Archlinux
+pacman -S oli
+# Debian / Ubuntu
+apt install oli
+# Rocky Linux / Fedora
+dnf install oli
+# macOS
+brew install oli
+```
+
+With `oli`, users can:
+
+- Upload files to s3: `oli cp books.csv s3://bucket/books.csv`
+- Download files from azblob: `oli cp azblob://bucket/books.csv /tmp/books.csv`
+- Move data between storage services: `oli mv s3://bucket/dir azblob://bucket/dir`
+- Delete all files: `oli rm -rf s3://bucket`
+
+`oli` also provide alias to make cloud data manipulating even natural:
+
+- `ocp` for `oli cp`
+- `ols` for `oli ls`
+- `omv` for `oli mv`
+- `orm` for `oli rm`
+- `ostat` for `oli stat`
+
+`oli` will provide profile management so users don't need to provide credential every time:
+
+- `oli profile add my_s3 --bucket test --access-key-id=example --secret-access-key=example`
+- `ocp my_s3://dir /tmp/dir`
+
+# Reference-level explanation
+
+`oli` will be a separate crate apart from `opendal` so we will not pollute the dependencies of `opendal`. But `oli` will be releases at the same time with the same version of `opendal`. That means `oli` will always use the same (latest) version of opendal.
+
+Most operations of `oli` should be trivial, we will propose new RFCs if requiring big changes.
+
+`oli` won't keep configuration. All config will go through environment, for example:
+
+- `OIL_COLOR=always`
+- `OIL_CONCURRENCY=16`
+
+Besides, `oil` will read profile from env like `cargo`:
+
+- `OIL_PROFILE_TEST_TYPE=s3`
+- `OIL_PROFILE_TEST_ENDPOINT=http://127.0.0.1:1090`
+- `OIL_PROFILE_TEST_BUCKET=test_bucket`
+- `OIL_PROFILE_TEST_ACCESS_KEPT_ID=access_key_id`
+- `OIL_PROFILE_TEST_SECRET_ACCESS_KEY=secret_access_key`
+
+With those environments, we can:
+
+```shell
+ocp path/to/dir test://test/to/dir
+```
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+## s3cmd
+
+[s3cmd](https://s3tools.org/s3cmd) is a command line s3 client for Linux and Mac.
+
+```shell
+Usage: s3cmd [options] COMMAND [parameters]
+
+S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
+making and removing "buckets" and uploading, downloading and removing
+"objects" from these buckets.
+
+Commands:
+  Make bucket
+      s3cmd mb s3://BUCKET
+  Remove bucket
+      s3cmd rb s3://BUCKET
+  List objects or buckets
+      s3cmd ls [s3://BUCKET[/PREFIX]]
+  List all object in all buckets
+      s3cmd la 
+  Put file into bucket
+      s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
+  Get file from bucket
+      s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
+  Delete file from bucket
+      s3cmd del s3://BUCKET/OBJECT
+  Delete file from bucket (alias for del)
+      s3cmd rm s3://BUCKET/OBJECT
+  Restore file from Glacier storage
+      s3cmd restore s3://BUCKET/OBJECT
+  Synchronize a directory tree to S3 (checks files freshness using 
+       size and md5 checksum, unless overridden by options, see below)
+      s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
+  Disk usage by buckets
+      s3cmd du [s3://BUCKET[/PREFIX]]
+  Get various information about Buckets or Files
+      s3cmd info s3://BUCKET[/OBJECT]
+  Copy object
+      s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
+  Modify object metadata
+      s3cmd modify s3://BUCKET1/OBJECT
+  Move object
+      s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
+  Modify Access control list for Bucket or Files
+      s3cmd setacl s3://BUCKET[/OBJECT]
+  Modify Bucket Policy
+      s3cmd setpolicy FILE s3://BUCKET
+  Delete Bucket Policy
+      s3cmd delpolicy s3://BUCKET
+  Modify Bucket CORS
+      s3cmd setcors FILE s3://BUCKET
+  Delete Bucket CORS
+      s3cmd delcors s3://BUCKET
+  Modify Bucket Requester Pays policy
+      s3cmd payer s3://BUCKET
+  Show multipart uploads
+      s3cmd multipart s3://BUCKET [Id]
+  Abort a multipart upload
+      s3cmd abortmp s3://BUCKET/OBJECT Id
+  List parts of a multipart upload
+      s3cmd listmp s3://BUCKET/OBJECT Id
+  Enable/disable bucket access logging
+      s3cmd accesslog s3://BUCKET
+  Sign arbitrary string using the secret key
+      s3cmd sign STRING-TO-SIGN
+  Sign an S3 URL to provide limited public access with expiry
+      s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
+  Fix invalid file names in a bucket
+      s3cmd fixbucket s3://BUCKET[/PREFIX]
+  Create Website from bucket
+      s3cmd ws-create s3://BUCKET
+  Delete Website
+      s3cmd ws-delete s3://BUCKET
+  Info about Website
+      s3cmd ws-info s3://BUCKET
+  Set or delete expiration rule for the bucket
+      s3cmd expire s3://BUCKET
+  Upload a lifecycle policy for the bucket
+      s3cmd setlifecycle FILE s3://BUCKET
+  Get a lifecycle policy for the bucket
+      s3cmd getlifecycle s3://BUCKET
+  Remove a lifecycle policy for the bucket
+      s3cmd dellifecycle s3://BUCKET
+  List CloudFront distribution points
+      s3cmd cflist 
+  Display CloudFront distribution point parameters
+      s3cmd cfinfo [cf://DIST_ID]
+  Create CloudFront distribution point
+      s3cmd cfcreate s3://BUCKET
+  Delete CloudFront distribution point
+      s3cmd cfdelete cf://DIST_ID
+  Change CloudFront distribution point parameters
+      s3cmd cfmodify cf://DIST_ID
+  Display CloudFront invalidation request(s) status
+      s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]
+```
+
+## aws-cli
+
+[aws-cli](https://aws.amazon.com/cli/) is the official cli provided by AWS.
+
+```shell
+$ aws s3 ls s3://mybucket
+        LastWriteTime            Length Name
+        ------------             ------ ----
+                                PRE myfolder/
+2013-09-03 10:00:00           1234 myfile.txt
+
+$ aws s3 cp myfolder s3://mybucket/myfolder --recursive
+upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt
+upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt
+
+$ aws s3 sync myfolder s3://mybucket/myfolder --exclude *.tmp
+upload: myfolder/newfile.txt to s3://mybucket/myfolder/newfile.txt
+```
+
+## azcopy
+
+[azcopy](https://github.com/Azure/azure-storage-azcopy) is the new Azure Storage data transfer utility.
+
+```shell
+azcopy copy 'C:\myDirectory\myTextFile.txt' 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt'
+
+azcopy copy 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt' 'C:\myDirectory\myTextFile.txt'
+
+azcopy sync 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive
+```
+
+## gsutil
+
+[gsutil](https://cloud.google.com/storage/docs/gsutil) is a Python application that lets you access Cloud Storage from the command line.
+
+```shell
+gsutil cp [OPTION]... src_url dst_url
+gsutil cp [OPTION]... src_url... dst_url
+gsutil cp [OPTION]... -I dst_url
+
+gsutil mv [-p] src_url dst_url
+gsutil mv [-p] src_url... dst_url
+gsutil mv [-p] -I dst_url
+
+gsutil rm [-f] [-r] url...
+gsutil rm [-f] [-r] -I
+```
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.
+
+[`aws-cli`]: https://github.com/aws/aws-cli
+[`s3cmd`]: https://s3tools.org/s3cmd 
+[`azcopy`]: https://github.com/Azure/azure-storage-azcopy
+[`gcloud`]: https://cloud.google.com/sdk/docs/install

data/core/src/docs/rfcs/0429_init_from_iter.md

@@ -0,0 +1,107 @@
+- Proposal Name: `init_from_iter`
+- Start Date: 2022-07-10
+- RFC PR: [apache/opendal#429](https://github.com/apache/opendal/pull/429)
+- Tracking Issue: [apache/opendal#430](https://github.com/apache/opendal/issues/430)
+
+# Summary
+
+Allow initializing opendal operators from an iterator.
+
+# Motivation
+
+To init OpenDAL operators, users have to init an accessor first.
+
+```rust
+let root = &env::var("OPENDAL_S3_ROOT").unwrap_or_else(|_| "/".to_string());
+let root = format!("/{}/{}", root, uuid::Uuid::new_v4());
+
+let mut builder = opendal::services::s3::Backend::build();
+builder.root(&root);
+builder.bucket(&env::var("OPENDAL_S3_BUCKET").expect("OPENDAL_S3_BUCKET must set"));
+builder.endpoint(&env::var("OPENDAL_S3_ENDPOINT").unwrap_or_default());
+builder.access_key_id(&env::var("OPENDAL_S3_ACCESS_KEY_ID").unwrap_or_default());
+builder.secret_access_key(&env::var("OPENDAL_S3_SECRET_ACCESS_KEY").unwrap_or_default());
+builder
+    .server_side_encryption(&env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION").unwrap_or_default());
+builder.server_side_encryption_customer_algorithm(
+    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM").unwrap_or_default(),
+);
+builder.server_side_encryption_customer_key(
+    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY").unwrap_or_default(),
+);
+builder.server_side_encryption_customer_key_md5(
+    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5").unwrap_or_default(),
+);
+builder.server_side_encryption_aws_kms_key_id(
+    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID").unwrap_or_default(),
+);
+if env::var("OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE").unwrap_or_default() == "on" {
+    builder.enable_virtual_host_style();
+}
+Ok(Some(builder.finish().await?))
+```
+
+We can simplify this logic if opendal has its native `from_iter` support.
+
+# Guide-level explanation
+
+Users can init an operator like the following:
+
+```rust
+// OPENDAL_S3_BUCKET = <bucket>
+// OPENDAL_S3_ENDPOINT = <endpoint>
+let op = Operator::from_env(Scheme::S3)?;
+```
+
+Or from a prefixed env:
+
+```rust
+// OIL_PROFILE_<name>_S3_BUCKET = <bucket>
+// OIL_PROFILE_<name>_S3_ENDPOINT = <endpoint>
+let op = Operator::from_env(Scheme::S3, "OIL_PROFILE_<name>")?;
+```
+
+Also, we call the underlying function directly:
+
+```rust
+// var it: impl Iterator<Item=(String, String)>
+let op = Operator::from_iter(Scheme::S3, it)?;
+```
+
+# Reference-level explanation
+
+Internally, every service's backend will implement the following functions:
+
+```rust
+fn from_iter(it: impl Iterator<Item=(String, String)>) -> Backend {}
+```
+
+Note: it's not a public API of `Accessor`, and it will never be. Instead, we will use this function inside the crate to keep the ability to refactor or even remove it.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+## Connection string
+
+It sounds a good idea to implement something like:
+
+```rust
+let op = Operator::open("s3://bucket?region=test")?
+```
+
+But there are no valid use cases. Let's implement this in the future if needed.

data/core/src/docs/rfcs/0438_multipart.md

@@ -0,0 +1,163 @@
+- Proposal Name: `multipart`
+- Start Date: 2022-07-11
+- RFC PR: [apache/opendal#438](https://github.com/apache/opendal/pull/438)
+- Tracking Issue: [apache/opendal#439](https://github.com/apache/opendal/issues/439)
+
+# Summary
+
+Add multipart support in OpenDAL.
+
+# Motivation
+
+[Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) APIs are widely used in object storage services to upload large files concurrently and resumable.
+
+A successful multipart upload includes the following steps:
+
+- `CreateMultipartUpload`: Start a new multipart upload.
+- `UploadPart`: Upload a single part with the previously uploaded id.
+- `CompleteMultipartUpload`: Complete a multipart upload to get a regular object.
+
+To cancel a multipart upload, users need to call `AbortMultipartUpload`.
+
+Apart from those APIs, most object services also provide a list API to get the current multipart uploads status:
+
+- `ListMultipartUploads`: List current ongoing multipart uploads
+- `ListParts`: List already uploaded parts.
+
+Before `CompleteMultipartUpload` has been called, users can't read already uploaded parts.
+
+After `CompleteMultipartUpload` or `AbortMultipartUpload` has been called, all uploaded parts will be removed.
+
+Object storage services commonly allow 10000 parts, and every part will allow up to 5 GiB. This way, users can upload a file up to 48.8 TiB.
+
+OpenDAL users can upload objects larger than 5 GiB via supporting multipart uploads.
+
+# Guide-level explanation
+
+Users can start a multipart upload via:
+
+```rust
+let mp = op.object("path/to/file").create_multipart().await?;
+```
+
+Or build a multipart via already known upload id:
+
+```rust
+let mp = op.object("path/to/file").into_multipart("<upload_id>");
+```
+
+With `Multipart`, we can upload a new part:
+
+```rust
+let part = mp.write(part_number, content).await?;
+```
+
+After all parts have been uploaded, we can finish this upload:
+
+```rust
+let _ = mp.complete(parts).await?;
+```
+
+Or, we can abort already uploaded parts:
+
+```rust
+let _ = mp.abort().await?;
+```
+
+# Reference-level explanation
+
+`Accessor` will add the following APIs:
+
+```rust
+pub trait Accessor: Send + Sync + Debug {
+    async fn create_multipart(&self, args: &OpCreateMultipart) -> Result<String> {
+        let _ = args;
+        unimplemented!()
+    }
+
+    async fn write_multipart(&self, args: &OpWriteMultipart) -> Result<PartWriter> {
+        let _ = args;
+        unimplemented!()
+    }
+
+    async fn complete_multipart(&self, args: &OpCompleteMultipart) -> Result<()> {
+        let _ = args;
+        unimplemented!()
+    }
+
+    async fn abort_multipart(&self, args: &OpAbortMultipart) -> Result<()> {
+        let _ = args;
+        unimplemented!()
+    }
+}
+```
+
+While closing a `PartWriter`, a `Part` will be generated.
+
+`Operator` will build APIs based on `Accessor`:
+
+```rust
+impl Object {
+    async fn create_multipart(&self) -> Result<Multipart> {}
+    fn into_multipart(&self, upload_id: &str) -> Multipart {}
+}
+
+impl Multipart {
+    async fn write(&self, part_number: usize, bs: impl AsRef<[u8]>) -> Result<Part> {}
+    async fn writer(&self, part_number: usize, size: u64) -> Result<impl PartWrite> {}
+    async fn complete(&self, ps: &[Part]) -> Result<()> {}
+    async fn abort(&self) -> Result<()> {}
+}
+```
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## Why not add new object modes?
+
+It seems natural to add a new object mode like `multipart`.
+
+```rust
+pub enum ObjectMode {
+    FILE,
+    DIR,
+    MULTIPART,
+    Unknown,
+}
+```
+
+However, to make this work, we need big API breaks that introduce `mode` in Object.
+
+And we need to change every API call to accept `mode` as args.
+
+For example:
+
+```rust
+let _ = op.object("path/to/dir/").list(ObjectMODE::MULTIPART);
+let _ = op.object("path/to/file").stat(ObjectMODE::MULTIPART)
+```
+
+## Why not split Object into File and Dir?
+
+We can split `Object` into `File` and `Dir` to avoid requiring `mode` in API. There is a vast API breakage too.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Support list multipart uploads
+
+We can support listing multipart uploads to list ongoing multipart uploads so we can resume an upload or abort them.
+
+## Support list part
+
+We can support listing parts to list already uploaded parts for an upload.

data/core/src/docs/rfcs/0443_gateway.md

@@ -0,0 +1,73 @@
+- Proposal Name: `gateway`
+- Start Date: 2022-07-18
+- RFC PR: [apache/opendal#443](https://github.com/apache/opendal/pull/443)
+- Tracking Issue: [apache/opendal#444](https://github.com/apache/opendal/issues/444)
+
+# Summary
+
+Add Gateway for OpenDAL.
+
+# Motivation
+
+Our users want features like [S3 Proxy](https://github.com/gaul/s3proxy) and [minio Gateway](https://blog.min.io/deprecation-of-the-minio-gateway/) so that they can access all their data in the same way.
+
+By providing a native gateway, we can empower users to access different storage in the same API.
+
+# Guide-level explanation
+
+OpenDAL will provide a new binary called: `oay`. It's a shortcut of `OpenDAL Gateway`.
+
+Uses can install this binary via:
+
+```shell
+cargo install oay
+```
+
+Or using they favourite package management:
+
+```shell
+# Archlinux
+pacman -S oay
+# Debian / Ubuntu
+apt install oay
+# Rocky Linux / Fedora
+dnf install oay
+# macOS
+brew install oay
+```
+
+With `oay`, users can:
+
+- Serve `fs` backend with S3 compatible API.
+- Serve `s3` backend with Azblob API
+- Serve as a s3 signing services
+
+# Reference-level explanation
+
+`oay` will be a separate crate apart from `opendal` so we will not pollute the dependencies of `opendal`. But `oay` will be releases at the same time with the same version of `opendal`. That means `oay` will always use the same (latest) version of opendal.
+
+Most operations of `oay` should be trivial, we will propose new RFCs if requiring big changes.
+
+`oay` won't keep configuration. All config will go through environment.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+- [S3 Proxy](https://github.com/gaul/s3proxy)
+- [minio Gateway](https://blog.min.io/deprecation-of-the-minio-gateway/)
+- [oxyno-zeta/s3-proxy](https://github.com/oxyno-zeta/s3-proxy)
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None