opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/services/onedrive/docs.md

@@ -0,0 +1,115 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [ ] append
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] presign
+- [ ] blocking
+
+## Notes
+
+Currently, OpenDAL supports OneDrive Personal only.
+
+### Write Operations and OneDrive Behavior
+
+For write-related operations, such as:
+
+- write
+- rename
+- copy
+- create_dir
+
+OpenDAL's OneDrive service replaces the destination folder instead of rename it.
+
+### Consistency Issues with Concurrent Requests
+
+OneDrive does not guarantee consistency when handling a large number of concurrent requests write operations.
+
+In some extreme cases, OneDrive may acknowledge an operation as successful but fail to commit the changes.
+This inconsistency can cause subsequent operations to fail, returning errors like:
+
+- 400 Bad Request: OneDrive considers folders in the path are not there yet
+- 404 Not Found: OneDrive doesn't recognize the created folder
+- 409 Conflict: OneDrive can't replace an existing folder
+
+You should consider [`RetryLayer`] and monitor your operations carefully.
+
+## Configuration
+
+- `access_token`: Set a short-live access token for Microsoft Graph API (also, OneDrive API)
+- `refresh_token`: Set a long term access token for Microsoft Graph API
+- `client_id`: Set the client ID for a Microsoft Graph API application (available though Azure's registration portal)
+- `client_secret`: Set the client secret for a Microsoft Graph API application
+- `root`: Set the work directory for OneDrive backend
+- `enable_versioning`: Enable versioning support for OneDrive items
+
+The configuration for tokens is one of the following:
+* `access_token` only, for short-lived access. Once the `access_token` expires, you must recreate the operator with a new token.
+* `refresh_token`, `client_id`, and an optional `client_secret`, for long-lived access. The operator will automatically get and refresh the access token.
+
+## How to get tokens
+
+1. Create an application: navigate to [Microsoft Entra Admin Center](https://entra.microsoft.com/) -> Applications -> App Registrations -> New registration
+2. In "Supported account types", choose "Accounts in any organizational directory (Any Microsoft Entra ID tenant - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)"
+3. Or in an existing application -> manifest, make sure `signInAudience` is `"AzureADandPersonalMicrosoftAccount"`. That's because we're calling Graph API with `/common` path segment.
+4. The platform you choose determines whether you have to provide a `client_secret` or not. See [Public and confidential client apps](https://learn.microsoft.com/en-us/entra/identity-platform/msal-client-applications) for more information.
+    1. In short, if you choose "Mobile and desktop applications" or "Single-page application" (Public Client), you must not provide `client_secret`.
+    2. If you choose "Web" (Confidential Client), create a secret in "Certificates & secrets -> Client secrets -> New client secret", and provide it as `client_secret`.
+5. Follow the [code grant flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-auth-code-flow) or other flows to get the access_token. The minimum scope is `Files.ReadWrite`. And make sure the access token represents a user, because it's accessing the user's onedrive by `/me/drive`. So "client credentials flow" won't work.
+6. If you need `refresh_token` for long-lived access, add an additional `offline_access` scope.
+
+Read more at [`OnedriveBuilder`].
+
+## Example
+
+### Via Builder
+
+When you have a current access token: 
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Onedrive;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Onedrive::default()
+        .access_token("my_access_token")
+        .root("/root/folder/for/operator");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```
+
+When you have an Application with a refresh token:
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Onedrive;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Onedrive::default()
+        .refresh_token("my_refresh_token")
+        .client_id("my_client_id")
+        .root("/root/folder/for/operator");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```
+
+
+[conflict-behavior]: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0#instance-attributes

data/core/src/services/opfs/docs.md

@@ -0,0 +1,18 @@
+## Capabilities
+
+This service can be used to:
+
+- [ ] stat
+- [ ] read
+- [ ] write
+- [ ] create_dir
+- [ ] delete
+- [ ] copy
+- [ ] rename
+- [ ] list
+- [ ] presign
+- [ ] blocking
+
+## Configuration
+
+## Example

data/core/src/services/oss/docs.md

@@ -0,0 +1,74 @@
+
+
+# Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] append
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [ ] rename
+- [x] list
+- [x] presign
+- [ ] blocking
+
+# Configuration
+
+- `root`: Set the work dir for backend.
+- `bucket`: Set the container name for backend.
+- `endpoint`: Set the endpoint for backend.
+- `addressing_style`: Set the addressing style for endpoint.
+- `presign_endpoint`: Set the endpoint for presign.
+- `presign_addressing_style`: Set the addressing style for presign endpoint.
+- `access_key_id`: Set the access_key_id for backend.
+- `access_key_secret`: Set the access_key_secret for backend.
+- `role_arn`: Set the role of backend.
+- `oidc_token`: Set the oidc_token for backend.
+- `allow_anonymous`: Set the backend access OSS in anonymous way.
+
+Refer to [`OssBuilder`]'s public API docs for more information.
+
+# Example
+
+## Via Builder
+
+```rust,no_run
+use std::sync::Arc;
+
+use anyhow::Result;
+use opendal::services::Oss;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // Create OSS backend builder.
+    let mut builder = Oss::default()
+        // Set the root for oss, all operations will happen under this root.
+        //
+        // NOTE: the root must be absolute path.
+        .root("/path/to/dir")
+        // Set the bucket name, this is required.
+        .bucket("test")
+        // Set the endpoint.
+        //
+        // For example:
+        // - "https://oss-ap-northeast-1.aliyuncs.com"
+        // - "https://oss-hangzhou.aliyuncs.com"
+        .endpoint("https://oss-cn-beijing.aliyuncs.com")
+        // Set the access_key_id and access_key_secret.
+        //
+        // OpenDAL will try load credential from the env.
+        // If credential not set and no valid credential in env, OpenDAL will
+        // send request without signing like anonymous user.
+        .access_key_id("access_key_id")
+        .access_key_secret("access_key_secret");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/pcloud/docs.md

@@ -0,0 +1,51 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] presign
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the work directory for backend
+- `endpoint`: Pcloud bucket name
+- `username` Pcloud username
+- `password` Pcloud password
+
+You can refer to [`PcloudBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Pcloud;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Pcloud::default()
+        // set the storage bucket for OpenDAL
+        .root("/")
+        // set the bucket for OpenDAL
+        .endpoint("[https](https://api.pcloud.com)")
+        // set the username for OpenDAL
+        .username("opendal@gmail.com")
+        // set the password name for OpenDAL
+        .password("opendal");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/persy/docs.md

@@ -0,0 +1,43 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [ ] list
+- [ ] ~~presign~~
+- [x] blocking
+
+## Configuration
+
+- `datafile`: Set the path to the persy data file. The directory in the path must already exist.
+- `segment`: Set the name of the persy segment.
+- `index`: Set the name of the persy index.
+
+You can refer to [`PersyBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Persy;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Persy::default()
+        .datafile("./test.persy")
+        .segment("data")
+        .index("index");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/postgresql/docs.md

@@ -0,0 +1,47 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the working directory of `OpenDAL`
+- `connection_string`: Set the connection string of postgres server
+- `table`: Set the table of postgresql
+- `key_field`: Set the key field of postgresql
+- `value_field`: Set the value field of postgresql
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Postgresql;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Postgresql::default()
+        .root("/")
+        .connection_string("postgresql://you_username:your_password@127.0.0.1:5432/your_database")
+        .table("your_table")
+        // key field type in the table should be compatible with Rust's &str like text
+        .key_field("key")
+        // value field type in the table should be compatible with Rust's Vec<u8> like bytea
+        .value_field("value");
+
+    let op = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/redb/docs.md

@@ -0,0 +1,41 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [x] blocking
+
+## Configuration
+
+- `datadir`: Set the path to the redb data directory.
+- `table`: Set the table name for Redb.
+
+You can refer to [`RedbBuilder`]'s docs for more information.
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Redb;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Redb::default()
+        .datadir("/tmp/opendal/redb")
+        .table("opendal-redb");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/redis/docs.md

@@ -0,0 +1,43 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] delete
+- [ ] ~~create_dir~~
+- [ ] ~~copy~~
+- [ ] ~~rename~~
+- [ ] ~~list~~
+- [ ] ~~presign~~
+
+## Configuration
+
+- `root`: Set the working directory of `OpenDAL`
+- `endpoint`: Set the network address of redis server
+- `cluster_endpoints`: Set the network address of redis cluster server. This parameter is mutually exclusive with the `endpoint` parameter.
+- `username`: Set the username of Redis
+- `password`: Set the password for authentication
+- `db`: Set the DB of redis
+
+You can refer to [`RedisBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Redis;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Redis::default();
+
+    // this will build a Operator accessing Redis which runs on tcp://localhost:6379
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/rocksdb/docs.md

@@ -0,0 +1,54 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [x] blocking
+
+## Note
+
+OpenDAL will build rocksdb from source by default.
+
+To link with existing rocksdb lib, please set one of the following:
+
+- `ROCKSDB_LIB_DIR` to the dir that contains `librocksdb.so`
+- `ROCKSDB_STATIC` to the dir that contains `librocksdb.a`
+
+If the version of RocksDB is below 6.0, you may encounter compatibility
+issues. It is advisable to follow the steps provided in the [`INSTALL`](https://github.com/facebook/rocksdb/blob/main/INSTALL.md)
+file to build rocksdb, rather than relying on system libraries that
+may be outdated and incompatible.
+
+## Configuration
+
+- `root`: Set the working directory of `OpenDAL`
+- `datadir`: Set the path to the rocksdb data directory
+
+You can refer to [`RocksdbBuilder`]'s docs for more information.
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Rocksdb;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Rocksdb::default()
+        .datadir("/tmp/opendal/rocksdb");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/s3/compatible_services.md

@@ -0,0 +1,126 @@
+
+## Compatible Services
+
+### AWS S3
+
+[AWS S3](https://aws.amazon.com/s3/) is the default implementations of s3 services. Only `bucket` is required.
+
+```rust,ignore
+builder.bucket("<bucket_name>");
+```
+
+### Alibaba Object Storage Service (OSS)
+
+[OSS](https://www.alibabacloud.com/product/object-storage-service) is a s3 compatible service provided by [Alibaba Cloud](https://www.alibabacloud.com).
+
+To connect to OSS, we need to set:
+
+- `endpoint`: The endpoint of oss, for example: `https://oss-cn-hangzhou.aliyuncs.com`
+- `bucket`: The bucket name of oss.
+
+> OSS provide internal endpoint for used at alibabacloud internally, please visit [OSS Regions and endpoints](https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints) for more details.
+
+> OSS only supports the virtual host style, users could meet errors like:
+>
+> ```xml
+> <?xml version="1.0" encoding="UTF-8"?>
+> <Error>
+>  <Code>SecondLevelDomainForbidden</Code>
+>  <Message>The bucket you are attempting to access must be addressed using OSS third level domain.</Message>
+>  <RequestId>62A1C265292C0632377F021F</RequestId>
+>  <HostId>oss-cn-hangzhou.aliyuncs.com</HostId>
+> </Error>
+> ```
+>
+> In that case, please enable virtual host style for requesting.
+
+```rust,ignore
+builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com");
+builder.region("<region>");
+builder.bucket("<bucket_name>");
+builder.enable_virtual_host_style();
+```
+
+### Minio
+
+[minio](https://min.io/) is an open-source s3 compatible services.
+
+To connect to minio, we need to set:
+
+- `endpoint`: The endpoint of minio, for example: `http://127.0.0.1:9000`
+- `region`: The region of minio. If you don't care about it, just set it to "auto", it will be ignored.
+- `bucket`: The bucket name of minio.
+
+```rust,ignore
+builder.endpoint("http://127.0.0.1:9000");
+builder.region("<region>");
+builder.bucket("<bucket_name>");
+```
+
+### QingStor Object Storage
+
+[QingStor Object Storage](https://www.qingcloud.com/products/qingstor) is a S3-compatible service provided by [QingCloud](https://www.qingcloud.com/).
+
+To connect to QingStor Object Storage, we need to set:
+
+- `endpoint`: The endpoint of QingStor s3 compatible endpoint, for example: `https://s3.pek3b.qingstor.com`
+- `bucket`: The bucket name.
+
+### Scaleway Object Storage
+
+[Scaleway Object Storage](https://www.scaleway.com/en/object-storage/) is a S3-compatible and multi-AZ redundant object storage service.
+
+To connect to Scaleway Object Storage, we need to set:
+
+- `endpoint`: The endpoint of scaleway, for example: `https://s3.nl-ams.scw.cloud`
+- `region`: The region of scaleway.
+- `bucket`: The bucket name of scaleway.
+
+### Tencent Cloud Object Storage (COS)
+
+[COS](https://intl.cloud.tencent.com/products/cos) is a s3 compatible service provided by [Tencent Cloud](https://intl.cloud.tencent.com/).
+
+To connect to COS, we need to set:
+
+- `endpoint`: The endpoint of cos, for example: `https://cos.ap-beijing.myqcloud.com`
+- `bucket`: The bucket name of cos.
+
+### Wasabi Object Storage
+
+[Wasabi](https://wasabi.com/) is a s3 compatible service.
+
+> Cloud storage pricing that is 80% less than Amazon S3.
+
+To connect to wasabi, we need to set:
+
+- `endpoint`: The endpoint of wasabi, for example: `https://s3.us-east-2.wasabisys.com`
+- `bucket`: The bucket name of wasabi.
+
+> Refer to [What are the service URLs for Wasabi's different storage regions?](https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031) for more details.
+
+### Cloudflare R2
+
+[Cloudflare R2](https://developers.cloudflare.com/r2/) provides s3 compatible API.
+
+> Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.
+
+
+To connect to r2, we need to set:
+
+- `endpoint`: The endpoint of r2, for example: `https://<account_id>.r2.cloudflarestorage.com`
+- `bucket`: The bucket name of r2.
+- `region`: When you create a new bucket, the data location is set to Automatic by default. So please use `auto` for region.
+- `batch_max_operations`: R2's delete objects will return `Internal Error` if the batch is larger than `700`. Please set this value `<= 700` to make sure batch delete work as expected.
+- `enable_exact_buf_write`: R2 requires the non-tailing parts size to be exactly the same. Please enable this option to avoid the error `All non-trailing parts must have the same length`.
+
+### Google Cloud Storage XML API
+[Google Cloud Storage XML API](https://cloud.google.com/storage/docs/xml-api/overview) provides s3 compatible API.
+- `endpoint`: The endpoint of Google Cloud Storage XML API, for example: `https://storage.googleapis.com`
+- `bucket`: The bucket name.
+- To access GCS via S3 API, please enable `features = ["native-tls"]` in your `Cargo.toml` to avoid connection being reset when using `rustls`. Tracking in <https://github.com/seanmonstar/reqwest/issues/1809>
+
+### Ceph Rados Gateway
+Ceph supports a RESTful API that is compatible with the basic data access model of the Amazon S3 API.
+
+For more information, refer: <https://docs.ceph.com/en/latest/radosgw/s3/>
+