opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/services/etcd/docs.md

@@ -0,0 +1,45 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the working directory of `OpenDAL`
+- `endpoints`: Set the network address of etcd servers
+- `username`: Set the username of Etcd
+- `password`: Set the password for authentication
+- `ca_path`: Set the ca path to the etcd connection
+- `cert_path`: Set the cert path to the etcd connection
+- `key_path`: Set the key path to the etcd connection
+
+You can refer to [`EtcdBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Etcd;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Etcd::default();
+
+    // this will build a Operator accessing etcd which runs on http://127.0.0.1:2379
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/foundationdb/docs.md

@@ -0,0 +1,42 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [ ] blocking
+
+**Note**: As for [Known Limitations - FoundationDB](https://apple.github.io/foundationdb/known-limitations), keys cannot exceed 10,000 bytes in size, and values cannot exceed 100,000 bytes in size. Errors will be raised by OpenDAL if these limits are exceeded.
+
+## Configuration
+
+- `root`: Set the work directory for this backend.
+- `config_path`: Set the configuration path for foundationdb. If not provided, the default configuration path will be used.
+
+You can refer to [`FoundationdbBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Foundationdb;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Foundationdb::default()
+        .config_path("/etc/foundationdb/foundationdb.conf");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/fs/docs.md

@@ -0,0 +1,49 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] append
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] ~~presign~~
+- [x] blocking
+
+## Configuration
+
+- `root`: Set the work dir for backend.
+- 
+You can refer to [`FsBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+
+```rust,no_run
+use std::sync::Arc;
+
+use anyhow::Result;
+use opendal::services::Fs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // Create fs backend builder.
+    let mut builder = Fs::default()
+        // Set the root for fs, all operations will happen under this root.
+        //
+        // NOTE: the root must be absolute path.
+        .root("/tmp");
+
+    // `Accessor` provides the low level APIs, we will use `Operator` normally.
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/ftp/docs.md

@@ -0,0 +1,42 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [x] list
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Configuration
+
+- `endpoint`: Set the endpoint for connection
+- `root`: Set the work directory for backend
+- `user`: Set the login user
+- `password`: Set the login password
+
+You can refer to [`FtpBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Ftp;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Ftp::default()
+        .endpoint("127.0.0.1");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/gcs/docs.md

@@ -0,0 +1,76 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [ ] rename
+- [x] list
+- [x] presign
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the work directory for backend
+- `bucket`: Set the container name for backend
+- `endpoint`: Customizable endpoint setting
+- `credential`: Service Account or External Account JSON, in base64
+- `credential_path`: local path to Service Account or External Account JSON file
+- `service_account`: name of Service Account
+- `predefined_acl`: Predefined ACL for GCS
+- `default_storage_class`: Default storage class for GCS
+
+Refer to public API docs for more information. For authentication related options, read on.
+
+## Options to authenticate to GCS
+
+OpenDAL supports the following authentication options:
+
+1. Provide a base64-ed JSON key string with `credential`
+2. Provide a JSON key file at explicit path with `credential_path`
+3. Provide a JSON key file at implicit path
+    - `GcsBackend` will attempt to load Service Account key from [ADC well-known places](https://cloud.google.com/docs/authentication/application-default-credentials).
+4. Fetch access token from [VM metadata](https://cloud.google.com/docs/authentication/rest#metadata-server)
+    - Only works when running inside Google Cloud.
+    - If a non-default Service Account name is required, set with `service_account`. Otherwise, nothing need to be set.
+5. A custom `TokenLoader` via `GcsBuilder.customized_token_loader()`
+
+Notes:
+
+- When a Service Account key is provided, it will be used to create access tokens (VM metadata will not be used).
+- Explicit Service Account key, in json or path, always take precedence over ADC-defined key paths.
+- Due to [limitation in GCS](https://cloud.google.com/storage/docs/authentication/signatures#signing-process), a private key is required to create Pre-signed URL. Currently, OpenDAL only supports Service Account key.
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Gcs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Gcs::default()
+       // set the storage bucket for OpenDAL
+       .bucket("test")
+       // set the working directory root for GCS
+       // all operations will happen within it
+       .root("/path/to/dir")
+       // set the credentials with service account
+       .credential("service account JSON in base64")
+       // set the predefined ACL for GCS
+       .predefined_acl("publicRead")
+       // set the default storage class for GCS
+       .default_storage_class("STANDARD");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/gdrive/docs.md

@@ -0,0 +1,65 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] delete
+- [x] create_dir
+- [x] list
+- [x] copy
+- [x] rename
+- [ ] batch
+
+
+# Configuration
+
+- `root`: Set the work directory for backend
+
+### Credentials related
+
+#### Just provide Access Token (Temporary)
+
+- `access_token`: set the access_token for google drive api
+Please notice its expiration.
+
+#### Or provide Client ID and Client Secret and refresh token (Long Term)
+
+If you want to let OpenDAL to refresh the access token automatically,
+please provide the following fields:
+
+- `refresh_token`: set the refresh_token for google drive api
+- `client_id`: set the client_id for google drive api
+- `client_secret`: set the client_secret for google drive api
+
+OpenDAL is a library, it cannot do the first step of OAuth2 for you.
+You need to get authorization code from user by calling GoogleDrive's authorize url
+and exchange it for refresh token.
+
+Make sure you have enabled Google Drive API in your Google Cloud Console.
+And your OAuth scope contains `https://www.googleapis.com/auth/drive`.
+
+Please refer to [GoogleDrive OAuth2 Flow](https://developers.google.com/identity/protocols/oauth2/)
+for more information.
+
+You can refer to [`GdriveBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Gdrive;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Gdrive::default()
+        .root("/test")
+        .access_token("<token>");
+
+    Ok(())
+}
+

data/core/src/services/ghac/docs.md

@@ -0,0 +1,84 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [ ] rename
+- [ ] list
+- [ ] presign
+- [ ] blocking
+
+## Notes
+
+This service is mainly provided by GitHub actions.
+
+Refer to [Caching dependencies to speed up workflows](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows) for more information.
+
+To make this service work as expected, please make sure to either call `endpoint` and `token` to
+configure the URL and credentials, or that the following environment has been setup correctly:
+
+- `ACTIONS_CACHE_URL`
+- `ACTIONS_RUNTIME_TOKEN`
+
+They can be exposed by following action:
+
+```yaml
+- name: Configure Cache Env
+  uses: actions/github-script@v6
+  with:
+    script: |
+      core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || '');
+      core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
+```
+
+To make `delete` work as expected, `GITHUB_TOKEN` should also be set via:
+
+```yaml
+env:
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+
+## Limitations
+
+Unlike other services, ghac doesn't support create empty files.
+We provide a `enable_create_simulation()` to support this operation but may result unexpected side effects.
+
+Also, `ghac` is a cache service which means the data store inside could
+be automatically evicted at any time.
+
+## Configuration
+
+- `root`: Set the work dir for backend.
+
+Refer to [`GhacBuilder`]'s public API docs for more information.
+
+## Example
+
+### Via Builder
+
+```no_run
+use std::sync::Arc;
+
+use anyhow::Result;
+use opendal::services::Ghac;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // Create ghac backend builder.
+    let mut builder = Ghac::default()
+        // Set the root for ghac, all operations will happen under this root.
+        //
+        // NOTE: the root must be absolute path.
+        .root("/path/to/dir");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/github/docs.md

@@ -0,0 +1,52 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [ ] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [x] list
+- [ ] presign
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the work directory for backend
+- `token`: Github access token
+- `owner`: Github owner
+- `repo`: Github repository
+
+You can refer to [`GithubBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Github;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Github::default()
+        // set the storage root for OpenDAL
+        .root("/")
+        // set the access token for Github API
+        .token("your_access_token")
+        // set the owner for Github
+        .owner("your_owner")
+        // set the repository for Github
+        .repo("your_repo");
+
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/gridfs/docs.md

@@ -0,0 +1,46 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the working directory of `OpenDAL`
+- `connection_string`: Set the connection string of mongodb server
+- `database`: Set the database of mongodb
+- `bucket`: Set the bucket of mongodb gridfs
+- `chunk_size`: Set the chunk size of mongodb gridfs
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Gridfs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Gridfs::default()
+        .root("/")
+        .connection_string("mongodb://myUser:myPassword@localhost:27017/myAuthDB")
+        .database("your_database")
+        .bucket("your_bucket")
+        // The chunk size in bytes used to break the user file into chunks.
+        .chunk_size(255);
+
+    let op = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/hdfs/docs.md

@@ -0,0 +1,140 @@
+A distributed file system that provides high-throughput access to application data.
+
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [x] rename
+- [x] list
+- [ ] ~~presign~~
+- [x] blocking
+- [x] append
+
+## Differences with webhdfs
+
+[Webhdfs][crate::services::Webhdfs] is powered by hdfs's RESTful HTTP API.
+
+## Features
+
+HDFS support needs to enable feature `services-hdfs`.
+
+## Configuration
+
+- `root`: Set the work dir for backend.
+- `name_node`: Set the name node for backend.
+- `kerberos_ticket_cache_path`: Set the kerberos ticket cache path for backend, this should be gotten by `klist` after `kinit`
+- `user`: Set the user for backend
+- `enable_append`: enable the append capacity. Default is false. 
+
+Refer to [`HdfsBuilder`]'s public API docs for more information.
+
+## Environment
+
+HDFS needs some environment set correctly.
+
+- `JAVA_HOME`: the path to java home, could be found via `java -XshowSettings:properties -version`
+- `HADOOP_HOME`: the path to hadoop home, opendal relays on this env to discover hadoop jars and set `CLASSPATH` automatically.
+
+Most of the time, setting `JAVA_HOME` and `HADOOP_HOME` is enough. But there are some edge cases:
+
+- If meeting errors like the following:
+
+```shell
+error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory
+```
+
+Java's lib are not including in pkg-config find path, please set `LD_LIBRARY_PATH`:
+
+```shell
+export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
+```
+
+The path of `libjvm.so` could be different, please keep an eye on it.
+
+- If meeting errors like the following:
+
+```shell
+(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
+```
+
+`CLASSPATH` is not set correctly or your hadoop installation is incorrect.
+
+To set `CLASSPATH`:
+```shell
+export CLASSPATH=$(find $HADOOP_HOME -iname "*.jar" | xargs echo | tr ' ' ':'):${CLASSPATH}
+```
+
+- If HDFS has High Availability (HA) enabled with multiple available NameNodes, some configuration is required:
+1. Obtain the entire HDFS config folder (usually located at HADOOP_HOME/etc/hadoop).
+2. Set the environment variable HADOOP_CONF_DIR to the path of this folder.
+```shell
+export HADOOP_CONF_DIR=<path of the config folder>
+```
+3. Append the HADOOP_CONF_DIR to the `CLASSPATH`
+```shell
+export CLASSPATH=$HADOOP_CONF_DIR:$HADOOP_CLASSPATH:$CLASSPATH
+```
+4. Use the `cluster_name` specified in the `core-site.xml` file (located in the HADOOP_CONF_DIR folder) to replace namenode:port.
+
+```ignore
+builder.name_node("hdfs://cluster_name");
+```
+
+### macOS Specific Note
+
+If you encounter an issue during the build process on macOS with an error message similar to:
+
+```shell
+ld: unknown file type in $HADOOP_HOME/lib/native/libhdfs.so.0.0.0
+clang: error: linker command failed with exit code 1 (use -v to see invocation)
+```
+This error is likely due to the fact that the official Hadoop build includes the libhdfs.so file for the x86-64 architecture, which is not compatible with aarch64 architecture required for MacOS.
+
+To resolve this issue, you can add hdrs as a dependency in your Rust application's Cargo.toml file, and enable the vendored feature:
+
+```toml
+[dependencies]
+hdrs = { version = "<version_number>", features = ["vendored"] }
+```
+Enabling the vendored feature ensures that hdrs includes the necessary libhdfs.so library built for the correct architecture.
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use std::sync::Arc;
+
+use anyhow::Result;
+use opendal::services::Hdfs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // Create fs backend builder.
+    let mut builder = Hdfs::default()
+        // Set the name node for hdfs.
+        // If the string starts with a protocol type such as file://, hdfs://, or gs://, this protocol type will be used.
+        .name_node("hdfs://127.0.0.1:9000")
+        // Set the root for hdfs, all operations will happen under this root.
+        //
+        // NOTE: the root must be absolute path.
+        .root("/tmp")
+        
+        // Enable the append capacity for hdfs. 
+        // 
+        // Note: HDFS run in non-distributed mode doesn't support append.
+        .enable_append(true);
+
+    // `Accessor` provides the low level APIs, we will use `Operator` normally.
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/hdfs_native/docs.md

@@ -0,0 +1,35 @@
+A distributed file system that provides high-throughput access to application data.
+Using [Native Rust HDFS client](https://github.com/Kimahriman/hdfs-native).
+
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] rename
+- [x] list
+- [x] blocking
+- [x] append
+
+## Differences with webhdfs
+
+[Webhdfs][crate::services::Webhdfs] is powered by hdfs's RESTful HTTP API.
+
+## Differences with hdfs
+
+[hdfs][crate::services::Hdfs] is powered by libhdfs and require the Java dependencies
+
+## Features
+
+HDFS-native support needs to enable feature `services-hdfs-native`.
+
+## Configuration
+
+- `root`: Set the work dir for backend.
+- `name_node`: Set the name node for backend.
+- `enable_append`: enable the append capacity. Default is false. 
+