opendal 0.1.6.pre.rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.standard.yml +20 -0
- data/.tool-versions +1 -0
- data/.yardopts +1 -0
- data/Cargo.toml +65 -0
- data/DEPENDENCIES.md +9 -0
- data/DEPENDENCIES.rust.tsv +277 -0
- data/Gemfile +35 -0
- data/README.md +159 -0
- data/Rakefile +149 -0
- data/build.rs +22 -0
- data/core/CHANGELOG.md +4929 -0
- data/core/CONTRIBUTING.md +61 -0
- data/core/Cargo.lock +10259 -0
- data/core/Cargo.toml +437 -0
- data/core/DEPENDENCIES.md +3 -0
- data/core/DEPENDENCIES.rust.tsv +185 -0
- data/core/LICENSE +201 -0
- data/core/README.md +228 -0
- data/core/benches/README.md +18 -0
- data/core/benches/ops/README.md +26 -0
- data/core/benches/ops/main.rs +25 -0
- data/core/benches/ops/read.rs +100 -0
- data/core/benches/ops/utils.rs +59 -0
- data/core/benches/ops/write.rs +106 -0
- data/core/benches/types/README.md +9 -0
- data/core/benches/types/buffer.rs +114 -0
- data/core/benches/types/main.rs +23 -0
- data/core/benches/types/tasks.rs +64 -0
- data/core/benches/vs_fs/Cargo.toml +32 -0
- data/core/benches/vs_fs/README.md +35 -0
- data/core/benches/vs_fs/src/main.rs +83 -0
- data/core/benches/vs_s3/Cargo.toml +38 -0
- data/core/benches/vs_s3/README.md +55 -0
- data/core/benches/vs_s3/src/main.rs +123 -0
- data/core/edge/README.md +3 -0
- data/core/edge/file_write_on_full_disk/Cargo.toml +31 -0
- data/core/edge/file_write_on_full_disk/README.md +14 -0
- data/core/edge/file_write_on_full_disk/src/main.rs +43 -0
- data/core/edge/s3_aws_assume_role_with_web_identity/Cargo.toml +30 -0
- data/core/edge/s3_aws_assume_role_with_web_identity/README.md +18 -0
- data/core/edge/s3_aws_assume_role_with_web_identity/src/main.rs +34 -0
- data/core/edge/s3_read_on_wasm/.gitignore +3 -0
- data/core/edge/s3_read_on_wasm/Cargo.toml +38 -0
- data/core/edge/s3_read_on_wasm/README.md +42 -0
- data/core/edge/s3_read_on_wasm/src/lib.rs +60 -0
- data/core/edge/s3_read_on_wasm/webdriver.json +15 -0
- data/core/examples/README.md +23 -0
- data/core/examples/basic/Cargo.toml +29 -0
- data/core/examples/basic/README.md +15 -0
- data/core/examples/basic/src/main.rs +51 -0
- data/core/examples/concurrent-upload/Cargo.toml +29 -0
- data/core/examples/concurrent-upload/README.md +15 -0
- data/core/examples/concurrent-upload/src/main.rs +68 -0
- data/core/examples/multipart-upload/Cargo.toml +29 -0
- data/core/examples/multipart-upload/README.md +15 -0
- data/core/examples/multipart-upload/src/main.rs +56 -0
- data/core/fuzz/.gitignore +5 -0
- data/core/fuzz/Cargo.toml +92 -0
- data/core/fuzz/README.md +68 -0
- data/core/fuzz/fuzz_reader.rs +102 -0
- data/core/fuzz/fuzz_writer.rs +123 -0
- data/core/src/blocking/delete.rs +74 -0
- data/core/src/blocking/list.rs +71 -0
- data/core/src/blocking/mod.rs +33 -0
- data/core/src/blocking/operator.rs +729 -0
- data/core/src/blocking/read/buffer_iterator.rs +66 -0
- data/core/src/blocking/read/mod.rs +27 -0
- data/core/src/blocking/read/reader.rs +124 -0
- data/core/src/blocking/read/std_bytes_iterator.rs +69 -0
- data/core/src/blocking/read/std_reader.rs +95 -0
- data/core/src/blocking/write/mod.rs +22 -0
- data/core/src/blocking/write/std_writer.rs +82 -0
- data/core/src/blocking/write/writer.rs +109 -0
- data/core/src/docs/comparisons/mod.rs +30 -0
- data/core/src/docs/comparisons/vs_object_store.md +183 -0
- data/core/src/docs/concepts.rs +135 -0
- data/core/src/docs/internals/accessor.rs +306 -0
- data/core/src/docs/internals/layer.rs +42 -0
- data/core/src/docs/internals/mod.rs +62 -0
- data/core/src/docs/mod.rs +43 -0
- data/core/src/docs/performance/concurrent_write.md +101 -0
- data/core/src/docs/performance/http_optimization.md +124 -0
- data/core/src/docs/performance/mod.rs +32 -0
- data/core/src/docs/rfcs/0000_example.md +74 -0
- data/core/src/docs/rfcs/0000_foyer_integration.md +111 -0
- data/core/src/docs/rfcs/0041_object_native_api.md +185 -0
- data/core/src/docs/rfcs/0044_error_handle.md +198 -0
- data/core/src/docs/rfcs/0057_auto_region.md +160 -0
- data/core/src/docs/rfcs/0069_object_stream.md +145 -0
- data/core/src/docs/rfcs/0090_limited_reader.md +155 -0
- data/core/src/docs/rfcs/0112_path_normalization.md +79 -0
- data/core/src/docs/rfcs/0191_async_streaming_io.md +328 -0
- data/core/src/docs/rfcs/0203_remove_credential.md +96 -0
- data/core/src/docs/rfcs/0221_create_dir.md +89 -0
- data/core/src/docs/rfcs/0247_retryable_error.md +87 -0
- data/core/src/docs/rfcs/0293_object_id.md +67 -0
- data/core/src/docs/rfcs/0337_dir_entry.md +191 -0
- data/core/src/docs/rfcs/0409_accessor_capabilities.md +67 -0
- data/core/src/docs/rfcs/0413_presign.md +154 -0
- data/core/src/docs/rfcs/0423_command_line_interface.md +268 -0
- data/core/src/docs/rfcs/0429_init_from_iter.md +107 -0
- data/core/src/docs/rfcs/0438_multipart.md +163 -0
- data/core/src/docs/rfcs/0443_gateway.md +73 -0
- data/core/src/docs/rfcs/0501_new_builder.md +111 -0
- data/core/src/docs/rfcs/0554_write_refactor.md +96 -0
- data/core/src/docs/rfcs/0561_list_metadata_reuse.md +210 -0
- data/core/src/docs/rfcs/0599_blocking_api.md +157 -0
- data/core/src/docs/rfcs/0623_redis_service.md +300 -0
- data/core/src/docs/rfcs/0627_split_capabilities.md +89 -0
- data/core/src/docs/rfcs/0661_path_in_accessor.md +126 -0
- data/core/src/docs/rfcs/0793_generic_kv_services.md +209 -0
- data/core/src/docs/rfcs/0926_object_reader.md +93 -0
- data/core/src/docs/rfcs/0977_refactor_error.md +151 -0
- data/core/src/docs/rfcs/1085_object_handler.md +73 -0
- data/core/src/docs/rfcs/1391_object_metadataer.md +110 -0
- data/core/src/docs/rfcs/1398_query_based_metadata.md +125 -0
- data/core/src/docs/rfcs/1420_object_writer.md +147 -0
- data/core/src/docs/rfcs/1477_remove_object_concept.md +159 -0
- data/core/src/docs/rfcs/1735_operation_extension.md +117 -0
- data/core/src/docs/rfcs/2083_writer_sink_api.md +106 -0
- data/core/src/docs/rfcs/2133_append_api.md +88 -0
- data/core/src/docs/rfcs/2299_chain_based_operator_api.md +99 -0
- data/core/src/docs/rfcs/2602_object_versioning.md +138 -0
- data/core/src/docs/rfcs/2758_merge_append_into_write.md +79 -0
- data/core/src/docs/rfcs/2774_lister_api.md +66 -0
- data/core/src/docs/rfcs/2779_list_with_metakey.md +143 -0
- data/core/src/docs/rfcs/2852_native_capability.md +58 -0
- data/core/src/docs/rfcs/2884_merge_range_read_into_read.md +80 -0
- data/core/src/docs/rfcs/3017_remove_write_copy_from.md +94 -0
- data/core/src/docs/rfcs/3197_config.md +237 -0
- data/core/src/docs/rfcs/3232_align_list_api.md +69 -0
- data/core/src/docs/rfcs/3243_list_prefix.md +128 -0
- data/core/src/docs/rfcs/3356_lazy_reader.md +111 -0
- data/core/src/docs/rfcs/3526_list_recursive.md +59 -0
- data/core/src/docs/rfcs/3574_concurrent_stat_in_list.md +80 -0
- data/core/src/docs/rfcs/3734_buffered_reader.md +64 -0
- data/core/src/docs/rfcs/3898_concurrent_writer.md +66 -0
- data/core/src/docs/rfcs/3911_deleter_api.md +165 -0
- data/core/src/docs/rfcs/4382_range_based_read.md +213 -0
- data/core/src/docs/rfcs/4638_executor.md +215 -0
- data/core/src/docs/rfcs/5314_remove_metakey.md +120 -0
- data/core/src/docs/rfcs/5444_operator_from_uri.md +162 -0
- data/core/src/docs/rfcs/5479_context.md +140 -0
- data/core/src/docs/rfcs/5485_conditional_reader.md +112 -0
- data/core/src/docs/rfcs/5495_list_with_deleted.md +81 -0
- data/core/src/docs/rfcs/5556_write_returns_metadata.md +121 -0
- data/core/src/docs/rfcs/5871_read_returns_metadata.md +112 -0
- data/core/src/docs/rfcs/6189_remove_native_blocking.md +106 -0
- data/core/src/docs/rfcs/6209_glob_support.md +132 -0
- data/core/src/docs/rfcs/6213_options_api.md +142 -0
- data/core/src/docs/rfcs/README.md +62 -0
- data/core/src/docs/rfcs/mod.rs +278 -0
- data/core/src/docs/upgrade.md +1556 -0
- data/core/src/layers/async_backtrace.rs +174 -0
- data/core/src/layers/await_tree.rs +202 -0
- data/core/src/layers/capability_check.rs +239 -0
- data/core/src/layers/chaos.rs +170 -0
- data/core/src/layers/complete.rs +385 -0
- data/core/src/layers/concurrent_limit.rs +322 -0
- data/core/src/layers/correctness_check.rs +440 -0
- data/core/src/layers/dtrace.rs +294 -0
- data/core/src/layers/error_context.rs +310 -0
- data/core/src/layers/fastmetrics.rs +525 -0
- data/core/src/layers/fastrace.rs +271 -0
- data/core/src/layers/http_client.rs +206 -0
- data/core/src/layers/immutable_index.rs +408 -0
- data/core/src/layers/logging.rs +842 -0
- data/core/src/layers/metrics.rs +182 -0
- data/core/src/layers/mime_guess.rs +199 -0
- data/core/src/layers/mod.rs +130 -0
- data/core/src/layers/observe/metrics.rs +936 -0
- data/core/src/layers/observe/mod.rs +93 -0
- data/core/src/layers/otelmetrics.rs +496 -0
- data/core/src/layers/oteltrace.rs +203 -0
- data/core/src/layers/prometheus.rs +686 -0
- data/core/src/layers/prometheus_client.rs +519 -0
- data/core/src/layers/retry.rs +933 -0
- data/core/src/layers/throttle.rs +204 -0
- data/core/src/layers/timeout.rs +513 -0
- data/core/src/layers/tracing.rs +349 -0
- data/core/src/layers/type_eraser.rs +91 -0
- data/core/src/lib.rs +204 -0
- data/core/src/raw/accessor.rs +856 -0
- data/core/src/raw/adapters/kv/api.rs +164 -0
- data/core/src/raw/adapters/kv/backend.rs +253 -0
- data/core/src/raw/adapters/kv/mod.rs +31 -0
- data/core/src/raw/adapters/mod.rs +50 -0
- data/core/src/raw/adapters/typed_kv/api.rs +171 -0
- data/core/src/raw/adapters/typed_kv/backend.rs +279 -0
- data/core/src/raw/adapters/typed_kv/mod.rs +29 -0
- data/core/src/raw/atomic_util.rs +57 -0
- data/core/src/raw/azure.rs +570 -0
- data/core/src/raw/chrono_util.rs +109 -0
- data/core/src/raw/enum_utils.rs +201 -0
- data/core/src/raw/futures_util.rs +470 -0
- data/core/src/raw/http_util/body.rs +144 -0
- data/core/src/raw/http_util/bytes_content_range.rs +239 -0
- data/core/src/raw/http_util/bytes_range.rs +260 -0
- data/core/src/raw/http_util/client.rs +276 -0
- data/core/src/raw/http_util/error.rs +68 -0
- data/core/src/raw/http_util/header.rs +356 -0
- data/core/src/raw/http_util/mod.rs +78 -0
- data/core/src/raw/http_util/multipart.rs +1180 -0
- data/core/src/raw/http_util/uri.rs +190 -0
- data/core/src/raw/layer.rs +295 -0
- data/core/src/raw/mod.rs +101 -0
- data/core/src/raw/oio/buf/flex_buf.rs +118 -0
- data/core/src/raw/oio/buf/mod.rs +25 -0
- data/core/src/raw/oio/buf/pooled_buf.rs +126 -0
- data/core/src/raw/oio/buf/queue_buf.rs +117 -0
- data/core/src/raw/oio/delete/api.rs +102 -0
- data/core/src/raw/oio/delete/batch_delete.rs +127 -0
- data/core/src/raw/oio/delete/mod.rs +30 -0
- data/core/src/raw/oio/delete/one_shot_delete.rs +79 -0
- data/core/src/raw/oio/entry.rs +89 -0
- data/core/src/raw/oio/list/api.rs +69 -0
- data/core/src/raw/oio/list/flat_list.rs +137 -0
- data/core/src/raw/oio/list/hierarchy_list.rs +135 -0
- data/core/src/raw/oio/list/mod.rs +35 -0
- data/core/src/raw/oio/list/page_list.rs +105 -0
- data/core/src/raw/oio/list/prefix_list.rs +64 -0
- data/core/src/raw/oio/mod.rs +40 -0
- data/core/src/raw/oio/read/api.rs +119 -0
- data/core/src/raw/oio/read/mod.rs +21 -0
- data/core/src/raw/oio/write/api.rs +103 -0
- data/core/src/raw/oio/write/append_write.rs +111 -0
- data/core/src/raw/oio/write/block_write.rs +405 -0
- data/core/src/raw/oio/write/mod.rs +42 -0
- data/core/src/raw/oio/write/multipart_write.rs +518 -0
- data/core/src/raw/oio/write/one_shot_write.rs +77 -0
- data/core/src/raw/oio/write/position_write.rs +284 -0
- data/core/src/raw/operation.rs +88 -0
- data/core/src/raw/ops.rs +917 -0
- data/core/src/raw/path.rs +451 -0
- data/core/src/raw/path_cache.rs +244 -0
- data/core/src/raw/rps.rs +249 -0
- data/core/src/raw/serde_util.rs +423 -0
- data/core/src/raw/std_io_util.rs +65 -0
- data/core/src/raw/tests/mod.rs +30 -0
- data/core/src/raw/tests/read.rs +116 -0
- data/core/src/raw/tests/utils.rs +80 -0
- data/core/src/raw/tests/write.rs +79 -0
- data/core/src/raw/tokio_util.rs +24 -0
- data/core/src/raw/version.rs +19 -0
- data/core/src/services/aliyun_drive/backend.rs +421 -0
- data/core/src/services/aliyun_drive/config.rs +72 -0
- data/core/src/services/aliyun_drive/core.rs +651 -0
- data/core/src/services/aliyun_drive/delete.rs +51 -0
- data/core/src/services/aliyun_drive/docs.md +61 -0
- data/core/src/services/aliyun_drive/error.rs +56 -0
- data/core/src/services/aliyun_drive/lister.rs +134 -0
- data/core/src/services/aliyun_drive/mod.rs +39 -0
- data/core/src/services/aliyun_drive/writer.rs +114 -0
- data/core/src/services/alluxio/backend.rs +257 -0
- data/core/src/services/alluxio/config.rs +50 -0
- data/core/src/services/alluxio/core.rs +367 -0
- data/core/src/services/alluxio/delete.rs +38 -0
- data/core/src/services/alluxio/docs.md +45 -0
- data/core/src/services/alluxio/error.rs +99 -0
- data/core/src/services/alluxio/lister.rs +73 -0
- data/core/src/services/alluxio/mod.rs +39 -0
- data/core/src/services/alluxio/writer.rs +74 -0
- data/core/src/services/azblob/backend.rs +594 -0
- data/core/src/services/azblob/config.rs +220 -0
- data/core/src/services/azblob/core.rs +937 -0
- data/core/src/services/azblob/delete.rs +108 -0
- data/core/src/services/azblob/docs.md +77 -0
- data/core/src/services/azblob/error.rs +164 -0
- data/core/src/services/azblob/lister.rs +107 -0
- data/core/src/services/azblob/mod.rs +38 -0
- data/core/src/services/azblob/writer.rs +177 -0
- data/core/src/services/azdls/backend.rs +435 -0
- data/core/src/services/azdls/config.rs +89 -0
- data/core/src/services/azdls/core.rs +388 -0
- data/core/src/services/azdls/delete.rs +48 -0
- data/core/src/services/azdls/docs.md +73 -0
- data/core/src/services/azdls/error.rs +107 -0
- data/core/src/services/azdls/lister.rs +165 -0
- data/core/src/services/azdls/mod.rs +38 -0
- data/core/src/services/azdls/writer.rs +129 -0
- data/core/src/services/azfile/backend.rs +373 -0
- data/core/src/services/azfile/config.rs +61 -0
- data/core/src/services/azfile/core.rs +435 -0
- data/core/src/services/azfile/delete.rs +51 -0
- data/core/src/services/azfile/docs.md +65 -0
- data/core/src/services/azfile/error.rs +108 -0
- data/core/src/services/azfile/lister.rs +217 -0
- data/core/src/services/azfile/mod.rs +39 -0
- data/core/src/services/azfile/writer.rs +92 -0
- data/core/src/services/b2/backend.rs +434 -0
- data/core/src/services/b2/config.rs +64 -0
- data/core/src/services/b2/core.rs +742 -0
- data/core/src/services/b2/delete.rs +56 -0
- data/core/src/services/b2/docs.md +54 -0
- data/core/src/services/b2/error.rs +132 -0
- data/core/src/services/b2/lister.rs +110 -0
- data/core/src/services/b2/mod.rs +39 -0
- data/core/src/services/b2/writer.rs +189 -0
- data/core/src/services/cacache/backend.rs +160 -0
- data/core/src/services/cacache/config.rs +28 -0
- data/core/src/services/cacache/core.rs +96 -0
- data/core/src/services/cacache/delete.rs +39 -0
- data/core/src/services/cacache/docs.md +38 -0
- data/core/src/services/cacache/mod.rs +34 -0
- data/core/src/services/cacache/writer.rs +61 -0
- data/core/src/services/cloudflare_kv/backend.rs +513 -0
- data/core/src/services/cloudflare_kv/config.rs +55 -0
- data/core/src/services/cloudflare_kv/core.rs +168 -0
- data/core/src/services/cloudflare_kv/delete.rs +119 -0
- data/core/src/services/cloudflare_kv/docs.md +21 -0
- data/core/src/services/cloudflare_kv/error.rs +79 -0
- data/core/src/services/cloudflare_kv/lister.rs +170 -0
- data/core/src/services/cloudflare_kv/mod.rs +39 -0
- data/core/src/services/cloudflare_kv/model.rs +76 -0
- data/core/src/services/cloudflare_kv/writer.rs +68 -0
- data/core/src/services/compfs/backend.rs +290 -0
- data/core/src/services/compfs/config.rs +30 -0
- data/core/src/services/compfs/core.rs +159 -0
- data/core/src/services/compfs/delete.rs +53 -0
- data/core/src/services/compfs/lister.rs +98 -0
- data/core/src/services/compfs/mod.rs +38 -0
- data/core/src/services/compfs/reader.rs +79 -0
- data/core/src/services/compfs/writer.rs +90 -0
- data/core/src/services/cos/backend.rs +442 -0
- data/core/src/services/cos/config.rs +54 -0
- data/core/src/services/cos/core.rs +761 -0
- data/core/src/services/cos/delete.rs +48 -0
- data/core/src/services/cos/docs.md +55 -0
- data/core/src/services/cos/error.rs +105 -0
- data/core/src/services/cos/lister.rs +237 -0
- data/core/src/services/cos/mod.rs +39 -0
- data/core/src/services/cos/writer.rs +234 -0
- data/core/src/services/d1/backend.rs +330 -0
- data/core/src/services/d1/config.rs +55 -0
- data/core/src/services/d1/docs.md +48 -0
- data/core/src/services/d1/error.rs +79 -0
- data/core/src/services/d1/mod.rs +29 -0
- data/core/src/services/d1/model.rs +125 -0
- data/core/src/services/dashmap/backend.rs +203 -0
- data/core/src/services/dashmap/config.rs +37 -0
- data/core/src/services/dashmap/core.rs +61 -0
- data/core/src/services/dashmap/delete.rs +40 -0
- data/core/src/services/dashmap/docs.md +38 -0
- data/core/src/services/dashmap/lister.rs +63 -0
- data/core/src/services/dashmap/mod.rs +36 -0
- data/core/src/services/dashmap/writer.rs +87 -0
- data/core/src/services/dbfs/backend.rs +258 -0
- data/core/src/services/dbfs/config.rs +48 -0
- data/core/src/services/dbfs/core.rs +191 -0
- data/core/src/services/dbfs/delete.rs +49 -0
- data/core/src/services/dbfs/docs.md +57 -0
- data/core/src/services/dbfs/error.rs +74 -0
- data/core/src/services/dbfs/lister.rs +96 -0
- data/core/src/services/dbfs/mod.rs +38 -0
- data/core/src/services/dbfs/writer.rs +64 -0
- data/core/src/services/dropbox/backend.rs +187 -0
- data/core/src/services/dropbox/builder.rs +222 -0
- data/core/src/services/dropbox/config.rs +47 -0
- data/core/src/services/dropbox/core.rs +496 -0
- data/core/src/services/dropbox/delete.rs +54 -0
- data/core/src/services/dropbox/docs.md +64 -0
- data/core/src/services/dropbox/error.rs +85 -0
- data/core/src/services/dropbox/lister.rs +117 -0
- data/core/src/services/dropbox/mod.rs +40 -0
- data/core/src/services/dropbox/writer.rs +51 -0
- data/core/src/services/etcd/backend.rs +345 -0
- data/core/src/services/etcd/config.rs +86 -0
- data/core/src/services/etcd/core.rs +143 -0
- data/core/src/services/etcd/deleter.rs +41 -0
- data/core/src/services/etcd/docs.md +45 -0
- data/core/src/services/etcd/error.rs +26 -0
- data/core/src/services/etcd/lister.rs +79 -0
- data/core/src/services/etcd/mod.rs +36 -0
- data/core/src/services/etcd/writer.rs +61 -0
- data/core/src/services/foundationdb/backend.rs +171 -0
- data/core/src/services/foundationdb/config.rs +45 -0
- data/core/src/services/foundationdb/docs.md +42 -0
- data/core/src/services/foundationdb/mod.rs +24 -0
- data/core/src/services/fs/backend.rs +299 -0
- data/core/src/services/fs/config.rs +33 -0
- data/core/src/services/fs/core.rs +227 -0
- data/core/src/services/fs/delete.rs +53 -0
- data/core/src/services/fs/docs.md +49 -0
- data/core/src/services/fs/error.rs +31 -0
- data/core/src/services/fs/lister.rs +81 -0
- data/core/src/services/fs/mod.rs +40 -0
- data/core/src/services/fs/reader.rs +83 -0
- data/core/src/services/fs/writer.rs +212 -0
- data/core/src/services/ftp/backend.rs +388 -0
- data/core/src/services/ftp/config.rs +46 -0
- data/core/src/services/ftp/core.rs +136 -0
- data/core/src/services/ftp/delete.rs +62 -0
- data/core/src/services/ftp/docs.md +42 -0
- data/core/src/services/ftp/err.rs +47 -0
- data/core/src/services/ftp/lister.rs +72 -0
- data/core/src/services/ftp/mod.rs +41 -0
- data/core/src/services/ftp/reader.rs +84 -0
- data/core/src/services/ftp/writer.rs +122 -0
- data/core/src/services/gcs/backend.rs +499 -0
- data/core/src/services/gcs/config.rs +168 -0
- data/core/src/services/gcs/core.rs +1079 -0
- data/core/src/services/gcs/delete.rs +98 -0
- data/core/src/services/gcs/docs.md +76 -0
- data/core/src/services/gcs/error.rs +122 -0
- data/core/src/services/gcs/lister.rs +136 -0
- data/core/src/services/gcs/mod.rs +40 -0
- data/core/src/services/gcs/uri.rs +75 -0
- data/core/src/services/gcs/writer.rs +163 -0
- data/core/src/services/gdrive/backend.rs +176 -0
- data/core/src/services/gdrive/builder.rs +228 -0
- data/core/src/services/gdrive/config.rs +47 -0
- data/core/src/services/gdrive/core.rs +499 -0
- data/core/src/services/gdrive/delete.rs +57 -0
- data/core/src/services/gdrive/docs.md +65 -0
- data/core/src/services/gdrive/error.rs +80 -0
- data/core/src/services/gdrive/lister.rs +110 -0
- data/core/src/services/gdrive/mod.rs +40 -0
- data/core/src/services/gdrive/writer.rs +77 -0
- data/core/src/services/ghac/backend.rs +285 -0
- data/core/src/services/ghac/config.rs +36 -0
- data/core/src/services/ghac/core.rs +459 -0
- data/core/src/services/ghac/docs.md +84 -0
- data/core/src/services/ghac/error.rs +52 -0
- data/core/src/services/ghac/mod.rs +35 -0
- data/core/src/services/ghac/writer.rs +201 -0
- data/core/src/services/github/backend.rs +285 -0
- data/core/src/services/github/config.rs +59 -0
- data/core/src/services/github/core.rs +351 -0
- data/core/src/services/github/delete.rs +41 -0
- data/core/src/services/github/docs.md +52 -0
- data/core/src/services/github/error.rs +101 -0
- data/core/src/services/github/lister.rs +112 -0
- data/core/src/services/github/mod.rs +38 -0
- data/core/src/services/github/writer.rs +51 -0
- data/core/src/services/gridfs/backend.rs +166 -0
- data/core/src/services/gridfs/config.rs +50 -0
- data/core/src/services/gridfs/core.rs +154 -0
- data/core/src/services/gridfs/docs.md +46 -0
- data/core/src/services/gridfs/mod.rs +26 -0
- data/core/src/services/hdfs/backend.rs +413 -0
- data/core/src/services/hdfs/config.rs +59 -0
- data/core/src/services/hdfs/delete.rs +62 -0
- data/core/src/services/hdfs/docs.md +140 -0
- data/core/src/services/hdfs/lister.rs +70 -0
- data/core/src/services/hdfs/mod.rs +36 -0
- data/core/src/services/hdfs/reader.rs +79 -0
- data/core/src/services/hdfs/writer.rs +104 -0
- data/core/src/services/hdfs_native/backend.rs +340 -0
- data/core/src/services/hdfs_native/config.rs +45 -0
- data/core/src/services/hdfs_native/delete.rs +47 -0
- data/core/src/services/hdfs_native/docs.md +35 -0
- data/core/src/services/hdfs_native/error.rs +59 -0
- data/core/src/services/hdfs_native/lister.rs +85 -0
- data/core/src/services/hdfs_native/mod.rs +39 -0
- data/core/src/services/hdfs_native/reader.rs +62 -0
- data/core/src/services/hdfs_native/writer.rs +61 -0
- data/core/src/services/http/backend.rs +291 -0
- data/core/src/services/http/config.rs +49 -0
- data/core/src/services/http/core.rs +125 -0
- data/core/src/services/http/docs.md +45 -0
- data/core/src/services/http/error.rs +53 -0
- data/core/src/services/http/mod.rs +32 -0
- data/core/src/services/huggingface/backend.rs +289 -0
- data/core/src/services/huggingface/config.rs +75 -0
- data/core/src/services/huggingface/core.rs +406 -0
- data/core/src/services/huggingface/docs.md +61 -0
- data/core/src/services/huggingface/error.rs +93 -0
- data/core/src/services/huggingface/lister.rs +91 -0
- data/core/src/services/huggingface/mod.rs +34 -0
- data/core/src/services/ipfs/backend.rs +257 -0
- data/core/src/services/ipfs/config.rs +32 -0
- data/core/src/services/ipfs/core.rs +239 -0
- data/core/src/services/ipfs/docs.md +45 -0
- data/core/src/services/ipfs/error.rs +52 -0
- data/core/src/services/ipfs/ipld.rs +162 -0
- data/core/src/services/ipfs/mod.rs +34 -0
- data/core/src/services/ipmfs/backend.rs +147 -0
- data/core/src/services/ipmfs/builder.rs +166 -0
- data/core/src/services/ipmfs/config.rs +32 -0
- data/core/src/services/ipmfs/core.rs +142 -0
- data/core/src/services/ipmfs/delete.rs +48 -0
- data/core/src/services/ipmfs/docs.md +14 -0
- data/core/src/services/ipmfs/error.rs +83 -0
- data/core/src/services/ipmfs/lister.rs +135 -0
- data/core/src/services/ipmfs/mod.rs +40 -0
- data/core/src/services/ipmfs/writer.rs +49 -0
- data/core/src/services/koofr/backend.rs +361 -0
- data/core/src/services/koofr/config.rs +50 -0
- data/core/src/services/koofr/core.rs +458 -0
- data/core/src/services/koofr/delete.rs +50 -0
- data/core/src/services/koofr/docs.md +51 -0
- data/core/src/services/koofr/error.rs +72 -0
- data/core/src/services/koofr/lister.rs +88 -0
- data/core/src/services/koofr/mod.rs +38 -0
- data/core/src/services/koofr/writer.rs +53 -0
- data/core/src/services/lakefs/backend.rs +309 -0
- data/core/src/services/lakefs/config.rs +81 -0
- data/core/src/services/lakefs/core.rs +261 -0
- data/core/src/services/lakefs/delete.rs +54 -0
- data/core/src/services/lakefs/docs.md +62 -0
- data/core/src/services/lakefs/error.rs +93 -0
- data/core/src/services/lakefs/lister.rs +120 -0
- data/core/src/services/lakefs/mod.rs +38 -0
- data/core/src/services/lakefs/writer.rs +50 -0
- data/core/src/services/memcached/backend.rs +284 -0
- data/core/src/services/memcached/binary.rs +289 -0
- data/core/src/services/memcached/config.rs +43 -0
- data/core/src/services/memcached/docs.md +47 -0
- data/core/src/services/memcached/mod.rs +27 -0
- data/core/src/services/memory/backend.rs +205 -0
- data/core/src/services/memory/config.rs +30 -0
- data/core/src/services/memory/core.rs +80 -0
- data/core/src/services/memory/delete.rs +42 -0
- data/core/src/services/memory/docs.md +36 -0
- data/core/src/services/memory/lister.rs +56 -0
- data/core/src/services/memory/mod.rs +36 -0
- data/core/src/services/memory/writer.rs +85 -0
- data/core/src/services/mini_moka/backend.rs +260 -0
- data/core/src/services/mini_moka/config.rs +56 -0
- data/core/src/services/mini_moka/core.rs +52 -0
- data/core/src/services/mini_moka/delete.rs +42 -0
- data/core/src/services/mini_moka/docs.md +19 -0
- data/core/src/services/mini_moka/lister.rs +68 -0
- data/core/src/services/mini_moka/mod.rs +36 -0
- data/core/src/services/mini_moka/writer.rs +84 -0
- data/core/src/services/mod.rs +206 -0
- data/core/src/services/moka/backend.rs +326 -0
- data/core/src/services/moka/config.rs +59 -0
- data/core/src/services/moka/core.rs +62 -0
- data/core/src/services/moka/delete.rs +42 -0
- data/core/src/services/moka/docs.md +42 -0
- data/core/src/services/moka/lister.rs +65 -0
- data/core/src/services/moka/mod.rs +41 -0
- data/core/src/services/moka/writer.rs +83 -0
- data/core/src/services/mongodb/backend.rs +291 -0
- data/core/src/services/mongodb/config.rs +54 -0
- data/core/src/services/mongodb/docs.md +49 -0
- data/core/src/services/mongodb/mod.rs +24 -0
- data/core/src/services/monoiofs/backend.rs +238 -0
- data/core/src/services/monoiofs/config.rs +34 -0
- data/core/src/services/monoiofs/core.rs +313 -0
- data/core/src/services/monoiofs/delete.rs +64 -0
- data/core/src/services/monoiofs/docs.md +46 -0
- data/core/src/services/monoiofs/mod.rs +36 -0
- data/core/src/services/monoiofs/reader.rs +147 -0
- data/core/src/services/monoiofs/writer.rs +189 -0
- data/core/src/services/mysql/backend.rs +256 -0
- data/core/src/services/mysql/config.rs +66 -0
- data/core/src/services/mysql/docs.md +47 -0
- data/core/src/services/mysql/mod.rs +24 -0
- data/core/src/services/obs/backend.rs +442 -0
- data/core/src/services/obs/config.rs +53 -0
- data/core/src/services/obs/core.rs +608 -0
- data/core/src/services/obs/delete.rs +48 -0
- data/core/src/services/obs/docs.md +54 -0
- data/core/src/services/obs/error.rs +106 -0
- data/core/src/services/obs/lister.rs +101 -0
- data/core/src/services/obs/mod.rs +38 -0
- data/core/src/services/obs/writer.rs +235 -0
- data/core/src/services/onedrive/backend.rs +127 -0
- data/core/src/services/onedrive/builder.rs +236 -0
- data/core/src/services/onedrive/config.rs +49 -0
- data/core/src/services/onedrive/core.rs +691 -0
- data/core/src/services/onedrive/delete.rs +47 -0
- data/core/src/services/onedrive/docs.md +115 -0
- data/core/src/services/onedrive/error.rs +61 -0
- data/core/src/services/onedrive/graph_model.rs +425 -0
- data/core/src/services/onedrive/lister.rs +150 -0
- data/core/src/services/onedrive/mod.rs +42 -0
- data/core/src/services/onedrive/writer.rs +168 -0
- data/core/src/services/opfs/backend.rs +50 -0
- data/core/src/services/opfs/config.rs +25 -0
- data/core/src/services/opfs/core.rs +74 -0
- data/core/src/services/opfs/docs.md +18 -0
- data/core/src/services/opfs/error.rs +27 -0
- data/core/src/services/opfs/mod.rs +30 -0
- data/core/src/services/opfs/utils.rs +70 -0
- data/core/src/services/oss/backend.rs +734 -0
- data/core/src/services/oss/config.rs +113 -0
- data/core/src/services/oss/core.rs +1088 -0
- data/core/src/services/oss/delete.rs +109 -0
- data/core/src/services/oss/docs.md +74 -0
- data/core/src/services/oss/error.rs +109 -0
- data/core/src/services/oss/lister.rs +256 -0
- data/core/src/services/oss/mod.rs +38 -0
- data/core/src/services/oss/writer.rs +228 -0
- data/core/src/services/pcloud/backend.rs +358 -0
- data/core/src/services/pcloud/config.rs +51 -0
- data/core/src/services/pcloud/core.rs +461 -0
- data/core/src/services/pcloud/delete.rs +66 -0
- data/core/src/services/pcloud/docs.md +51 -0
- data/core/src/services/pcloud/error.rs +88 -0
- data/core/src/services/pcloud/lister.rs +95 -0
- data/core/src/services/pcloud/mod.rs +38 -0
- data/core/src/services/pcloud/writer.rs +66 -0
- data/core/src/services/persy/backend.rs +226 -0
- data/core/src/services/persy/config.rs +32 -0
- data/core/src/services/persy/docs.md +43 -0
- data/core/src/services/persy/mod.rs +24 -0
- data/core/src/services/postgresql/backend.rs +258 -0
- data/core/src/services/postgresql/config.rs +66 -0
- data/core/src/services/postgresql/docs.md +47 -0
- data/core/src/services/postgresql/mod.rs +24 -0
- data/core/src/services/redb/backend.rs +280 -0
- data/core/src/services/redb/config.rs +34 -0
- data/core/src/services/redb/docs.md +41 -0
- data/core/src/services/redb/mod.rs +24 -0
- data/core/src/services/redis/backend.rs +442 -0
- data/core/src/services/redis/config.rs +79 -0
- data/core/src/services/redis/core.rs +209 -0
- data/core/src/services/redis/delete.rs +40 -0
- data/core/src/services/redis/docs.md +43 -0
- data/core/src/services/redis/mod.rs +34 -0
- data/core/src/services/redis/writer.rs +57 -0
- data/core/src/services/rocksdb/backend.rs +159 -0
- data/core/src/services/rocksdb/config.rs +34 -0
- data/core/src/services/rocksdb/docs.md +54 -0
- data/core/src/services/rocksdb/mod.rs +24 -0
- data/core/src/services/s3/backend.rs +1293 -0
- data/core/src/services/s3/compatible_services.md +126 -0
- data/core/src/services/s3/config.rs +327 -0
- data/core/src/services/s3/core.rs +1741 -0
- data/core/src/services/s3/delete.rs +109 -0
- data/core/src/services/s3/docs.md +244 -0
- data/core/src/services/s3/error.rs +171 -0
- data/core/src/services/s3/lister.rs +405 -0
- data/core/src/services/s3/mod.rs +38 -0
- data/core/src/services/s3/writer.rs +262 -0
- data/core/src/services/seafile/backend.rs +297 -0
- data/core/src/services/seafile/config.rs +56 -0
- data/core/src/services/seafile/core.rs +475 -0
- data/core/src/services/seafile/delete.rs +40 -0
- data/core/src/services/seafile/docs.md +54 -0
- data/core/src/services/seafile/error.rs +86 -0
- data/core/src/services/seafile/lister.rs +83 -0
- data/core/src/services/seafile/mod.rs +38 -0
- data/core/src/services/seafile/writer.rs +55 -0
- data/core/src/services/sftp/backend.rs +397 -0
- data/core/src/services/sftp/config.rs +50 -0
- data/core/src/services/sftp/core.rs +154 -0
- data/core/src/services/sftp/delete.rs +55 -0
- data/core/src/services/sftp/docs.md +49 -0
- data/core/src/services/sftp/error.rs +57 -0
- data/core/src/services/sftp/lister.rs +88 -0
- data/core/src/services/sftp/mod.rs +42 -0
- data/core/src/services/sftp/reader.rs +78 -0
- data/core/src/services/sftp/utils.rs +51 -0
- data/core/src/services/sftp/writer.rs +67 -0
- data/core/src/services/sled/backend.rs +194 -0
- data/core/src/services/sled/config.rs +45 -0
- data/core/src/services/sled/docs.md +39 -0
- data/core/src/services/sled/mod.rs +24 -0
- data/core/src/services/sqlite/backend.rs +326 -0
- data/core/src/services/sqlite/config.rs +70 -0
- data/core/src/services/sqlite/docs.md +46 -0
- data/core/src/services/sqlite/mod.rs +24 -0
- data/core/src/services/surrealdb/backend.rs +365 -0
- data/core/src/services/surrealdb/config.rs +64 -0
- data/core/src/services/surrealdb/docs.md +54 -0
- data/core/src/services/surrealdb/mod.rs +24 -0
- data/core/src/services/swift/backend.rs +275 -0
- data/core/src/services/swift/compatible_services.md +53 -0
- data/core/src/services/swift/config.rs +53 -0
- data/core/src/services/swift/core.rs +310 -0
- data/core/src/services/swift/delete.rs +49 -0
- data/core/src/services/swift/docs.md +52 -0
- data/core/src/services/swift/error.rs +90 -0
- data/core/src/services/swift/lister.rs +119 -0
- data/core/src/services/swift/mod.rs +38 -0
- data/core/src/services/swift/writer.rs +53 -0
- data/core/src/services/tikv/backend.rs +237 -0
- data/core/src/services/tikv/config.rs +52 -0
- data/core/src/services/tikv/docs.md +43 -0
- data/core/src/services/tikv/mod.rs +24 -0
- data/core/src/services/upyun/backend.rs +317 -0
- data/core/src/services/upyun/config.rs +51 -0
- data/core/src/services/upyun/core.rs +521 -0
- data/core/src/services/upyun/delete.rs +50 -0
- data/core/src/services/upyun/docs.md +51 -0
- data/core/src/services/upyun/error.rs +97 -0
- data/core/src/services/upyun/lister.rs +101 -0
- data/core/src/services/upyun/mod.rs +38 -0
- data/core/src/services/upyun/writer.rs +127 -0
- data/core/src/services/vercel_artifacts/backend.rs +99 -0
- data/core/src/services/vercel_artifacts/builder.rs +117 -0
- data/core/src/services/vercel_artifacts/config.rs +39 -0
- data/core/src/services/vercel_artifacts/core.rs +112 -0
- data/core/src/services/vercel_artifacts/docs.md +40 -0
- data/core/src/services/vercel_artifacts/error.rs +50 -0
- data/core/src/services/vercel_artifacts/mod.rs +36 -0
- data/core/src/services/vercel_artifacts/writer.rs +58 -0
- data/core/src/services/vercel_blob/backend.rs +251 -0
- data/core/src/services/vercel_blob/config.rs +45 -0
- data/core/src/services/vercel_blob/core.rs +449 -0
- data/core/src/services/vercel_blob/delete.rs +38 -0
- data/core/src/services/vercel_blob/docs.md +45 -0
- data/core/src/services/vercel_blob/error.rs +110 -0
- data/core/src/services/vercel_blob/lister.rs +69 -0
- data/core/src/services/vercel_blob/mod.rs +38 -0
- data/core/src/services/vercel_blob/writer.rs +143 -0
- data/core/src/services/webdav/backend.rs +318 -0
- data/core/src/services/webdav/config.rs +53 -0
- data/core/src/services/webdav/core.rs +859 -0
- data/core/src/services/webdav/delete.rs +47 -0
- data/core/src/services/webdav/docs.md +49 -0
- data/core/src/services/webdav/error.rs +53 -0
- data/core/src/services/webdav/lister.rs +106 -0
- data/core/src/services/webdav/mod.rs +38 -0
- data/core/src/services/webdav/writer.rs +56 -0
- data/core/src/services/webhdfs/backend.rs +376 -0
- data/core/src/services/webhdfs/config.rs +52 -0
- data/core/src/services/webhdfs/core.rs +398 -0
- data/core/src/services/webhdfs/delete.rs +46 -0
- data/core/src/services/webhdfs/docs.md +90 -0
- data/core/src/services/webhdfs/error.rs +126 -0
- data/core/src/services/webhdfs/lister.rs +130 -0
- data/core/src/services/webhdfs/message.rs +249 -0
- data/core/src/services/webhdfs/mod.rs +41 -0
- data/core/src/services/webhdfs/writer.rs +177 -0
- data/core/src/services/yandex_disk/backend.rs +267 -0
- data/core/src/services/yandex_disk/config.rs +45 -0
- data/core/src/services/yandex_disk/core.rs +340 -0
- data/core/src/services/yandex_disk/delete.rs +54 -0
- data/core/src/services/yandex_disk/docs.md +45 -0
- data/core/src/services/yandex_disk/error.rs +104 -0
- data/core/src/services/yandex_disk/lister.rs +113 -0
- data/core/src/services/yandex_disk/mod.rs +38 -0
- data/core/src/services/yandex_disk/writer.rs +52 -0
- data/core/src/types/buffer.rs +991 -0
- data/core/src/types/builder.rs +152 -0
- data/core/src/types/capability.rs +209 -0
- data/core/src/types/context/mod.rs +22 -0
- data/core/src/types/context/read.rs +231 -0
- data/core/src/types/context/write.rs +441 -0
- data/core/src/types/delete/deleter.rs +220 -0
- data/core/src/types/delete/futures_delete_sink.rs +176 -0
- data/core/src/types/delete/input.rs +97 -0
- data/core/src/types/delete/mod.rs +26 -0
- data/core/src/types/entry.rs +69 -0
- data/core/src/types/error.rs +570 -0
- data/core/src/types/execute/api.rs +110 -0
- data/core/src/types/execute/executor.rs +96 -0
- data/core/src/types/execute/executors/mod.rs +27 -0
- data/core/src/types/execute/executors/tokio_executor.rs +60 -0
- data/core/src/types/execute/mod.rs +25 -0
- data/core/src/types/list.rs +137 -0
- data/core/src/types/metadata.rs +436 -0
- data/core/src/types/mod.rs +72 -0
- data/core/src/types/mode.rs +68 -0
- data/core/src/types/operator/builder.rs +535 -0
- data/core/src/types/operator/info.rs +63 -0
- data/core/src/types/operator/mod.rs +33 -0
- data/core/src/types/operator/operator.rs +2236 -0
- data/core/src/types/operator/operator_futures.rs +1430 -0
- data/core/src/types/operator/registry.rs +129 -0
- data/core/src/types/options.rs +548 -0
- data/core/src/types/read/buffer_stream.rs +273 -0
- data/core/src/types/read/futures_async_reader.rs +289 -0
- data/core/src/types/read/futures_bytes_stream.rs +157 -0
- data/core/src/types/read/mod.rs +29 -0
- data/core/src/types/read/reader.rs +604 -0
- data/core/src/types/scheme.rs +475 -0
- data/core/src/types/write/buffer_sink.rs +188 -0
- data/core/src/types/write/futures_async_writer.rs +136 -0
- data/core/src/types/write/futures_bytes_sink.rs +103 -0
- data/core/src/types/write/mod.rs +26 -0
- data/core/src/types/write/writer.rs +411 -0
- data/core/tests/behavior/README.md +77 -0
- data/core/tests/behavior/async_copy.rs +314 -0
- data/core/tests/behavior/async_create_dir.rs +53 -0
- data/core/tests/behavior/async_delete.rs +354 -0
- data/core/tests/behavior/async_list.rs +739 -0
- data/core/tests/behavior/async_presign.rs +175 -0
- data/core/tests/behavior/async_read.rs +871 -0
- data/core/tests/behavior/async_rename.rs +210 -0
- data/core/tests/behavior/async_stat.rs +628 -0
- data/core/tests/behavior/async_write.rs +819 -0
- data/core/tests/behavior/main.rs +78 -0
- data/core/tests/behavior/utils.rs +187 -0
- data/core/tests/data/normal_dir/.gitkeep +0 -0
- data/core/tests/data/normal_file.txt +1041 -0
- data/core/tests/data/special_dir !@#$%^&()_+-=;',/.gitkeep +0 -0
- data/core/tests/data/special_file !@#$%^&()_+-=;',.txt +1041 -0
- data/core/users.md +13 -0
- data/extconf.rb +24 -0
- data/lib/opendal.rb +25 -0
- data/lib/opendal_ruby/entry.rb +35 -0
- data/lib/opendal_ruby/io.rb +70 -0
- data/lib/opendal_ruby/metadata.rb +44 -0
- data/lib/opendal_ruby/operator.rb +29 -0
- data/lib/opendal_ruby/operator_info.rb +26 -0
- data/src/capability.rs +146 -0
- data/src/io.rs +464 -0
- data/src/lib.rs +63 -0
- data/src/lister.rs +141 -0
- data/src/metadata.rs +111 -0
- data/src/middlewares.rs +174 -0
- data/src/operator.rs +310 -0
- data/src/operator_info.rs +83 -0
- data/test/blocking_op_test.rb +112 -0
- data/test/capability_test.rb +42 -0
- data/test/io_test.rb +172 -0
- data/test/lister_test.rb +77 -0
- data/test/metadata_test.rb +78 -0
- data/test/middlewares_test.rb +46 -0
- data/test/operator_info_test.rb +35 -0
- data/test/test_helper.rb +36 -0
- metadata +857 -0
|
@@ -0,0 +1,1556 @@
|
|
|
1
|
+
# Upgrade to v0.54
|
|
2
|
+
|
|
3
|
+
## Public API
|
|
4
|
+
|
|
5
|
+
### RFC-6189: Remove Native Blocking Support
|
|
6
|
+
|
|
7
|
+
OpenDAL v0.54 implements [RFC-6189](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_6189_remove_native_blocking/index.html), which removes all native blocking support in favor of using `block_on` from async runtimes.
|
|
8
|
+
|
|
9
|
+
The following breaking changes have been made:
|
|
10
|
+
|
|
11
|
+
- `blocking::Operator` can no longer be used within async contexts
|
|
12
|
+
- Using blocking APIs now requires an async runtime
|
|
13
|
+
- All `Blocking*` types have been moved to the `opendal::blocking` module
|
|
14
|
+
|
|
15
|
+
To migrate:
|
|
16
|
+
|
|
17
|
+
```diff
|
|
18
|
+
- use opendal::BlockingOperator;
|
|
19
|
+
+ use opendal::blocking::Operator;
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
### RFC-6213: Options Based API
|
|
23
|
+
|
|
24
|
+
OpenDAL v0.54 implements [RFC-6213](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_6213_options_api/index.html), which introduces options-based APIs for more structured and extensible operation configuration.
|
|
25
|
+
|
|
26
|
+
New APIs added:
|
|
27
|
+
|
|
28
|
+
- `read_options(path, ReadOptions)`
|
|
29
|
+
- `write_options(path, data, WriteOptions)`
|
|
30
|
+
- `list_options(path, ListOptions)`
|
|
31
|
+
- `stat_options(path, StatOptions)`
|
|
32
|
+
- `delete_options(path, DeleteOptions)`
|
|
33
|
+
|
|
34
|
+
Example usage:
|
|
35
|
+
|
|
36
|
+
```rust
|
|
37
|
+
// Read with options
|
|
38
|
+
let options = ReadOptions::new()
|
|
39
|
+
.range(0..1024)
|
|
40
|
+
.if_match("etag");
|
|
41
|
+
let data = op.read_options("path/to/file", options).await?;
|
|
42
|
+
|
|
43
|
+
// Write with options
|
|
44
|
+
let options = WriteOptions::new()
|
|
45
|
+
.content_type("text/plain")
|
|
46
|
+
.cache_control("max-age=3600");
|
|
47
|
+
op.write_options("path/to/file", data, options).await?;
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
### Remove `stat_has_xxx` and `list_has_xxx` APIs
|
|
51
|
+
|
|
52
|
+
All `stat_has_*` and `list_has_*` capability check APIs have been removed. Instead, check capabilities directly on the `Capability` struct:
|
|
53
|
+
|
|
54
|
+
```diff
|
|
55
|
+
- if op.info().full_capability().stat_has_content_length() {
|
|
56
|
+
+ if op.info().full_capability().stat.content_length {
|
|
57
|
+
// ...
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Fix `with_user_metadata` signature
|
|
62
|
+
|
|
63
|
+
The signature of `with_user_metadata` has been changed. Please update your code accordingly if you use this method.
|
|
64
|
+
|
|
65
|
+
### Services removed due to lack of maintainer
|
|
66
|
+
|
|
67
|
+
The following services have been removed due to lack of maintainers:
|
|
68
|
+
|
|
69
|
+
- `atomicserver`
|
|
70
|
+
- `icloud`
|
|
71
|
+
- `nebula_graph`
|
|
72
|
+
|
|
73
|
+
If you need these services, please consider maintaining them or use alternative services.
|
|
74
|
+
|
|
75
|
+
### HttpClientLayer replaces `update_http_client`
|
|
76
|
+
|
|
77
|
+
The `Operator::update_http_client()` method has been replaced by `HttpClientLayer`:
|
|
78
|
+
|
|
79
|
+
```diff
|
|
80
|
+
- op.update_http_client(client);
|
|
81
|
+
+ op = op.layer(HttpClientLayer::new(client));
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Expose `presign_xxx_options` API
|
|
85
|
+
|
|
86
|
+
New options-based presign APIs have been exposed:
|
|
87
|
+
|
|
88
|
+
```rust
|
|
89
|
+
let options = PresignOptions::new()
|
|
90
|
+
.expire(Duration::from_secs(3600));
|
|
91
|
+
|
|
92
|
+
let url = op.presign_read_options("path/to/file", options).await?;
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
## Raw API
|
|
96
|
+
|
|
97
|
+
### Remove native blocking support
|
|
98
|
+
|
|
99
|
+
All native blocking implementations have been removed from the raw API. Services and layers no longer need to implement blocking-specific methods.
|
|
100
|
+
|
|
101
|
+
# Upgrade to v0.53
|
|
102
|
+
|
|
103
|
+
## Public API
|
|
104
|
+
|
|
105
|
+
### Supabase service is now an S3-compatible servcice
|
|
106
|
+
|
|
107
|
+
Supabase Storage is now an S3-compatible service instead: https://github.com/supabase/storage.
|
|
108
|
+
|
|
109
|
+
We removed the supabase native service support in OpenDAL v0.53. Users who want to access Supabase Storage can use the S3 service instead.
|
|
110
|
+
|
|
111
|
+
### All metrics related layers have been refactored
|
|
112
|
+
|
|
113
|
+
All metrics layers have been refactored:
|
|
114
|
+
|
|
115
|
+
- `PrometheusLayer`
|
|
116
|
+
- `PrometheusClientLayer`
|
|
117
|
+
- `MetricsLayer`
|
|
118
|
+
|
|
119
|
+
They are now provides more metrics and more detailed information. All their public API have been redesigned.
|
|
120
|
+
|
|
121
|
+
For more details, please refer to `opendal::layers::observe`'s module documentation.
|
|
122
|
+
|
|
123
|
+
### `Operator::default_executor` has been replaced by `Operator::executor`
|
|
124
|
+
|
|
125
|
+
In opendal v0.53, we introduced a new concept of `Context` which is used to store the context of the current operator. Thanks to this design, we can now get and set the `executor` and `http_client` for given Operator instead.
|
|
126
|
+
|
|
127
|
+
All services `http_client` API has been deprecated and replaced by `Operator::update_http_client` API.
|
|
128
|
+
|
|
129
|
+
### OpenDAL MSRV bumped to `1.82`
|
|
130
|
+
|
|
131
|
+
Since v0.53, OpenDAL will require Rust 1.82.0 or later to build.
|
|
132
|
+
|
|
133
|
+
## Raw API
|
|
134
|
+
|
|
135
|
+
### Operation enum merge
|
|
136
|
+
|
|
137
|
+
To reduce the complexity of the `Operation`, we have merged the duplicated `Operation`.
|
|
138
|
+
|
|
139
|
+
For example:
|
|
140
|
+
|
|
141
|
+
- `Operation::ReaderRead` has been merged into `Operation::Read`
|
|
142
|
+
- `Operation::BlockingRead` has been merged into `Operation::Read`
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
# Upgrade to v0.52
|
|
146
|
+
|
|
147
|
+
## Public API
|
|
148
|
+
|
|
149
|
+
### RFC-5556: Write Returns Metadata
|
|
150
|
+
|
|
151
|
+
Since v0.52, all write APIs in OpenDAL have been updated to return `Metadata` instead of `()`. This metadata includes useful information provided by the service, such as `content-length`, `etag`, `version`, and `last-modified`.
|
|
152
|
+
|
|
153
|
+
This feature is not fully ready yet, and many available metadata fields are still not returned. Please visit [Tracking Issues of RFC-5556: Write Returns Metadata](https://github.com/apache/opendal/issues/5557) for progress and contributions.
|
|
154
|
+
|
|
155
|
+
Affected API:
|
|
156
|
+
|
|
157
|
+
- `opendal::Operator::write`
|
|
158
|
+
- `opendal::Operator::write_with`
|
|
159
|
+
- `opendal::Operator::writer::close`
|
|
160
|
+
- `opendal::raw::oio::Write::close`
|
|
161
|
+
|
|
162
|
+
### Github Actions Cache (ghac) service v2
|
|
163
|
+
|
|
164
|
+
As [requested](https://github.com/apache/opendal/issues/5620) by GitHub, we have upgraded our GHAC service to ensure compatibility with the latest GitHub Actions cache API.
|
|
165
|
+
|
|
166
|
+
By upgrading to OpenDAL v0.52, your services will continue functioning after the deprecation of the legacy service (2025/03/01). GHES does not yet support GHAC v2, but OpenDAL has handled this properly to prevent any disruptions.
|
|
167
|
+
|
|
168
|
+
ghac service doesn't support `delete` anymore, please use github's API to delete cache instead.
|
|
169
|
+
|
|
170
|
+
This upgrade is mandatory and enabled by default using an environment variable in the GitHub CI environment. No changes are required at the code level.
|
|
171
|
+
|
|
172
|
+
### Breaking Changes in Dependencies
|
|
173
|
+
|
|
174
|
+
- `OtelTraceLayer` and `OtelMetricsLayer`'s dependence `opentelemetry` bumped to `0.28`
|
|
175
|
+
- `PrometheusClientLayer`'s dependence `prometheus-client` bumped to `0.23.1`
|
|
176
|
+
|
|
177
|
+
# Upgrade to v0.51
|
|
178
|
+
|
|
179
|
+
## Public API
|
|
180
|
+
|
|
181
|
+
### New VISION: One Layer, All Storage
|
|
182
|
+
|
|
183
|
+
OpenDAL has refined its vision to **One Layer, All Storage**, driven by the following core principles: **Open Community**, **Solid Foundation**, **Fast Access**, **Object Storage First**, and **Extensible Architecture**.
|
|
184
|
+
|
|
185
|
+
Explore the detailed vision at [OpenDAL Vision](https://opendal.apache.org/vision).
|
|
186
|
+
|
|
187
|
+
### RFC-5313: Remove Metakey
|
|
188
|
+
|
|
189
|
+
OpenDAL v0.51 implements [RFC-5313](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_5314_remove_metakey/index.html), which removes the concept of metakey.
|
|
190
|
+
|
|
191
|
+
The following structs have been removed:
|
|
192
|
+
|
|
193
|
+
- `Metakey`
|
|
194
|
+
|
|
195
|
+
The following APIs have been removed:
|
|
196
|
+
|
|
197
|
+
- `list_with(path).metakey()`
|
|
198
|
+
|
|
199
|
+
Users no longer need to pass the metakey into the list. Instead, services will make their best effort to return as much metadata as possible. Users can check items like `Capability::list_has_etag` before making a call.
|
|
200
|
+
|
|
201
|
+
### Remove not used capability: `write_multi_align_size`
|
|
202
|
+
|
|
203
|
+
The capability `write_multi_align_size` is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.
|
|
204
|
+
|
|
205
|
+
### CapabilityCheckLayer and CorrectnessCheckLayer
|
|
206
|
+
|
|
207
|
+
OpenDAL used to perform capability checks for all services, but since v0.51, it only conducts checks that impact data correctness like `write_with_if_not_exists` or `delete_with_version` by default in the `CorrectnessCheckLayer`. If users wish to verify other non-critical capabilities like `write_with_content_type` or `write_with_cache_control`, they should manually enable the `CapabilityCheckLayer`.
|
|
208
|
+
|
|
209
|
+
### RFC-3911: Deleter API
|
|
210
|
+
|
|
211
|
+
OpenDAL v0.51 implements [RFC-3911](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_3911_deleter_api/index.html), which adds `Deleter` in OpenDAL to replace `batch` operation.
|
|
212
|
+
|
|
213
|
+
The following new APIs have been added:
|
|
214
|
+
|
|
215
|
+
- [`Operator::delete_iter`]
|
|
216
|
+
- [`Operator::delete_try_iter`]
|
|
217
|
+
- [`Operator::delete_stream`]
|
|
218
|
+
- [`Operator::delete_try_stream`]
|
|
219
|
+
- [`Operator::deleter`]
|
|
220
|
+
- [`Deleter::delete`]
|
|
221
|
+
- [`Deleter::delete_iter`]
|
|
222
|
+
- [`Deleter::delete_try_iter`]
|
|
223
|
+
- [`Deleter::delete_stream`]
|
|
224
|
+
- [`Deleter::delete_try_stream`]
|
|
225
|
+
- [`Deleter::flush`]
|
|
226
|
+
- [`Deleter::close`]
|
|
227
|
+
- [`Deleter::into_sink`]
|
|
228
|
+
- [`DeleteInput`]
|
|
229
|
+
- [`IntoDeleteInput`]
|
|
230
|
+
- [`FuturesDeleteSink`]
|
|
231
|
+
|
|
232
|
+
The following APIs have been deprecated and will be removed in the future releases:
|
|
233
|
+
|
|
234
|
+
- `Operator::remove` (replace with [`Operator::delete_iter`])
|
|
235
|
+
- `Operator::remove_via` (replace with [`Operator::delete_stream`])
|
|
236
|
+
|
|
237
|
+
As a result of this change, the `limit` and `with_limit` APIs on `Operator` have also been deprecated; they are currently no-ops.
|
|
238
|
+
|
|
239
|
+
## Raw API
|
|
240
|
+
|
|
241
|
+
### `adapter::kv` now returns `Scanner` instead of `Vec<String>`
|
|
242
|
+
|
|
243
|
+
To support returning key-value entries in a streaming manner instead of loading them all into memory, OpenDAL updated its adapter API to return a `Scanner` instead of a `Vec<String>`.
|
|
244
|
+
|
|
245
|
+
```diff
|
|
246
|
+
- async fn scan(&self, path: &str) -> Result<Vec<String>>
|
|
247
|
+
+ async fn scan(&self, path: &str) -> Result<Self::Scanner>
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
All services intending to implement `kv::Adapter` should adhere to this API change.
|
|
251
|
+
|
|
252
|
+
## Align `metadata` API to `info`
|
|
253
|
+
|
|
254
|
+
OpenDAL changes it's old `metadata` API to `info` to align with the new `AccessorInfo` struct.
|
|
255
|
+
|
|
256
|
+
```diff
|
|
257
|
+
- fn metadata(&self) -> Arc<AccessorInfo>
|
|
258
|
+
+ fn info(&self) -> Arc<AccessorInfo>
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
### Remove not used struct: `RangeWriter`
|
|
262
|
+
|
|
263
|
+
The struct `RangeWriter` is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.
|
|
264
|
+
|
|
265
|
+
# Upgrade to v0.50
|
|
266
|
+
|
|
267
|
+
## Public API
|
|
268
|
+
|
|
269
|
+
### `services-postgresql`'s connect string now supports only URL format
|
|
270
|
+
|
|
271
|
+
Previously, it supports both URL format and key-value format. After switching the implementation from `tokio-postgres` to `sqlx`, the service now supports only the URL format.
|
|
272
|
+
|
|
273
|
+
### `list` now returns path itself
|
|
274
|
+
|
|
275
|
+
Previously, `list("a/b")` would not return `a/b` even if it does exist. Since v0.50.0, this behavior has been changed. OpenDAL will now return the path itself if it exists. This change applies to all cases, whether the path is a directory or a file.
|
|
276
|
+
|
|
277
|
+
### Refactoring of the metrics-related layer
|
|
278
|
+
|
|
279
|
+
In OpenDAL v0.50.0, we did a refactor on all metrics-related layers. They are now sharing the same underlying implementations. `PrometheusLayer`, `PrometheusClientLayer` and `MetricsLayer` are now have similar public APIs and exactly the same metrics value.
|
|
280
|
+
|
|
281
|
+
# Upgrade to v0.49
|
|
282
|
+
|
|
283
|
+
## Public API
|
|
284
|
+
|
|
285
|
+
### `Configurator` now returns associated builder instead
|
|
286
|
+
|
|
287
|
+
`Configurator` used to return `impl Builder`, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way.
|
|
288
|
+
|
|
289
|
+
```diff
|
|
290
|
+
impl Configurator for MemoryConfig {
|
|
291
|
+
- fn into_builder(self) -> impl Builder {
|
|
292
|
+
+ type Builder = MemoryBuilder;
|
|
293
|
+
+ fn into_builder(self) -> Self::Builder {
|
|
294
|
+
MemoryBuilder { config: self }
|
|
295
|
+
}
|
|
296
|
+
}
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
### `LoggingLayer` now accepts `LoggingInterceptor`
|
|
300
|
+
|
|
301
|
+
`LoggingLayer` now accepts `LoggingInterceptor` trait instead of configuration. This change will allow users to customize the logging behavior more flexibly.
|
|
302
|
+
|
|
303
|
+
```diff
|
|
304
|
+
pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static {
|
|
305
|
+
fn log(
|
|
306
|
+
&self,
|
|
307
|
+
info: &AccessorInfo,
|
|
308
|
+
operation: Operation,
|
|
309
|
+
context: &[(&str, &str)],
|
|
310
|
+
message: &str,
|
|
311
|
+
err: Option<&Error>,
|
|
312
|
+
);
|
|
313
|
+
}
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
Users can now implement the log in the way they want.
|
|
317
|
+
|
|
318
|
+
# Upgrade to v0.48
|
|
319
|
+
|
|
320
|
+
## Public API
|
|
321
|
+
|
|
322
|
+
### Typo in `customized_credential_load`
|
|
323
|
+
|
|
324
|
+
Since v0.48, the `customed_credential_load` function has been renamed to `customized_credential_load` to fix the typo of `customized`.
|
|
325
|
+
|
|
326
|
+
```diff
|
|
327
|
+
- builder.customed_credential_load(v);
|
|
328
|
+
+ builder.customized_credential_load(v);
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
### S3 service rename `security_token` to `session_token`
|
|
332
|
+
|
|
333
|
+
[In 2014 Amazon switched](https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/) from `AWS_SECURITY_TOKEN` to `AWS_SESSION_TOKEN`. To be consistent with the naming of AWS STS, we have renamed the `security_token` field to `session_token` in the S3 service.
|
|
334
|
+
|
|
335
|
+
```diff
|
|
336
|
+
- builder.security_token(v);
|
|
337
|
+
+ builder.session_token(v);
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
### Operator `from_iter` and `via_iter` replaces `from_map` and `via_map`
|
|
341
|
+
|
|
342
|
+
Since v0.48, Operator's new APIs `from_iter` and `via_iter` methods have deprecated the `from_map` and `via_map` methods.
|
|
343
|
+
|
|
344
|
+
```diff
|
|
345
|
+
- Operator::from_map::<Fs>(map)?.finish();
|
|
346
|
+
+ Operator::from_iter::<Fs>(map)?.finish();
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
New API `from_iter` and `via_iter` should cover all use cases of `from_map` and `via_map`.
|
|
350
|
+
|
|
351
|
+
### Service builder now takes ownership
|
|
352
|
+
|
|
353
|
+
Since v0.48, all service builder now takes ownership `self` instead of `&mut self`. This change will allow users to configure the service in a more flexible way.
|
|
354
|
+
|
|
355
|
+
```diff
|
|
356
|
+
- let mut builder = S3::default();
|
|
357
|
+
- builder.bucket("test");
|
|
358
|
+
- builder.root("/path/to/root");
|
|
359
|
+
+ let builder = S3::default().bucket("test").root("/path/to/root");
|
|
360
|
+
let op = Operator::new(builder)?.finish();
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
## Raw API
|
|
364
|
+
|
|
365
|
+
### `oio::Write::write` will write the whole buffer
|
|
366
|
+
|
|
367
|
+
Starting from version 0.48, `oio::Write::write` now writes the entire buffer. This update aligns the API more closely with `oio::Read::read` and simplifies the implementation of concurrent writing.
|
|
368
|
+
|
|
369
|
+
```diff
|
|
370
|
+
trait Write {
|
|
371
|
+
- fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<usize>>;
|
|
372
|
+
+ fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<()>>;
|
|
373
|
+
}
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
`write` will now return `Result<()>` instead of `Result<usize>`. The number of bytes written can be obtained from the buffer's length.
|
|
377
|
+
|
|
378
|
+
### `Access::metadata()` will return `Arc<AccessInfo>`
|
|
379
|
+
|
|
380
|
+
Starting from version 0.48, `Access::metadata()` will return `Arc<AccessInfo>` instead of `AccessInfo`. This change is intended to improve performance and reduce memory usage.
|
|
381
|
+
|
|
382
|
+
```diff
|
|
383
|
+
trait Access {
|
|
384
|
+
- fn metadata(&self) -> AccessInfo;
|
|
385
|
+
+ fn metadata(&self) -> Arc<AccessInfo>;
|
|
386
|
+
}
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
### `MinitraceLayer` renamed to `FastraceLayer`
|
|
390
|
+
|
|
391
|
+
The `MinitraceLayer` has been renamed to `FastraceLayer` to respond to the [transition from `minitrace` to `fastrace`](https://github.com/tikv/minitrace-rust/issues/229).
|
|
392
|
+
|
|
393
|
+
```diff
|
|
394
|
+
- use opendal::layers::MinitraceLayer;
|
|
395
|
+
+ use opendal::layers::FastraceLayer;
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
### Use `Configurator` to replace `Builder::from_config`
|
|
399
|
+
|
|
400
|
+
Since v0.48, the `Builder::from_config` and `Builder::from_map` method has been replaced by the `Configurator` trait. The `Configurator` trait provides a more flexible and extensible way to configure OpenDAL.
|
|
401
|
+
|
|
402
|
+
Service implementers should update their code to use the `Configurator` trait instead:
|
|
403
|
+
|
|
404
|
+
```rust
|
|
405
|
+
impl Configurator for MemoryConfig {
|
|
406
|
+
type Builder = MemoryBuilder;
|
|
407
|
+
fn into_builder(self) -> Self::Builder {
|
|
408
|
+
MemoryBuilder { config: self }
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
|
|
412
|
+
impl Builder for MemoryBuilder {
|
|
413
|
+
const SCHEME: Scheme = Scheme::Memory;
|
|
414
|
+
type Config = MemoryConfig;
|
|
415
|
+
|
|
416
|
+
fn build(self) -> Result<impl Access> {
|
|
417
|
+
...
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
# Upgrade to v0.47
|
|
423
|
+
|
|
424
|
+
## Public API
|
|
425
|
+
|
|
426
|
+
### Reader `into_xxx` APIs
|
|
427
|
+
|
|
428
|
+
Since v0.47, `Reader`'s `into_xxx` APIs requires `async` and returns `Result` instead.
|
|
429
|
+
|
|
430
|
+
```diff
|
|
431
|
+
- let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048);
|
|
432
|
+
+ let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?;
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
Affected API includes:
|
|
436
|
+
|
|
437
|
+
- `Reader::into_futures_async_read`
|
|
438
|
+
- `Reader::into_bytes_stream`
|
|
439
|
+
- `BlockingReader::into_std_read`
|
|
440
|
+
- `BlockingReader::into_bytes_iterator`
|
|
441
|
+
|
|
442
|
+
## Raw API
|
|
443
|
+
|
|
444
|
+
### Bring Streaming Read Back
|
|
445
|
+
|
|
446
|
+
As explained in [core: Bring Streaming Read Back](https://github.com/apache/opendal/issues/4672), we do need read streaming back for better performance and low memory usage.
|
|
447
|
+
|
|
448
|
+
So our `oio::Read` changed back to streaming read instead:
|
|
449
|
+
|
|
450
|
+
```diff
|
|
451
|
+
trait Read {
|
|
452
|
+
- async fn read(&self, offset: u64, size: usize) -> Result<Buffer>;
|
|
453
|
+
+ async fn read(&mut self) -> Result<Buffer>;
|
|
454
|
+
}
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
All services and layers should be updated to meet this change.
|
|
458
|
+
|
|
459
|
+
# Upgrade to v0.46
|
|
460
|
+
|
|
461
|
+
## Public API
|
|
462
|
+
|
|
463
|
+
### MSRV Changed to 1.75
|
|
464
|
+
|
|
465
|
+
Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like [`RPITIT`](https://rust-lang.github.io/rfcs/3425-return-position-impl-trait-in-traits.html) and [`AFIT`](https://rust-lang.github.io/rfcs/3185-static-async-fn-in-trait.html).
|
|
466
|
+
|
|
467
|
+
### Services Feature Flag
|
|
468
|
+
|
|
469
|
+
Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.
|
|
470
|
+
|
|
471
|
+
Additionally, we have removed all `reqwest`-related feature flags:
|
|
472
|
+
|
|
473
|
+
- Users must now directly use `reqwest`'s feature flags for options like `rustls`, `native-tls`, etc.
|
|
474
|
+
- The `rustls` feature is no longer enabled by default; it must be activated manually.
|
|
475
|
+
- OpenDAL no longer offers the `trust-dns` option; users should configure the client builder directly.
|
|
476
|
+
|
|
477
|
+
### Range Based Read
|
|
478
|
+
|
|
479
|
+
Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.
|
|
480
|
+
|
|
481
|
+
`opendal::Reader` now have APIs like:
|
|
482
|
+
|
|
483
|
+
```rust
|
|
484
|
+
let r = op.reader("test.txt").await?;
|
|
485
|
+
let buf = r.read(1024..2048).await?;
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
### Buffer Based IO
|
|
489
|
+
|
|
490
|
+
Since version 0.46, OpenDAL features a native `Buffer` struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.
|
|
491
|
+
|
|
492
|
+
OpenDAL's `Reader` will return `Buffer` and `Writer` will accept `Buffer` as input. Users who have implemented their own IO traits should update their code to use the new `Buffer` struct.
|
|
493
|
+
|
|
494
|
+
```rust
|
|
495
|
+
let r = op.reader("test.txt").await?;
|
|
496
|
+
// read returns `Buffer`
|
|
497
|
+
let buf: Buffer = r.read(1024..2048).await?;
|
|
498
|
+
|
|
499
|
+
let w = op.writer("test.txt").await?;
|
|
500
|
+
|
|
501
|
+
// Buffer can be created from continues bytes.
|
|
502
|
+
w.write("hello, world").await?;
|
|
503
|
+
// Buffer can also be created from non-continues bytes.
|
|
504
|
+
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;
|
|
505
|
+
|
|
506
|
+
// Make sure file has been written completely.
|
|
507
|
+
w.close().await?;
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
To enhance usability, we've integrated bridges into `bytes::Buf` and `bytes::BufMut`, allowing users to directly interact with the bytes API.
|
|
511
|
+
|
|
512
|
+
```rust
|
|
513
|
+
let r = op.reader("test.txt").await?;
|
|
514
|
+
let mut bs = vec![];
|
|
515
|
+
// read_into accepts bytes::BufMut
|
|
516
|
+
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;
|
|
517
|
+
|
|
518
|
+
let w = op.writer("test.txt").await?;
|
|
519
|
+
|
|
520
|
+
// write_from accepts bytes::Buf
|
|
521
|
+
w.write_from("hello, world".as_bytes()).await?;
|
|
522
|
+
|
|
523
|
+
// Make sure file has been written completely.
|
|
524
|
+
w.close().await?;
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
### Bridge API
|
|
528
|
+
|
|
529
|
+
OpenDAL's `Reader` and `Writer` previously implemented APIs such as `AsyncRead` and `AsyncWrite` directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.
|
|
530
|
+
|
|
531
|
+
Since v0.46, OpenDAL provides bridge APIs for `Reader` and `Writer` instead.
|
|
532
|
+
|
|
533
|
+
```rust
|
|
534
|
+
let r = op.reader("test.txt").await?;
|
|
535
|
+
|
|
536
|
+
// Convert into futures AsyncRead + AsyncSeek.
|
|
537
|
+
let reader = r.into_futures_async_read(1024..2048);
|
|
538
|
+
// Convert into futures bytes stream.
|
|
539
|
+
let stream = r.into_bytes_stream(1024..2048);
|
|
540
|
+
|
|
541
|
+
let w = op.writer("test.txt").await?;
|
|
542
|
+
|
|
543
|
+
// Convert into futures AsyncWrite
|
|
544
|
+
let writer = w.into_futures_async_write();
|
|
545
|
+
// Convert into futures bytes sink;
|
|
546
|
+
let sink = w.into_bytes_sink();
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
## Raw API
|
|
550
|
+
|
|
551
|
+
### Async in IO trait
|
|
552
|
+
|
|
553
|
+
Since version 0.46, OpenDAL has adopted Rust's native [`async_in_trait`](https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits.html) for our core IO traits, including `oio::Read`, `oio::Write`, and `oio::List`.
|
|
554
|
+
|
|
555
|
+
This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.
|
|
556
|
+
|
|
557
|
+
Users who have implemented their own IO traits should update their code to use the new async trait syntax.
|
|
558
|
+
|
|
559
|
+
# Upgrade to v0.45
|
|
560
|
+
|
|
561
|
+
## Public API
|
|
562
|
+
|
|
563
|
+
### BlockingLayer is not enabled by default
|
|
564
|
+
|
|
565
|
+
To further enhance the optionality of `tokio`, we have introduced a new feature called `layers-blocking`. The default usage of the blocking layer has been disabled. To utilize the `BlockingLayer`, please enable the `layers-blocking` feature.
|
|
566
|
+
|
|
567
|
+
### TimeoutLayer deprecated `with_speed`
|
|
568
|
+
|
|
569
|
+
The `with_speed` API has been deprecated. Please use `with_io_timeout` instead.
|
|
570
|
+
|
|
571
|
+
## Raw API
|
|
572
|
+
|
|
573
|
+
No raw API changes.
|
|
574
|
+
|
|
575
|
+
# Upgrade to v0.44
|
|
576
|
+
|
|
577
|
+
## Public API
|
|
578
|
+
|
|
579
|
+
### Moka Service Configuration
|
|
580
|
+
|
|
581
|
+
- The `thread_pool_enabled` option has been removed.
|
|
582
|
+
|
|
583
|
+
### List Prefix Supported
|
|
584
|
+
|
|
585
|
+
After [RFC: List Prefix](crate::docs::rfcs::rfc_3243_list_prefix) landed, we have changed the behavior of `list` a path without `/`. OpenDAL used to return `NotADirectory` error, but now we will return the list of entries that start with given prefix instead.
|
|
586
|
+
|
|
587
|
+
# Upgrade to v0.43
|
|
588
|
+
|
|
589
|
+
## Public API
|
|
590
|
+
|
|
591
|
+
### List Recursive
|
|
592
|
+
|
|
593
|
+
After [RFC-3526: List Recursive](crate::docs::rfcs::rfc_3526_list_recursive) landed, we have changed the `list` API to accept `recursive` instead of `delimiter`:
|
|
594
|
+
|
|
595
|
+
Users will need to change the following usage:
|
|
596
|
+
|
|
597
|
+
- `op.list_with(path).delimiter("")` -> `op.list_with(path).recursive(true)`
|
|
598
|
+
- `op.list_with(path).delimiter("/")` -> `op.list_with(path).recursive(false)`
|
|
599
|
+
|
|
600
|
+
`delimiter` other than `""` and `"/"` is not supported anymore.
|
|
601
|
+
|
|
602
|
+
### Stat a dir path
|
|
603
|
+
|
|
604
|
+
After [RFC: List Prefix](crate::docs::rfcs::rfc_3243_list_prefix) landed, we have changed the behavior of `stat` a dir path:
|
|
605
|
+
|
|
606
|
+
Here are the behavior list:
|
|
607
|
+
|
|
608
|
+
| Case | Path | Result |
|
|
609
|
+
|------------------------|-----------------|--------------------------------------------|
|
|
610
|
+
| stat existing dir | `abc/` | Metadata with dir mode |
|
|
611
|
+
| stat existing file | `abc/def_file` | Metadata with file mode |
|
|
612
|
+
| stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode |
|
|
613
|
+
| stat file with `/` | `abc/def_file/` | Error `NotFound` |
|
|
614
|
+
| stat not existing path | `xyz` | Error `NotFound` |
|
|
615
|
+
|
|
616
|
+
Services like s3, azblob can handle `stat("abc/")` correctly by check if there are objects with prefix `abc/`.
|
|
617
|
+
|
|
618
|
+
## Raw API
|
|
619
|
+
|
|
620
|
+
### Lister Align
|
|
621
|
+
|
|
622
|
+
We changed our internal `lister` implementation to align with the `list` public API for better performance and readability.
|
|
623
|
+
|
|
624
|
+
- trait `Page` => `List`
|
|
625
|
+
- struct `Pager` => `Lister`
|
|
626
|
+
- trait `BlockingPage` => `BlockingList`
|
|
627
|
+
- struct `BlockingPager` => `BlockingLister`
|
|
628
|
+
|
|
629
|
+
Every call to `next` will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of `async_trait`.
|
|
630
|
+
|
|
631
|
+
# Upgrade to v0.42
|
|
632
|
+
|
|
633
|
+
## Public API
|
|
634
|
+
|
|
635
|
+
### MSRV Changed
|
|
636
|
+
|
|
637
|
+
OpenDAL bumps it's MSRV to 1.67.0.
|
|
638
|
+
|
|
639
|
+
### S3 Service Configuration
|
|
640
|
+
|
|
641
|
+
- The `enable_exact_buf_write` option has been deprecated and is superseded by `BufferedWriter`, introduced in version 0.40.
|
|
642
|
+
|
|
643
|
+
### Oss Service Configuration
|
|
644
|
+
|
|
645
|
+
- The `write_min_size` option has been deprecated and replaced by `BufferedWriter`, also introduced in version 0.40.
|
|
646
|
+
- A new setting, `allow_anonymous`, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling `allow_anonymous` to fallback to request without credentials.
|
|
647
|
+
|
|
648
|
+
### Ghac Service Configuration
|
|
649
|
+
|
|
650
|
+
- The `enable_create_simulation` option has been removed. We add this option to allow ghac simulate create empty file, but it could result in unexpected behavior when users create a file with content length `1`. So we remove it.
|
|
651
|
+
|
|
652
|
+
### Wasabi Service Removed
|
|
653
|
+
|
|
654
|
+
`wasabi` service native support has been removed. Users who want to access wasabi can use our `s3` service instead.
|
|
655
|
+
|
|
656
|
+
# Upgrade to v0.41
|
|
657
|
+
|
|
658
|
+
There is no public API and raw API changes.
|
|
659
|
+
|
|
660
|
+
# Upgrade to v0.40
|
|
661
|
+
|
|
662
|
+
## Public API
|
|
663
|
+
|
|
664
|
+
### RFC-2578 Merge Append Into Write
|
|
665
|
+
|
|
666
|
+
[RFC-2578](crate::docs::rfcs::rfc_2758_merge_append_into_write) merges `append` into `write` and removes `append` API.
|
|
667
|
+
|
|
668
|
+
- For writing a file at once, please use `op.write()` for convenience.
|
|
669
|
+
- For appending a file, please use `op.write_with().append(true)` instead of `op.append()`.
|
|
670
|
+
|
|
671
|
+
The same rule applies to `writer()` and `writer_with()`.
|
|
672
|
+
|
|
673
|
+
### RFC-2774 Lister API
|
|
674
|
+
|
|
675
|
+
[RFC-2774](crate::docs::rfcs::rfc_2774_lister_api) proposes a new `lister` API to replace current `list` and `scan`. And we add a new API `list` to return entries directly.
|
|
676
|
+
|
|
677
|
+
- For listing a directory at once, please use `list()` for convenience.
|
|
678
|
+
- For listing a directory recursively, please use `list_with().delimiter("")` or `lister_with().delimiter("")` instead of `scan()`.
|
|
679
|
+
- For listing in streaming, please use `lister()` or `lister_with()` instead.
|
|
680
|
+
|
|
681
|
+
### RFC-2779 List With Metakey
|
|
682
|
+
|
|
683
|
+
[RFC-2779](crate::docs::rfcs::rfc_2779_list_with_metakey) proposes a new `op.list_with().metakey()` API to allow list with metakey and removes `op.metadata(&entry)` API.
|
|
684
|
+
|
|
685
|
+
Please use `op.list_with().metakey()` instead of `op.metadata(&entry)`, for example:
|
|
686
|
+
|
|
687
|
+
```rust
|
|
688
|
+
// Before
|
|
689
|
+
let entries: Vec<Entry> = op.list("dir/").await?;
|
|
690
|
+
for entry in entries {
|
|
691
|
+
let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
|
|
692
|
+
println!("{} {}", entry.name(), entry.metadata().content_length());
|
|
693
|
+
}
|
|
694
|
+
|
|
695
|
+
// After
|
|
696
|
+
let entries: Vec<Entry> = op
|
|
697
|
+
.list_with("dir/")
|
|
698
|
+
.metakey(Metakey::ContentLength | Metakey::ContentType).await?;
|
|
699
|
+
for entry in entries {
|
|
700
|
+
println!("{} {}", entry.name(), entry.metadata().content_length());
|
|
701
|
+
}
|
|
702
|
+
```
|
|
703
|
+
|
|
704
|
+
### RFC-2852: Native Capability
|
|
705
|
+
|
|
706
|
+
[RFC-2852](crate::docs::rfcs::rfc_2852_native_capability) proposes new `native_capability` and `full_capability` API to allow users to check if the underlying service supports a capability natively.
|
|
707
|
+
|
|
708
|
+
- `native_capability` returns `true` if the capability is supported natively.
|
|
709
|
+
- `full_capability` returns `true` if the capability is supported, maybe via a layer.
|
|
710
|
+
|
|
711
|
+
Most of time, you can use `full_capability` to replace `capability` call. But to check if the capability is supported natively for better performance design, please use `native_capability` instead.
|
|
712
|
+
|
|
713
|
+
### Buffered Writer
|
|
714
|
+
|
|
715
|
+
OpenDAL v0.40 added buffered writer support!
|
|
716
|
+
|
|
717
|
+
Users don't need to specify the `content_length()` for writer anymore!
|
|
718
|
+
|
|
719
|
+
```diff
|
|
720
|
+
- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
|
|
721
|
+
+ let mut w = op.writer_with("path/to/file").await?;
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
Users can specify the `buffer()` to control the size we call underlying storage:
|
|
725
|
+
|
|
726
|
+
```rust
|
|
727
|
+
let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;
|
|
728
|
+
```
|
|
729
|
+
|
|
730
|
+
If buffer is not specified, we will call underlying storage everytime we call `write`. Otherwise, we will make sure to call underlying storage when buffer is full or `close` is called.
|
|
731
|
+
|
|
732
|
+
### RangeRead and RangeReader
|
|
733
|
+
|
|
734
|
+
OpenDAL v0.40 removed the origin `range_read` and `range_reader` interfaces, please use `read_with().range()` or `reader_with().range()`.
|
|
735
|
+
|
|
736
|
+
```diff
|
|
737
|
+
- op.range_read(path, range_start..range_end).await?;
|
|
738
|
+
+ op.read_with(path).range(range_start..range_end).await?;
|
|
739
|
+
```
|
|
740
|
+
|
|
741
|
+
```diff
|
|
742
|
+
- let reader = op.range_reader(path, range_start..range_end).await?;
|
|
743
|
+
+ let reader = op.reader_with(path).range(range_start..range_end).await?;
|
|
744
|
+
```
|
|
745
|
+
|
|
746
|
+
|
|
747
|
+
|
|
748
|
+
## Raw API
|
|
749
|
+
|
|
750
|
+
### RFC-3017 Remove Write Copy From
|
|
751
|
+
|
|
752
|
+
[RFC-3017](crate::docs::rfcs::rfc_3017_remove_write_copy_from) removes `copy_from` API from the `oio::Write` trait. Users who implements services and layers by hand should remove this API.
|
|
753
|
+
|
|
754
|
+
# Upgrade to v0.39
|
|
755
|
+
|
|
756
|
+
## Public API
|
|
757
|
+
|
|
758
|
+
### Service S3 Role Arn Behavior
|
|
759
|
+
|
|
760
|
+
In PR #2687, OpenDAL changed the behavior when `role_arn` has been specified.
|
|
761
|
+
|
|
762
|
+
OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified `role_arn` and `external_id` (if supplied).
|
|
763
|
+
|
|
764
|
+
### RetryLayer supports RetryInterceptor
|
|
765
|
+
|
|
766
|
+
In PR #2666, `RetryLayer` supports `RetryInterceptor`. To implement this change, `RetryLayer` changed it's in-memory layout by adding a new generic parameter `I` to `RetryLayer<I>`.
|
|
767
|
+
|
|
768
|
+
Users who stores `RetryLayer` in struct or enum will need to change the type if they don't want to use default behavior.
|
|
769
|
+
|
|
770
|
+
## Raw API
|
|
771
|
+
|
|
772
|
+
In PR #2698, OpenDAL re-org the internal structure of `opendal::raw::oio` and changed some APIs name.
|
|
773
|
+
|
|
774
|
+
# Upgrade to v0.38
|
|
775
|
+
|
|
776
|
+
There are no public API changes.
|
|
777
|
+
|
|
778
|
+
## Raw API
|
|
779
|
+
|
|
780
|
+
OpenDAL add the `Write::sink` API to enable streaming writing. This is a breaking change for users who depend on the raw API.
|
|
781
|
+
|
|
782
|
+
For a quick fix, users who have implemented `opendal::raw::oio::Write` can return an `Unsupported` error for `Write::sink()`.
|
|
783
|
+
|
|
784
|
+
More details could be found at [RFC: Writer `sink` API][crate::docs::rfcs::rfc_2083_writer_sink_api].
|
|
785
|
+
|
|
786
|
+
# Upgrade to v0.37
|
|
787
|
+
|
|
788
|
+
In v0.37.0, OpenDAL bump the version of `reqsign` to v0.13.0.
|
|
789
|
+
|
|
790
|
+
There are no public API and raw API changes.
|
|
791
|
+
|
|
792
|
+
# Upgrade to v0.36
|
|
793
|
+
|
|
794
|
+
## Public API
|
|
795
|
+
|
|
796
|
+
In v0.36, OpenDAL improving the `xxx_with` API by allow it to be called in chain:
|
|
797
|
+
|
|
798
|
+
After this change, all `xxx_with` alike call will be changed from
|
|
799
|
+
|
|
800
|
+
```rust
|
|
801
|
+
let bs = op.read_with(
|
|
802
|
+
"path/to/file",
|
|
803
|
+
OpRead::new()
|
|
804
|
+
.with_range(0..=1024)
|
|
805
|
+
.with_if_match("<etag>")
|
|
806
|
+
.with_if_none_match("<etag>")
|
|
807
|
+
.with_override_cache_control("<cache_control>")
|
|
808
|
+
.with_override_content_disposition("<content_disposition>")
|
|
809
|
+
).await?;
|
|
810
|
+
```
|
|
811
|
+
|
|
812
|
+
to
|
|
813
|
+
|
|
814
|
+
```rust
|
|
815
|
+
let bs = op.read_with("path/to/file")
|
|
816
|
+
.range(0..=1024)
|
|
817
|
+
.if_match("<etag>")
|
|
818
|
+
.if_none_match("<etag>")
|
|
819
|
+
.override_cache_control("<cache_control>")
|
|
820
|
+
.override_content_disposition("<content_disposition>")
|
|
821
|
+
.await?;
|
|
822
|
+
```
|
|
823
|
+
|
|
824
|
+
For blocking API calls, we will need a `call()` at the end:
|
|
825
|
+
|
|
826
|
+
```rust
|
|
827
|
+
let bs = bop.read_with("path/to/file")
|
|
828
|
+
.range(0..=1024)
|
|
829
|
+
.if_match("<etag>")
|
|
830
|
+
.if_none_match("<etag>")
|
|
831
|
+
.override_cache_control("<cache_control>")
|
|
832
|
+
.override_content_disposition("<content_disposition>")
|
|
833
|
+
.call()?;
|
|
834
|
+
```
|
|
835
|
+
|
|
836
|
+
Along with this change, users don't need to call `OpXxx` anymore so we moved it to `raw` API.
|
|
837
|
+
|
|
838
|
+
More details could be found at [RFC: Chain Based Operator API][crate::docs::rfcs::rfc_2299_chain_based_operator_api].
|
|
839
|
+
|
|
840
|
+
## Raw API
|
|
841
|
+
|
|
842
|
+
Migrated `opendal::ops` to `opendal::raw::ops`.
|
|
843
|
+
|
|
844
|
+
# Upgrade to v0.35
|
|
845
|
+
|
|
846
|
+
## Public API
|
|
847
|
+
|
|
848
|
+
- OpenDAL removes rarely used `Operator::from_env` and `Operator::from_iter` APIs
|
|
849
|
+
- Users can use `Operator::via_map` instead.
|
|
850
|
+
|
|
851
|
+
## Raw API
|
|
852
|
+
|
|
853
|
+
- OpenDAL adds `append` support with could break existing layers. Please make sure `append` requests have been forward correctly.
|
|
854
|
+
- After the merging of `scan` and `list`, OpenDAL removes the `scan` from raw API. Please use `list_without_delimiter` instead.
|
|
855
|
+
|
|
856
|
+
# Upgrade to v0.34
|
|
857
|
+
|
|
858
|
+
## Public API
|
|
859
|
+
|
|
860
|
+
- OpenDAL raises it's MSRV to 1.65 for dependencies changes
|
|
861
|
+
- `OperatorInfo::can_scan` has been removed, to check if underlying services support scan a dir natively, please use `Capability::list_without_delimiter` instead.
|
|
862
|
+
|
|
863
|
+
## Raw API
|
|
864
|
+
|
|
865
|
+
### Merged `scan` into `list`
|
|
866
|
+
|
|
867
|
+
After `Capability` introduced, we have added `delimiter` in `OpList`. Users can specify the delimiter to `""` or `"/"` to control the list behavior.
|
|
868
|
+
|
|
869
|
+
Along with this change, `Operator::scan()` becomes a short alias of `Operator::list_with(OpList::new().with_delimiter(""))`.
|
|
870
|
+
|
|
871
|
+
### Typed Kv Adapter
|
|
872
|
+
|
|
873
|
+
In v0.34, OpenDAL adds a typed kv adapter for zero-copy read and write. If you are implemented kv adapter for a rust in-memory data struct, please consider migrate.
|
|
874
|
+
|
|
875
|
+
# Upgrade to v0.33
|
|
876
|
+
|
|
877
|
+
## Public API
|
|
878
|
+
|
|
879
|
+
OpenDAL 0.33 has redesigned the `Writer` API, replacing all instances of `writer.append()` with `writer.write()`. For more information, please refer to [`Writer`](crate::Writer).
|
|
880
|
+
|
|
881
|
+
## Raw API
|
|
882
|
+
|
|
883
|
+
In addition to the redesign of the `Writer` API, we have removed `append` from `oio::Write`. Therefore, users who implement services and layers should also remove it.
|
|
884
|
+
|
|
885
|
+
After v0.33 landing, services should handle `OpWrite::content_length` correctly by following these guidelines:
|
|
886
|
+
|
|
887
|
+
- If the writer does not support uploading unsized data, return a response of `NotSupported` if content length is `None`.
|
|
888
|
+
- Otherwise, continue writing data until either `close` or `abort` has been called.
|
|
889
|
+
|
|
890
|
+
Furthermore, OpenDAL 0.33 introduces a new concept called `Capability` which replaces `AccessorCapability`. Services must adapt to this change.
|
|
891
|
+
|
|
892
|
+
# Upgrade to v0.32
|
|
893
|
+
|
|
894
|
+
OpenDAL 0.32 doesn't have much breaking changes.
|
|
895
|
+
|
|
896
|
+
We changed `Accessor::create` into `Accessor::create_dir`. Only users who implement `Layer` need to change.
|
|
897
|
+
|
|
898
|
+
# Upgrade to v0.31
|
|
899
|
+
|
|
900
|
+
In version v0.31 of OpenDAL, we made some internal refactoring to improve its compatibility with the ecosystem.
|
|
901
|
+
|
|
902
|
+
## MSRV Bump
|
|
903
|
+
|
|
904
|
+
We increased the MSRV to 1.64 from v0.31 onwards. Although it is still possible to build OpenDAL under older rustc versions, we cannot guarantee that any issues related to them will be fixed.
|
|
905
|
+
|
|
906
|
+
## Accept `std::time::Duration` instead
|
|
907
|
+
|
|
908
|
+
Previously, OpenDAL accepted `time::Duration` as input for `presign_xxx`. However, since v0.31, we have changed this to accept `std::time::Duration` so that users do not need to depend on `time`. Internally, we migrated from `time` to `chrono` for better integration with other parts of the ecosystem.
|
|
909
|
+
|
|
910
|
+
## `disable_ec2_metadata` for services s3
|
|
911
|
+
|
|
912
|
+
We have added a new configuration option called `disable_ec2_metadata` for the S3 service in response to a mistake where it was mixed up with another option called `disable_config_load`. Users who want to disable loading credentials from EC2 metadata should set this option instead.
|
|
913
|
+
|
|
914
|
+
## Services Feature Flag
|
|
915
|
+
|
|
916
|
+
Starting from v0.31, all services in OpenDAL are split into different feature flags. To enable only S3 support, use the following TOML configuration:
|
|
917
|
+
|
|
918
|
+
```toml
|
|
919
|
+
opendal = {
|
|
920
|
+
version = "0.31",
|
|
921
|
+
default-features = false,
|
|
922
|
+
features = ["services-s3"]
|
|
923
|
+
}
|
|
924
|
+
```
|
|
925
|
+
|
|
926
|
+
# Upgrade to v0.30
|
|
927
|
+
|
|
928
|
+
In version 0.30, we made significant breaking changes by removing objects. Our goal in doing so was to provide our users with APIs that are easier to understand and maintain.
|
|
929
|
+
|
|
930
|
+
More details could be found at [RFC: Remove Object Concept][crate::docs::rfcs::rfc_1477_remove_object_concept].
|
|
931
|
+
|
|
932
|
+
To upgrade to OpenDAL v0.30, users need to make the following changes:
|
|
933
|
+
|
|
934
|
+
- regex replace `object\((.*)\).reader\(\)` to `reader($1)`
|
|
935
|
+
- replace the function on your case, it's recommended to do it one by one
|
|
936
|
+
- rename `ObjectMetakey` => `Metakey`
|
|
937
|
+
- rename `ObjectMode` => `EntryMode`
|
|
938
|
+
- replace `ErrorKind::ObjectXxx` to `ErrorKind::Xxx`
|
|
939
|
+
- rename `AccessorMetadata` => `AccessorInfo`
|
|
940
|
+
- rename `ObjectMetadata` => `Metadata`
|
|
941
|
+
- replace `operator.metadata()` => `operator.info()`
|
|
942
|
+
|
|
943
|
+
# Upgrade to v0.29
|
|
944
|
+
|
|
945
|
+
In v0.29, we introduced [Object Writer][crate::docs::rfcs::rfc_1420_object_writer] to replace existing Multipart related APIs.
|
|
946
|
+
|
|
947
|
+
Users can now append multiparts bytes into object via:
|
|
948
|
+
|
|
949
|
+
```rust
|
|
950
|
+
let mut w = o.writer().await?;
|
|
951
|
+
w.write(bs1).await?;
|
|
952
|
+
w.write(bs2).await?;
|
|
953
|
+
w.close()
|
|
954
|
+
```
|
|
955
|
+
|
|
956
|
+
Along with this change, we cleaned up a lot of internal structs and traits. Users who used to depend on `opendal::raw::io::{input,output}` should use `opendal::raw::oio` instead.
|
|
957
|
+
|
|
958
|
+
Also, decompress related feature also removed. Users can use `async-compression` with `ObjectReader` directly.
|
|
959
|
+
|
|
960
|
+
# Upgrade to v0.28
|
|
961
|
+
|
|
962
|
+
In v0.28, we introduced [Query Based Metadata][crate::docs::rfcs::rfc_1398_query_based_metadata]. Users can query cached metadata with `ObjectMetakey` to make sure that OpenDAL always makes the best decision.
|
|
963
|
+
|
|
964
|
+
```diff
|
|
965
|
+
- pub async fn metadata(&self) -> Result<ObjectMetadata>;
|
|
966
|
+
+ pub async fn metadata(
|
|
967
|
+
+ &self,
|
|
968
|
+
+ flags: impl Into<FlagSet<ObjectMetakey>>,
|
|
969
|
+
+ ) -> Result<Arc<ObjectMetadata>>;
|
|
970
|
+
```
|
|
971
|
+
|
|
972
|
+
Please visit `Object::metadata()`'s example for more details.
|
|
973
|
+
|
|
974
|
+
# Upgrade to v0.27
|
|
975
|
+
|
|
976
|
+
In v0.27, we refactored our `list` related logic and added `scan` support. So make `Pager` and `BlockingPager` associated types in `Accessor` too!
|
|
977
|
+
|
|
978
|
+
```diff
|
|
979
|
+
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
|
|
980
|
+
type Reader: output::Read;
|
|
981
|
+
type BlockingReader: output::BlockingRead;
|
|
982
|
+
+ type Pager: output::Page;
|
|
983
|
+
+ type BlockingPager: output::BlockingPage;
|
|
984
|
+
}
|
|
985
|
+
```
|
|
986
|
+
|
|
987
|
+
## User defined layers
|
|
988
|
+
|
|
989
|
+
Due to this change, all layers implementation should be changed. If there is not changed over pager, they can be changed like the following:
|
|
990
|
+
|
|
991
|
+
```diff
|
|
992
|
+
impl<A: Accessor> LayeredAccessor for MyAccessor<A> {
|
|
993
|
+
type Inner = A;
|
|
994
|
+
type Reader = MyReader<A::Reader>;
|
|
995
|
+
type BlockingReader = MyReader<A::BlockingReader>;
|
|
996
|
+
+ type Pager = A::Pager;
|
|
997
|
+
+ type BlockingPager = A::BlockingPager;
|
|
998
|
+
|
|
999
|
+
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> {
|
|
1000
|
+
+ self.inner.list(path, args).await
|
|
1001
|
+
+ }
|
|
1002
|
+
|
|
1003
|
+
+ async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> {
|
|
1004
|
+
+ self.inner.scan(path, args).await
|
|
1005
|
+
+ }
|
|
1006
|
+
|
|
1007
|
+
+ fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> {
|
|
1008
|
+
+ self.inner.blocking_list(path, args)
|
|
1009
|
+
+ }
|
|
1010
|
+
|
|
1011
|
+
+ fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> {
|
|
1012
|
+
+ self.inner.blocking_scan(path, args)
|
|
1013
|
+
+ }
|
|
1014
|
+
}
|
|
1015
|
+
```
|
|
1016
|
+
|
|
1017
|
+
## Usage of ops
|
|
1018
|
+
|
|
1019
|
+
To reduce the understanding overhead, we move all `OpXxx` into `opendal::ops` now. User may need to change:
|
|
1020
|
+
|
|
1021
|
+
```diff
|
|
1022
|
+
- use opendal::OpWrite;
|
|
1023
|
+
+ use opendal::ops::OpWrite;
|
|
1024
|
+
```
|
|
1025
|
+
|
|
1026
|
+
## Usage of RetryLayer
|
|
1027
|
+
|
|
1028
|
+
`backon` is the implementation detail of our `RetryLayer`, so we hide it from our public API. Users of `RetryLayer` need to change the code like:
|
|
1029
|
+
|
|
1030
|
+
```diff
|
|
1031
|
+
- RetryLayer::new(backon::ExponentialBackoff::default())
|
|
1032
|
+
+ RetryLayer::new()
|
|
1033
|
+
```
|
|
1034
|
+
|
|
1035
|
+
# Upgrade to v0.26
|
|
1036
|
+
|
|
1037
|
+
In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost.
|
|
1038
|
+
|
|
1039
|
+
Due to this change, we have to refactor the logic of `Operator`'s init logic. In v0.26, we added `opendal::Builder` trait and `opendal::OperatorBuilder`. For the first glance, the only change to existing code will be like:
|
|
1040
|
+
|
|
1041
|
+
```diff
|
|
1042
|
+
- let op = Operator::new(builder.build()?);
|
|
1043
|
+
+ let op = Operator::new(builder.build()?).finish();
|
|
1044
|
+
```
|
|
1045
|
+
|
|
1046
|
+
By adding a `finish()` call, we will erase all generic types so that `Operator` can still be easily used everywhere as before.
|
|
1047
|
+
|
|
1048
|
+
## Accessor
|
|
1049
|
+
|
|
1050
|
+
In v0.26, `Accessor` has been changed into trait with associated types.
|
|
1051
|
+
|
|
1052
|
+
All services need to declare the types returned as `Reader` or `BlockingReader`:
|
|
1053
|
+
|
|
1054
|
+
```rust
|
|
1055
|
+
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
|
|
1056
|
+
type Reader: output::Read;
|
|
1057
|
+
type BlockingReader: output::BlockingRead;
|
|
1058
|
+
}
|
|
1059
|
+
```
|
|
1060
|
+
|
|
1061
|
+
If your service doesn't support `read` or `blocking_read`, we can use `()` to represent a dummy reader:
|
|
1062
|
+
|
|
1063
|
+
```rust
|
|
1064
|
+
impl Accessor for MyDummyAccessor {
|
|
1065
|
+
type Reader = ();
|
|
1066
|
+
type BlockingReader = ();
|
|
1067
|
+
}
|
|
1068
|
+
```
|
|
1069
|
+
|
|
1070
|
+
## Layer
|
|
1071
|
+
|
|
1072
|
+
As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new `Layer` and `LayeredAccessor` trait:
|
|
1073
|
+
|
|
1074
|
+
```rust
|
|
1075
|
+
pub trait Layer<A: Accessor> {
|
|
1076
|
+
type LayeredAccessor: Accessor;
|
|
1077
|
+
|
|
1078
|
+
fn layer(&self, inner: A) -> Self::LayeredAccessor;
|
|
1079
|
+
}
|
|
1080
|
+
|
|
1081
|
+
#[async_trait]
|
|
1082
|
+
pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static {
|
|
1083
|
+
type Inner: Accessor;
|
|
1084
|
+
type Reader: output::Read;
|
|
1085
|
+
type BlockingReader: output::BlockingRead;
|
|
1086
|
+
}
|
|
1087
|
+
```
|
|
1088
|
+
|
|
1089
|
+
`LayeredAccessor` is a wrapper of `Accessor` with the typed `Innder`. All methods that not implemented will be forward to inner instead.
|
|
1090
|
+
|
|
1091
|
+
## Builder
|
|
1092
|
+
|
|
1093
|
+
Since v0.26, we implement `opendal::Builder` for all services, and services' mod will not be exported.
|
|
1094
|
+
|
|
1095
|
+
```diff
|
|
1096
|
+
- use opendal::services::s3::Builder;
|
|
1097
|
+
+ use opendal::services::S3;
|
|
1098
|
+
```
|
|
1099
|
+
|
|
1100
|
+
## Conclusion
|
|
1101
|
+
|
|
1102
|
+
Sorry again for the big changes in this release. It's a big step for OpenDAL to work in more critical systems.
|
|
1103
|
+
|
|
1104
|
+
# Upgrade to v0.25
|
|
1105
|
+
|
|
1106
|
+
In v0.25, we bring the same feature sets from `ObjectReader` to `BlockingObjectReader`.
|
|
1107
|
+
|
|
1108
|
+
Due to this change, all code that depends on `BlockingBytesReader` should be refactored.
|
|
1109
|
+
|
|
1110
|
+
- `BlockingBytesReader` => `input::BlockingReader`
|
|
1111
|
+
- `BlockingOutputBytesReader` => `output::BlockingReader`
|
|
1112
|
+
|
|
1113
|
+
Most changes only happen inside. Users not using `opendal::raw::*` will not be affected.
|
|
1114
|
+
|
|
1115
|
+
Apart from this change, we refactored s3 credential loading logic. After this change, we can disable the config load instead of the credential methods.
|
|
1116
|
+
|
|
1117
|
+
- `builder.disable_credential_loader` => `builder.disable_config_load`
|
|
1118
|
+
|
|
1119
|
+
# Upgrade to v0.24
|
|
1120
|
+
|
|
1121
|
+
In v0.24, we made a big refactor on our internal IO-related traits. In this version, we split our IO traits into `input` and `output` versions:
|
|
1122
|
+
|
|
1123
|
+
Take `Reader` as an example:
|
|
1124
|
+
|
|
1125
|
+
`input::Reader` is the user input reader, which only requires `futures::AsyncRead + Send`.
|
|
1126
|
+
|
|
1127
|
+
`output::Reader` is the reader returned by `OpenDAL`, which implements `futures::AsyncRead`, `futures::AsyncSeek`, and `futures::Stream<Item=io::Result<Bytes>>`. Besides, `output::Reader` also implements `Send + Sync`, which makes it useful for users.
|
|
1128
|
+
|
|
1129
|
+
Due to this change, all code that depends on `BytesReader` should be refactored.
|
|
1130
|
+
|
|
1131
|
+
- `BytesReader` => `input::Reader`
|
|
1132
|
+
- `OutputBytesReader` => `output::Reader`
|
|
1133
|
+
|
|
1134
|
+
Thanks to the change of IO trait split, we make `ObjectReader` implements all needed traits:
|
|
1135
|
+
|
|
1136
|
+
- `futures::AsyncRead`
|
|
1137
|
+
- `futures::AsyncSeek`
|
|
1138
|
+
- `futures::Stream<Item=io::Result<Bytes>>`
|
|
1139
|
+
|
|
1140
|
+
Thus, we removed the `seekable_reader` API. They can be replaced by `range_reader`:
|
|
1141
|
+
|
|
1142
|
+
- `o.seekable_reader` => `o.range_reader`
|
|
1143
|
+
|
|
1144
|
+
Most changes only happen inside. Users not using `opendal::raw::*` will not be affected.
|
|
1145
|
+
|
|
1146
|
+
Sorry for the inconvenience. I think those changes are required and make OpenDAL better! Welcome any comments at [Discussion](https://github.com/apache/opendal/discussions).
|
|
1147
|
+
|
|
1148
|
+
# Upgrade to v0.21
|
|
1149
|
+
|
|
1150
|
+
v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our `Accessor` APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.
|
|
1151
|
+
|
|
1152
|
+
Let's start with our errors.
|
|
1153
|
+
|
|
1154
|
+
## Error Handling
|
|
1155
|
+
|
|
1156
|
+
As described in [RFC-0977: Refactor Error](https://opendal.apache.org/rfcs/0977-refactor-error.html), we refactor opendal error by a new error
|
|
1157
|
+
called [`opendal::Error`](https://opendal.apache.org/opendal/struct.Error.html).
|
|
1158
|
+
|
|
1159
|
+
This change will affect all APIs that are used to return `io::Error`.
|
|
1160
|
+
|
|
1161
|
+
To migrate this, please replace `std::io::Error` with `opendal::Error`:
|
|
1162
|
+
|
|
1163
|
+
```diff
|
|
1164
|
+
- use std::io::Result;
|
|
1165
|
+
+ use opendal::Result;
|
|
1166
|
+
```
|
|
1167
|
+
|
|
1168
|
+
And the following error kinds should be updated:
|
|
1169
|
+
|
|
1170
|
+
- `std::io::ErrorKind::NotFound` => `opendal::ErrorKind::ObjectNotFound`
|
|
1171
|
+
- `std::io::ErrorKind::PermissionDenied` => `opendal::ErrorKind::ObjectPermissionDenied`
|
|
1172
|
+
|
|
1173
|
+
And since v0.21, we will return errors `ObjectIsADirectory` and `ObjectNotADirectory` instead of `anyhow::Error`.
|
|
1174
|
+
|
|
1175
|
+
## Accessor API
|
|
1176
|
+
|
|
1177
|
+
In v0.21, we refactor the whole `Accessor`'s API:
|
|
1178
|
+
|
|
1179
|
+
```diff
|
|
1180
|
+
- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
|
|
1181
|
+
+ async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>
|
|
1182
|
+
```
|
|
1183
|
+
|
|
1184
|
+
Since v0.21, we will return a reply struct for different operations called `RpWrite` instead of an exact type. We can split OpenDAL's public API and raw API with this change.
|
|
1185
|
+
|
|
1186
|
+
## ObjectList and Page
|
|
1187
|
+
|
|
1188
|
+
Since v0.21, `Accessor` will return `Pager` for `List`:
|
|
1189
|
+
|
|
1190
|
+
```diff
|
|
1191
|
+
- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
|
|
1192
|
+
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, output::Pager)>
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
And `Object` will return an `ObjectLister` which is built upon `Page`:
|
|
1196
|
+
|
|
1197
|
+
```rust
|
|
1198
|
+
pub async fn list(&self) -> Result<ObjectLister> { ... }
|
|
1199
|
+
```
|
|
1200
|
+
|
|
1201
|
+
`ObjectLister` can be used as an object stream as before. It also provides the function `next_page` to get the underlying pages directly:
|
|
1202
|
+
|
|
1203
|
+
```rust
|
|
1204
|
+
impl ObjectLister {
|
|
1205
|
+
pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
|
|
1206
|
+
}
|
|
1207
|
+
```
|
|
1208
|
+
|
|
1209
|
+
## Code Layout
|
|
1210
|
+
|
|
1211
|
+
Since v0.21, we have categorized all APIs into `public` and `raw`.
|
|
1212
|
+
|
|
1213
|
+
Public APIs are exposed under `opendal::Xxx`; they are user-face APIs that are easy to use and understand.
|
|
1214
|
+
|
|
1215
|
+
Raw APIs are exposed under `opendal::raw::Xxx`; they are implementation details for underlying services and layers.
|
|
1216
|
+
|
|
1217
|
+
Please replace all usage of `opendal::io_util::*` and `opendal::http_util::*` to `opendal::raw::*` instead.
|
|
1218
|
+
|
|
1219
|
+
With this change, new users of OpenDAL maybe be it easier to get started.
|
|
1220
|
+
|
|
1221
|
+
## Summary
|
|
1222
|
+
|
|
1223
|
+
Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.
|
|
1224
|
+
|
|
1225
|
+
# Upgrade to v0.20
|
|
1226
|
+
|
|
1227
|
+
v0.20 is a big release that we introduce a lot of performance related changes.
|
|
1228
|
+
|
|
1229
|
+
To make the best of information from `read` operation, we propose and implemented [RFC-0926: Object Reader](https://opendal.apache.org/rfcs/0926-object-reader.html). By this RFC, we can fetch content length from `ObjectReader` now!
|
|
1230
|
+
|
|
1231
|
+
```rust
|
|
1232
|
+
pub struct ObjectReader {
|
|
1233
|
+
inner: BytesReader
|
|
1234
|
+
meta: ObjectMetadata,
|
|
1235
|
+
}
|
|
1236
|
+
|
|
1237
|
+
impl ObjectReader {
|
|
1238
|
+
pub fn content_length(&self) -> u64 {}
|
|
1239
|
+
pub fn last_modified(&self) -> Option<OffsetDateTime> {}
|
|
1240
|
+
pub fn etag(&self) -> Option<String> {}
|
|
1241
|
+
}
|
|
1242
|
+
```
|
|
1243
|
+
|
|
1244
|
+
To make this happen, we changed our `Accessor` API:
|
|
1245
|
+
|
|
1246
|
+
```diff
|
|
1247
|
+
- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
|
|
1248
|
+
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}
|
|
1249
|
+
```
|
|
1250
|
+
|
|
1251
|
+
All layers should be updated to meet this change. Also, it's required to return `content_length` while building `ObjectReader`. Please make sure the returning `ObjectMetadata` is used correctly.
|
|
1252
|
+
|
|
1253
|
+
# Upgrade to v0.19
|
|
1254
|
+
|
|
1255
|
+
OpenDAL deprecate some features:
|
|
1256
|
+
|
|
1257
|
+
- `serde`: We will enable it by default.
|
|
1258
|
+
- `layers-retry`: We will enable retry support by default.
|
|
1259
|
+
- `layers-metadata-cache`: We will enable it by default.
|
|
1260
|
+
|
|
1261
|
+
Deprecated types like `DirEntry` has been removed.
|
|
1262
|
+
|
|
1263
|
+
# Upgrade to v0.18
|
|
1264
|
+
|
|
1265
|
+
OpenDAL v0.18 introduces the following breaking changes:
|
|
1266
|
+
|
|
1267
|
+
- Deprecated feature flag `services-http` has been removed.
|
|
1268
|
+
- All `DirXxx` items have been renamed to `ObjectXxx` to make them more consistent.
|
|
1269
|
+
- `DirEntry` -> `Entry`
|
|
1270
|
+
- `DirStream` -> `ObjectStream`
|
|
1271
|
+
- `DirStreamer` -> `ObjectStream`
|
|
1272
|
+
- `DirIterate` -> `ObjectIterate`
|
|
1273
|
+
- `DirIterator` -> `ObjectIterator`
|
|
1274
|
+
|
|
1275
|
+
Besides, we also make a big change to our `Entry` API. Since v0.18, we can fully reuse the metadata that fetched during `list`. Take `entry.content_length()` for example:
|
|
1276
|
+
|
|
1277
|
+
- If `content_length` is already known, we will return directly.
|
|
1278
|
+
- If not, we will check if the object entry is `complete`:
|
|
1279
|
+
- If `complete`, the entry already fetched all metadata that it could have, return directly.
|
|
1280
|
+
- If not, we will send a `stat` call to get the `metadata` and refresh our cache.
|
|
1281
|
+
|
|
1282
|
+
This change means:
|
|
1283
|
+
|
|
1284
|
+
- All API like `content_length` will be changed into async functions.
|
|
1285
|
+
- `metadata` and `blocking_metadata` will not return errors anymore.
|
|
1286
|
+
- To retrieve the latest meta, please use `entry.into_object().metadata()` instead.
|
|
1287
|
+
|
|
1288
|
+
# Upgrade to v0.17
|
|
1289
|
+
|
|
1290
|
+
OpenDAL v0.17 refactor the `Accessor` to make space for future features.
|
|
1291
|
+
|
|
1292
|
+
We move `path String` out of the `OpXxx` to function args so that we don't need to clone twice.
|
|
1293
|
+
|
|
1294
|
+
```diff
|
|
1295
|
+
- async fn read(&self, args: OpRead) -> Result<BytesReader>
|
|
1296
|
+
+ async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>
|
|
1297
|
+
```
|
|
1298
|
+
|
|
1299
|
+
For more information about this change, please refer to [RFC-0661: Path In Accessor](https://opendal.apache.org/rfcs/0661-path-in-accessor.html).
|
|
1300
|
+
|
|
1301
|
+
And since OpenDAL v0.17, we will use `rustls` as default tls engine for our underlying http client. Since this release, we will not depend on `openssl` anymore.
|
|
1302
|
+
|
|
1303
|
+
# Upgrade to v0.16
|
|
1304
|
+
|
|
1305
|
+
OpenDAL v0.16 refactor the internal implementation of `http` service. Since v0.16, http service can be used directly without enabling `services-http` feature. Accompany by these changes, http service has the following breaking changes:
|
|
1306
|
+
|
|
1307
|
+
- `services-http` feature has been deprecated. Enabling `services-http` is a no-op now.
|
|
1308
|
+
- http service is read only services and can't be used to `list` or `write`.
|
|
1309
|
+
|
|
1310
|
+
OpenDAL introduces a new layer `ImmutableIndexLayer` that can add `list` capability for services:
|
|
1311
|
+
|
|
1312
|
+
```rust
|
|
1313
|
+
use opendal::layers::ImmutableIndexLayer;
|
|
1314
|
+
use opendal::Operator;
|
|
1315
|
+
use opendal::Scheme;
|
|
1316
|
+
|
|
1317
|
+
async fn main() {
|
|
1318
|
+
let mut iil = ImmutableIndexLayer::default();
|
|
1319
|
+
|
|
1320
|
+
for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] {
|
|
1321
|
+
iil.insert(i.to_string())
|
|
1322
|
+
}
|
|
1323
|
+
|
|
1324
|
+
let op = Operator::from_env(Scheme::Http)?.layer(iil);
|
|
1325
|
+
}
|
|
1326
|
+
```
|
|
1327
|
+
|
|
1328
|
+
For more information about this change, please refer to [RFC-0627: Split Capabilities](https://opendal.apache.org/rfcs/0627-split-capabilities.html).
|
|
1329
|
+
|
|
1330
|
+
# Upgrade to v0.14
|
|
1331
|
+
|
|
1332
|
+
OpenDAL v0.14 removed all deprecated APIs in previous versions, including:
|
|
1333
|
+
|
|
1334
|
+
- `Operator::with_backoff` in v0.13
|
|
1335
|
+
- All services `Builder::finish()` in v0.12
|
|
1336
|
+
- All services `Backend::build()` in v0.12
|
|
1337
|
+
|
|
1338
|
+
Please visit related version's upgrade guide for migration.
|
|
1339
|
+
|
|
1340
|
+
And in OpenDAL v0.14, we introduce a break change for `write` operations.
|
|
1341
|
+
|
|
1342
|
+
```diff
|
|
1343
|
+
pub trait Accessor {
|
|
1344
|
+
- async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
|
|
1345
|
+
+ async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
|
|
1346
|
+
}
|
|
1347
|
+
```
|
|
1348
|
+
|
|
1349
|
+
The following APIs have affected by this change:
|
|
1350
|
+
|
|
1351
|
+
- `Object::write` now accept `impl Into<Vec<u8>>` instead of `AsRef<&[u8]>`
|
|
1352
|
+
- `Object::writer` has been removed.
|
|
1353
|
+
- `Object::write_from` has been added to support write from a reader.
|
|
1354
|
+
- All layers should be refactored to adapt new `Accessor` trait.
|
|
1355
|
+
|
|
1356
|
+
For more information about this change, please refer to [RFC-0554: Write Refactor](https://opendal.apache.org/rfcs/0554-write-refactor.html).
|
|
1357
|
+
|
|
1358
|
+
# Upgrade to v0.13
|
|
1359
|
+
|
|
1360
|
+
OpenDAL deprecate `Operator::with_backoff` since v0.13.
|
|
1361
|
+
|
|
1362
|
+
Please use [`RetryLayer`](https://opendal.apache.org/opendal/layers/struct.RetryLayer.html) instead:
|
|
1363
|
+
|
|
1364
|
+
```rust
|
|
1365
|
+
use anyhow::Result;
|
|
1366
|
+
use backon::ExponentialBackoff;
|
|
1367
|
+
use opendal::layers::RetryLayer;
|
|
1368
|
+
use opendal::Operator;
|
|
1369
|
+
use opendal::Scheme;
|
|
1370
|
+
|
|
1371
|
+
let _ = Operator::from_env(Scheme::Fs)
|
|
1372
|
+
.expect("must init")
|
|
1373
|
+
.layer(RetryLayer::new(ExponentialBackoff::default()));
|
|
1374
|
+
```
|
|
1375
|
+
|
|
1376
|
+
# Upgrade to v0.12
|
|
1377
|
+
|
|
1378
|
+
OpenDAL introduces breaking changes for services initiation.
|
|
1379
|
+
|
|
1380
|
+
Since v0.12, `Operator::new` will accept `impl Accessor + 'static` instead of `Arc<dyn Accessor>`:
|
|
1381
|
+
|
|
1382
|
+
```rust
|
|
1383
|
+
impl Operator {
|
|
1384
|
+
pub fn new(accessor: impl Accessor + 'static) -> Self { .. }
|
|
1385
|
+
}
|
|
1386
|
+
```
|
|
1387
|
+
|
|
1388
|
+
Every service's `Builder` now have a `build()` API which can be run without async:
|
|
1389
|
+
|
|
1390
|
+
```rust
|
|
1391
|
+
let mut builder = fs::Builder::default();
|
|
1392
|
+
let op: Operator = Operator::new(builder.build()?);
|
|
1393
|
+
```
|
|
1394
|
+
|
|
1395
|
+
Along with these changes, `Operator::from_iter` and `Operator::from_env` now is a blocking API too.
|
|
1396
|
+
|
|
1397
|
+
For more information about this change, please refer to [RFC-0501: New Builder](https://opendal.apache.org/rfcs/0501-new-builder.html).
|
|
1398
|
+
|
|
1399
|
+
The following APIs have been deprecated:
|
|
1400
|
+
|
|
1401
|
+
- All services `Builder::finish()` (replaced by `Builder::build()`)
|
|
1402
|
+
- All services `Backend::build()` (replace by `Builder::default()`)
|
|
1403
|
+
|
|
1404
|
+
The following APIs have been removed:
|
|
1405
|
+
|
|
1406
|
+
- public struct `Metadata` (deprecated in v0.8, replaced by `ObjectMetadata`)
|
|
1407
|
+
|
|
1408
|
+
# Upgrade to v0.8
|
|
1409
|
+
|
|
1410
|
+
OpenDAL introduces a breaking change of `list` related operations in v0.8.
|
|
1411
|
+
|
|
1412
|
+
Since v0.8, `list` will return `DirStreamer` instead:
|
|
1413
|
+
|
|
1414
|
+
```rust
|
|
1415
|
+
pub trait Accessor: Send + Sync + Debug {
|
|
1416
|
+
async fn list(&self, args: &OpList) -> Result<DirStreamer> {}
|
|
1417
|
+
}
|
|
1418
|
+
```
|
|
1419
|
+
|
|
1420
|
+
`DirStreamer` streams `DirEntry` which carries `ObjectMode`, so that we don't need an extra call to get object mode:
|
|
1421
|
+
|
|
1422
|
+
```rust
|
|
1423
|
+
impl DirEntry {
|
|
1424
|
+
pub fn mode(&self) -> ObjectMode {
|
|
1425
|
+
self.mode
|
|
1426
|
+
}
|
|
1427
|
+
}
|
|
1428
|
+
```
|
|
1429
|
+
|
|
1430
|
+
And `DirEntry` can be converted into `Object` without overhead:
|
|
1431
|
+
|
|
1432
|
+
```rust
|
|
1433
|
+
let o: Object = de.into()
|
|
1434
|
+
```
|
|
1435
|
+
|
|
1436
|
+
Since `v0.8`, `opendal::Metadata` has been deprecated by `opendal::ObjectMetadata`.
|
|
1437
|
+
|
|
1438
|
+
# Upgrade to v0.7
|
|
1439
|
+
|
|
1440
|
+
OpenDAL introduces a breaking change of `decompress_read` related in v0.7.
|
|
1441
|
+
|
|
1442
|
+
Since v0.7, `decompress_read` and `decompress_reader` will return `Ok(None)` while OpenDAL can't detect the correct compress algorithm.
|
|
1443
|
+
|
|
1444
|
+
```rust
|
|
1445
|
+
impl Object {
|
|
1446
|
+
pub async fn decompress_read(&self) -> Result<Option<Vec<u8>>> {}
|
|
1447
|
+
pub async fn decompress_reader(&self) -> Result<Option<impl BytesRead>> {}
|
|
1448
|
+
}
|
|
1449
|
+
```
|
|
1450
|
+
|
|
1451
|
+
So users should match and check the `None` case:
|
|
1452
|
+
|
|
1453
|
+
```rust
|
|
1454
|
+
let bs = o.decompress_read().await?.expect("must have valid compress algorithm");
|
|
1455
|
+
```
|
|
1456
|
+
|
|
1457
|
+
# Upgrade to v0.4
|
|
1458
|
+
|
|
1459
|
+
OpenDAL introduces many breaking changes in v0.4.
|
|
1460
|
+
|
|
1461
|
+
## Object::reader() is not `AsyncSeek` anymore
|
|
1462
|
+
|
|
1463
|
+
Since v0.4, `Object::reader()` will return `impl BytesRead` instead of `Reader` that implements `AsyncRead` and `AsyncSeek`. Users who want `AsyncSeek` please wrapped with `opendal::io_util::seekable_read`:
|
|
1464
|
+
|
|
1465
|
+
```rust
|
|
1466
|
+
use opendal::io_util::seekable_read;
|
|
1467
|
+
|
|
1468
|
+
let o = op.object("test");
|
|
1469
|
+
let mut r = seekable_read(&o, 10..);
|
|
1470
|
+
r.seek(SeekFrom::Current(10)).await?;
|
|
1471
|
+
let mut bs = vec![0;10];
|
|
1472
|
+
r.read(&mut bs).await?;
|
|
1473
|
+
```
|
|
1474
|
+
|
|
1475
|
+
## Use RangeBounds instead
|
|
1476
|
+
|
|
1477
|
+
Since v0.4, the following APIs will be removed.
|
|
1478
|
+
|
|
1479
|
+
- `Object::limited_reader(size: u64)`
|
|
1480
|
+
- `Object::offset_reader(offset: u64)`
|
|
1481
|
+
- `Object::range_reader(offset: u64, size: u64)`
|
|
1482
|
+
|
|
1483
|
+
Instead, OpenDAL is providing a more general `range_reader` powered by `RangeBounds`:
|
|
1484
|
+
|
|
1485
|
+
```rust
|
|
1486
|
+
pub async fn range_reader(&self, range: impl RangeBounds<u64>) -> Result<impl BytesRead>
|
|
1487
|
+
```
|
|
1488
|
+
|
|
1489
|
+
Users can use their familiar rust range syntax:
|
|
1490
|
+
|
|
1491
|
+
```rust
|
|
1492
|
+
let r = o.range_reader(1024..2048).await?;
|
|
1493
|
+
```
|
|
1494
|
+
|
|
1495
|
+
## Return io::Result instead
|
|
1496
|
+
|
|
1497
|
+
Since v0.4, all functions in OpenDAL will return `std::io::Result` instead.
|
|
1498
|
+
|
|
1499
|
+
Please check via `std::io::ErrorKind` directly:
|
|
1500
|
+
|
|
1501
|
+
```rust
|
|
1502
|
+
use std::io::ErrorKind;
|
|
1503
|
+
|
|
1504
|
+
if let Err(e) = op.object("test_file").metadata().await {
|
|
1505
|
+
if e.kind() == ErrorKind::NotFound {
|
|
1506
|
+
println!("object not exist")
|
|
1507
|
+
}
|
|
1508
|
+
}
|
|
1509
|
+
```
|
|
1510
|
+
|
|
1511
|
+
## Removing Credential
|
|
1512
|
+
|
|
1513
|
+
Since v0.4, `Credential` has been removed, please use the API provided by `Builder` directly.
|
|
1514
|
+
|
|
1515
|
+
```rust
|
|
1516
|
+
builder.access_key_id("access_key_id");
|
|
1517
|
+
builder.secret_access_key("secret_access_key");
|
|
1518
|
+
```
|
|
1519
|
+
|
|
1520
|
+
## Write returns `BytesWriter` instead
|
|
1521
|
+
|
|
1522
|
+
Since v0.4, `Accessor::write` will return a `BytesWriter` instead accepting a `BoxedAsyncReader`.
|
|
1523
|
+
|
|
1524
|
+
Along with this change, the old `Writer` has been replaced by a new set of write functions:
|
|
1525
|
+
|
|
1526
|
+
```rust
|
|
1527
|
+
pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
|
|
1528
|
+
pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}
|
|
1529
|
+
```
|
|
1530
|
+
|
|
1531
|
+
Users can write into an object more easily:
|
|
1532
|
+
|
|
1533
|
+
```rust
|
|
1534
|
+
let _ = op.object("path/to/file").write("Hello, World!").await?;
|
|
1535
|
+
```
|
|
1536
|
+
|
|
1537
|
+
## `io_util` replaces `readers`
|
|
1538
|
+
|
|
1539
|
+
Since v0.4, mod `io_util` will replace `readers`. In `io_utils`, OpenDAL provides helpful functions like:
|
|
1540
|
+
|
|
1541
|
+
- `into_reader`: Convert `BytesStream` into `BytesRead`
|
|
1542
|
+
- `into_sink`: Convert `BytesWrite` into `BytesSink`
|
|
1543
|
+
- `into_stream`: Convert `BytesRead` into `BytesStream`
|
|
1544
|
+
- `into_writer`: Convert `BytesSink` into `BytesWrite`
|
|
1545
|
+
- `observe_read`: Add callback for `BytesReader`
|
|
1546
|
+
- `observe_write`: Add callback for `BytesWrite`
|
|
1547
|
+
|
|
1548
|
+
## New type alias
|
|
1549
|
+
|
|
1550
|
+
For better naming, types that OpenDAL returns have been renamed:
|
|
1551
|
+
|
|
1552
|
+
- `AsyncRead + Unpin + Send` => `BytesRead`
|
|
1553
|
+
- `BoxedAsyncReader` => `BytesReader`
|
|
1554
|
+
- `AsyncWrite + Unpin + Send` => `BytesWrite`
|
|
1555
|
+
- `BoxedAsyncWriter` => `BytesWriter`
|
|
1556
|
+
- `ObjectStream` => `ObjectStreamer`
|