cloud-files 5.5.0__py3-none-any.whl → 5.6.0__py3-none-any.whl

{cloud_files-5.5.0.dist-info → cloud_files-5.6.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cloud-files
-Version: 5.5.0
+Version: 5.6.0
 Summary: Fast access to cloud storage and local FS.
 Home-page: https://github.com/seung-lab/cloud-files/
 Author: William Silversmith
@@ -30,6 +30,7 @@ Requires-Dist: google-auth >=1.10.0
 Requires-Dist: google-cloud-core >=1.1.0
 Requires-Dist: google-cloud-storage >=1.31.1
 Requires-Dist: google-crc32c >=1.0.0
+Requires-Dist: intervaltree
 Requires-Dist: orjson
 Requires-Dist: pathos
 Requires-Dist: protobuf >=3.3.0
@@ -41,6 +42,12 @@ Requires-Dist: urllib3 >=1.26.3
 Requires-Dist: zstandard
 Requires-Dist: rsa >=4.7.2
 Requires-Dist: fasteners
+Provides-Extra: apache
+Requires-Dist: lxml ; extra == 'apache'
+Provides-Extra: monitoring
+Requires-Dist: psutil ; extra == 'monitoring'
+Requires-Dist: intervaltree ; extra == 'monitoring'
+Requires-Dist: matplotlib ; extra == 'monitoring'
 Provides-Extra: numpy
 Requires-Dist: numpy ; extra == 'numpy'
 Provides-Extra: test
@@ -97,8 +104,20 @@ cf.touch([ "example", "example2" ])
 cf = CloudFile("gs://bucket/file1")
 info = cf.head()
 binary = cf.get()
+obj = cf.get_json()
 cf.put(binary)
+cf.put_json()
 cf[:30] # get first 30 bytes of file
+
+num_bytes = cf.size() # get size in bytes (also in head)
+exists = cf.exists() # true or false
+cf.delete() # deletes the file
+cf.touch() # create the file if it doesn't exist
+cf.move("gs://example/destination/directory") # copy then delete source
+cf.transfer_from("gs://example/source/file.txt") # copies file efficiently
+cf.transfer_to("gs://example/dest/file.txt") # copies file efficiently
+
+path = cf.join([ path1, path2, path3 ]) # use the appropriate path separator
 ```
 
 CloudFiles was developed to access files from object storage without ever touching disk. The goal was to reliably and rapidly access a petabyte of image data broken down into tens to hundreds of millions of files being accessed in parallel across thousands of cores. CloudFiles has been used to processes dozens of images, many of which were in the hundreds of terabyte range. It has reliably read and written tens of billions of files to date.
@@ -124,6 +143,7 @@ CloudFiles was developed to access files from object storage without ever touchi
 ```bash
 pip install cloud-files
 pip install cloud-files[test] # to enable testing with pytest
+pip install cloud-files[monitoring] # enable plotting network performance
 ```
 
 If you run into trouble installing dependenies, make sure you're using at least Python3.6 and you have updated pip. On Linux, some dependencies require manylinux2010 or manylinux2014 binaries which earlier versions of pip do not search for. MacOS, Linux, and Windows are supported platforms.
@@ -176,7 +196,9 @@ You can create the `google-secret.json` file [here](https://console.cloud.google
 
 ## API Documentation  
 
-Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations.  
+Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations. 
+
+`CloudFile` is a more intuitive version of `CloudFiles` designed for managing single files instead of groups of files. See examples above. There is an analogus method for each `CloudFiles` method (where it makes sense).
 
 ### Constructor
 ```python
@@ -236,6 +258,10 @@ binary = cf['filename'][0:5] # same result, fetches 11 bytes
 >> b'hello' # represents byte range 0-4 inclusive of filename
 
 binaries = cf[:100] # download the first 100 files
+
+# Get the TransmissionMonitor object that records
+# the flight time of each file.
+binaries, tm = cf.get(..., return_recording=True)
 ```
 
 `get` supports several different styles of input. The simplest takes a scalar filename and returns the contents of that file. However, you can also specify lists of filenames, a byte range request, and lists of byte range requests. You can provide a generator or iterator as input as well. Order is not guaranteed.
@@ -265,6 +291,10 @@ cf.puts([{
 cf.puts([ (path, content), (path, content) ], compression='gzip')
 cf.put_jsons(...)
 
+# Get the TransmissionMonitor object that records the
+# flight times of each object.
+_, tm = cf.puts(..., return_recording=True)
+
 # Definition of put, put_json is identical
 def put(
     self, 
@@ -469,6 +499,12 @@ cloudfiles -p 2 cp --progress -r s3://bkt/ gs://bkt2/
 cloudfiles cp -c br s3://bkt/file.txt gs://bkt2/
 # decompress
 cloudfiles cp -c none s3://bkt/file.txt gs://bkt2/
+# save chart of file flight times
+cloudfiles cp --flight-time s3://bkt/file.txt gs://bkt2/
+# save a chart of estimated bandwidth usage from these files alone
+cloudfiles cp --io-rate s3://bkt/file.txt gs://bkt2/
+# save a chart of measured bandwidth usage for the machine
+cloudfiles cp --machine-io-rate s3://bkt/file.txt gs://bkt2/
 # move or rename files
 cloudfiles mv s3://bkt/file.txt gs://bkt2/
 # create an empty file if not existing
@@ -528,6 +564,40 @@ cloudfiles alias rm example # remove example://
 
 The alias file is only accessed (and cached) if CloudFiles encounters an unknown protocol. If you stick to default protocols and use the syntax `s3://https://example.com/` for alternative endpoints, you can still use CloudFiles in environments without filesystem access.
 
+## Performance Monitoring
+
+CloudFiles now comes with two tools inside of `cloudfiles.monitoring` for measuring the performance of transfer operations both via the CLI and the programatic interface.
+
+A `TransmissionMonitor` object is created during each download or upload (e.g. `cf.get` or `cf.puts`) call. You can access this object by using the `return_recording=True` flag. This object saves the flight times of each object along with its size in an interval tree datastructure. It comes with methods for estimating the peak bits per a second and can plot both time of flight and the estimated transfer rates (assuming the transfer is evenly divided over the flight of an object, an assumption that is not always true). This allows you to estimate the contribution of a given CloudFiles operation to a machine's network IO.
+
+```python
+from cloudfiles import CloudFiles
+
+... 
+
+results, tm = cf.get([ ... some files ... ], return_recording=True)
+
+value = tm.peak_Mbps() # estimated peak transfer rate
+value = tm.total_Mbps() # estimated average transfer rate
+tm.plot_gantt() # time of flight chart
+tm.plot_histogram() # transfer rate chart
+```
+
+A second object, `IOSampler`, can sample the OS network counters using a background thread and provides a global view of the machine's network performance during the life of the transfer. It is enabled on the CLI for the `cp` command when the `--machine-io-rate` flag is enabled, but must be manually started programatically. This is to avoid accidentally starting unnecessary sampling threads. The samples are accumulated into a circular buffer, so make sure to set the buffer length long enough for your points of interest to be captured.
+
+```python
+from cloudfiles.monitoring import IOSampler
+
+sampler = IOSampler(buffer_sec=600, interval=0.25)
+sampler.start_sampling()
+
+...
+
+
+sampler.stop_sampling()
+sampler.plot_histogram()
+```
+
 ## Credits
 
 CloudFiles is derived from the [CloudVolume.Storage](https://github.com/seung-lab/cloud-volume/tree/master/cloudvolume/storage) system.  

{cloud_files-5.5.0.dist-info → cloud_files-5.6.0.dist-info}/RECORD

@@ -1,11 +1,12 @@
 cloudfiles/__init__.py,sha256=pLB4CcV2l3Jgv_ni1520Np1pfzFj8Cpr87vNxFT3rNI,493
-cloudfiles/cloudfiles.py,sha256=nUIW-SDyeldvmUu_SETJXS4x38Nv3Kj7TBwMwLQhjns,51196
+cloudfiles/cloudfiles.py,sha256=tPG1PBLEjABPu-KLe93yf6xW_zbafPsQ6z5NuofyUoU,56743
 cloudfiles/compression.py,sha256=WXJHnoNLJ_NWyoY9ygZmFA2qMou35_9xS5dzF7-2H-M,6262
 cloudfiles/connectionpools.py,sha256=aL8RiSjRepECfgAFmJcz80aJFKbou7hsbuEgugDKwB8,4814
 cloudfiles/exceptions.py,sha256=N0oGQNG-St6RvnT8e5p_yC_E61q2kgAe2scwAL0F49c,843
 cloudfiles/gcs.py,sha256=unqu5KxGKaPq6N4QeHSpCDdtnK1BzPOAerTZ8FLt2_4,3820
-cloudfiles/interfaces.py,sha256=3_ybsCKf74DPpI5Vv-BwFLLzc1cLUMfZLBtIKWkfoHQ,44478
+cloudfiles/interfaces.py,sha256=5rUh2DWOVlg13fAxyZ0wAaQyfW04xc2zlUfTItFV-zQ,45325
 cloudfiles/lib.py,sha256=HHjCvjmOjA0nZWSvHGoSeYpxqd6FAG8xk8LM212LAUA,5382
+cloudfiles/monitoring.py,sha256=N5Xq0PYZK1OxoYtwBFsnnfaq7dElTgY8Rn2Ez_I3aoo,20897
 cloudfiles/paths.py,sha256=HOvtdLSIYGwlwvnZt9d_Ez3TXOe7WWd18bZNDpExUDQ,12231
 cloudfiles/resumable_tools.py,sha256=NyuSoGh1SaP5akrHCpd9kgy2-JruEWrHW9lvJxV7jpE,6711
 cloudfiles/scheduler.py,sha256=ioqBT5bMPCVHEHlnL-SW_wHulxGgjeThiKHlnaDOydo,3831
@@ -15,12 +16,12 @@ cloudfiles/threaded_queue.py,sha256=Nl4vfXhQ6nDLF8PZpSSBpww0M2zWtcd4DLs3W3BArBw,
 cloudfiles/typing.py,sha256=f3ZYkNfN9poxhGu5j-P0KCxjCCqSn9HAg5KiIPkjnCg,416
 cloudfiles_cli/LICENSE,sha256=Jna4xYE8CCQmaxjr5Fs-wmUBnIQJ1DGcNn9MMjbkprk,1538
 cloudfiles_cli/__init__.py,sha256=Wftt3R3F21QsHtWqx49ODuqT9zcSr0em7wk48kcH0WM,29
-cloudfiles_cli/cloudfiles_cli.py,sha256=L9xr_z9GCf5PcpmA21Iandim7uMviaLxTZ4Pf1B9Ui0,34865
-cloud_files-5.5.0.dist-info/AUTHORS,sha256=BFVmobgAhaVFI5fqbuqAY5XmBQxe09ZZAsAOTy87hKQ,318
-cloud_files-5.5.0.dist-info/LICENSE,sha256=Jna4xYE8CCQmaxjr5Fs-wmUBnIQJ1DGcNn9MMjbkprk,1538
-cloud_files-5.5.0.dist-info/METADATA,sha256=m3Rlk29z4XnqeLhpKX613g7KMiT-5vGVNsWpjXNfXmU,26970
-cloud_files-5.5.0.dist-info/WHEEL,sha256=GJ7t_kWBFywbagK5eo9IoUwLW6oyOeTKmQ-9iHFVNxQ,92
-cloud_files-5.5.0.dist-info/entry_points.txt,sha256=xlirb1FVhn1mbcv4IoyMEGumDqKOA4VMVd3drsRQxIg,51
-cloud_files-5.5.0.dist-info/pbr.json,sha256=k1LwgnAaBImOMXKnlbOx2sBAWZ_Jm5AdFAhAugTuvrY,46
-cloud_files-5.5.0.dist-info/top_level.txt,sha256=xPyrST3okJbsmdCF5IC2gYAVxg_aD5AYVTnNo8UuoZU,26
-cloud_files-5.5.0.dist-info/RECORD,,
+cloudfiles_cli/cloudfiles_cli.py,sha256=IT6CHH7rI2B79ecV526uFa6ruAC8Yg25OO36Uwfxenc,37464
+cloud_files-5.6.0.dist-info/AUTHORS,sha256=BFVmobgAhaVFI5fqbuqAY5XmBQxe09ZZAsAOTy87hKQ,318
+cloud_files-5.6.0.dist-info/LICENSE,sha256=Jna4xYE8CCQmaxjr5Fs-wmUBnIQJ1DGcNn9MMjbkprk,1538
+cloud_files-5.6.0.dist-info/METADATA,sha256=vXFDHg7VwfvoIlQlQZR6bqS-SqlPI7O7vUe8gSNcnGI,30509
+cloud_files-5.6.0.dist-info/WHEEL,sha256=GJ7t_kWBFywbagK5eo9IoUwLW6oyOeTKmQ-9iHFVNxQ,92
+cloud_files-5.6.0.dist-info/entry_points.txt,sha256=xlirb1FVhn1mbcv4IoyMEGumDqKOA4VMVd3drsRQxIg,51
+cloud_files-5.6.0.dist-info/pbr.json,sha256=YPRBUS6pO0tlB43cAfdmW2Lb_kpuL4KYmVzrlj7hjr8,46
+cloud_files-5.6.0.dist-info/top_level.txt,sha256=xPyrST3okJbsmdCF5IC2gYAVxg_aD5AYVTnNo8UuoZU,26
+cloud_files-5.6.0.dist-info/RECORD,,

cloud_files-5.6.0.dist-info/pbr.json

@@ -0,0 +1 @@
+{"git_version": "4a9dd39", "is_release": true}