cloud-files 5.5.0__tar.gz → 5.6.1__tar.gz

{cloud_files-5.5.0 → cloud_files-5.6.1}/.github/workflows/test-suite.yml

@@ -26,7 +26,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        if [ -f requirements.txt ]; then pip install -e ".[test]"; fi
+        if [ -f requirements.txt ]; then pip install -e ".[test,monitoring]"; fi
     - name: Test with pytest
       run: |
         python -m pytest -v -x automated_test.py

{cloud_files-5.5.0 → cloud_files-5.6.1}/ChangeLog

@@ -1,6 +1,22 @@
 CHANGES
 =======
 
+5.6.1
+-----
+
+* install: monitoring code requires numpy
+
+5.6.0
+-----
+
+* docs: errata in how to use performance monitoring tools
+* feat: performance monitoring (#117)
+* docs: describe the cloudfiles constructor
+* fix(cli): rm can interpret aliases properly
+* fix: remove streaming for small s3 uploads
+* docs: give more examples for CloudFile
+* perf: aws-chunked was a bug in moto, that's now fixed. Stop adding copy\_object
+
 5.5.0
 -----
 

{cloud_files-5.5.0 → cloud_files-5.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cloud-files
-Version: 5.5.0
+Version: 5.6.1
 Summary: Fast access to cloud storage and local FS.
 Home-page: https://github.com/seung-lab/cloud-files/
 Author: William Silversmith
@@ -30,6 +30,8 @@ Requires-Dist: google-auth>=1.10.0
 Requires-Dist: google-cloud-core>=1.1.0
 Requires-Dist: google-cloud-storage>=1.31.1
 Requires-Dist: google-crc32c>=1.0.0
+Requires-Dist: intervaltree
+Requires-Dist: numpy
 Requires-Dist: orjson
 Requires-Dist: pathos
 Requires-Dist: protobuf>=3.3.0
@@ -46,6 +48,12 @@ Requires-Dist: pytest; extra == "test"
 Requires-Dist: moto>=5; extra == "test"
 Provides-Extra: numpy
 Requires-Dist: numpy; extra == "numpy"
+Provides-Extra: monitoring
+Requires-Dist: psutil; extra == "monitoring"
+Requires-Dist: intervaltree; extra == "monitoring"
+Requires-Dist: matplotlib; extra == "monitoring"
+Provides-Extra: apache
+Requires-Dist: lxml; extra == "apache"
 
 [![PyPI version](https://badge.fury.io/py/cloud-files.svg)](https://badge.fury.io/py/cloud-files) [![Test Suite](https://github.com/seung-lab/cloud-files/workflows/Test%20Suite/badge.svg)](https://github.com/seung-lab/cloud-files/actions?query=workflow%3A%22Test+Suite%22)
 
@@ -97,8 +105,20 @@ cf.touch([ "example", "example2" ])
 cf = CloudFile("gs://bucket/file1")
 info = cf.head()
 binary = cf.get()
+obj = cf.get_json()
 cf.put(binary)
+cf.put_json()
 cf[:30] # get first 30 bytes of file
+
+num_bytes = cf.size() # get size in bytes (also in head)
+exists = cf.exists() # true or false
+cf.delete() # deletes the file
+cf.touch() # create the file if it doesn't exist
+cf.move("gs://example/destination/directory") # copy then delete source
+cf.transfer_from("gs://example/source/file.txt") # copies file efficiently
+cf.transfer_to("gs://example/dest/file.txt") # copies file efficiently
+
+path = cf.join([ path1, path2, path3 ]) # use the appropriate path separator
 ```
 
 CloudFiles was developed to access files from object storage without ever touching disk. The goal was to reliably and rapidly access a petabyte of image data broken down into tens to hundreds of millions of files being accessed in parallel across thousands of cores. CloudFiles has been used to processes dozens of images, many of which were in the hundreds of terabyte range. It has reliably read and written tens of billions of files to date.
@@ -124,6 +144,7 @@ CloudFiles was developed to access files from object storage without ever touchi
 ```bash
 pip install cloud-files
 pip install cloud-files[test] # to enable testing with pytest
+pip install cloud-files[monitoring] # enable plotting network performance
 ```
 
 If you run into trouble installing dependenies, make sure you're using at least Python3.6 and you have updated pip. On Linux, some dependencies require manylinux2010 or manylinux2014 binaries which earlier versions of pip do not search for. MacOS, Linux, and Windows are supported platforms.
@@ -176,7 +197,9 @@ You can create the `google-secret.json` file [here](https://console.cloud.google
 
 ## API Documentation  
 
-Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations.  
+Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations. 
+
+`CloudFile` is a more intuitive version of `CloudFiles` designed for managing single files instead of groups of files. See examples above. There is an analogus method for each `CloudFiles` method (where it makes sense).
 
 ### Constructor
 ```python
@@ -236,6 +259,10 @@ binary = cf['filename'][0:5] # same result, fetches 11 bytes
 >> b'hello' # represents byte range 0-4 inclusive of filename
 
 binaries = cf[:100] # download the first 100 files
+
+# Get the TransmissionMonitor object that records
+# the flight time of each file.
+binaries, tm = cf.get(..., return_recording=True)
 ```
 
 `get` supports several different styles of input. The simplest takes a scalar filename and returns the contents of that file. However, you can also specify lists of filenames, a byte range request, and lists of byte range requests. You can provide a generator or iterator as input as well. Order is not guaranteed.
@@ -265,6 +292,10 @@ cf.puts([{
 cf.puts([ (path, content), (path, content) ], compression='gzip')
 cf.put_jsons(...)
 
+# Get the TransmissionMonitor object that records the
+# flight times of each object.
+_, tm = cf.puts(..., return_recording=True)
+
 # Definition of put, put_json is identical
 def put(
     self, 
@@ -469,6 +500,12 @@ cloudfiles -p 2 cp --progress -r s3://bkt/ gs://bkt2/
 cloudfiles cp -c br s3://bkt/file.txt gs://bkt2/
 # decompress
 cloudfiles cp -c none s3://bkt/file.txt gs://bkt2/
+# save chart of file flight times
+cloudfiles cp --flight-time s3://bkt/file.txt gs://bkt2/
+# save a chart of estimated bandwidth usage from these files alone
+cloudfiles cp --io-rate s3://bkt/file.txt gs://bkt2/
+# save a chart of measured bandwidth usage for the machine
+cloudfiles cp --machine-io-rate s3://bkt/file.txt gs://bkt2/
 # move or rename files
 cloudfiles mv s3://bkt/file.txt gs://bkt2/
 # create an empty file if not existing
@@ -528,6 +565,40 @@ cloudfiles alias rm example # remove example://
 
 The alias file is only accessed (and cached) if CloudFiles encounters an unknown protocol. If you stick to default protocols and use the syntax `s3://https://example.com/` for alternative endpoints, you can still use CloudFiles in environments without filesystem access.
 
+## Performance Monitoring
+
+CloudFiles now comes with two tools inside of `cloudfiles.monitoring` for measuring the performance of transfer operations both via the CLI and the programatic interface.
+
+A `TransmissionMonitor` object is created during each download or upload (e.g. `cf.get` or `cf.puts`) call. You can access this object by using the `return_recording=True` flag. This object saves the flight times of each object along with its size in an interval tree datastructure. It comes with methods for estimating the peak bits per a second and can plot both time of flight and the estimated transfer rates (assuming the transfer is evenly divided over the flight of an object, an assumption that is not always true). This allows you to estimate the contribution of a given CloudFiles operation to a machine's network IO.
+
+```python
+from cloudfiles import CloudFiles
+
+... 
+
+results, tm = cf.get([ ... some files ... ], return_recording=True)
+
+value = tm.peak_Mbps() # estimated peak transfer rate
+value = tm.total_Mbps() # estimated average transfer rate
+tm.plot_gantt() # time of flight chart
+tm.plot_histogram() # transfer rate chart
+```
+
+A second object, `IOSampler`, can sample the OS network counters using a background thread and provides a global view of the machine's network performance during the life of the transfer. It is enabled on the CLI for the `cp` command when the `--machine-io-rate` flag is enabled, but must be manually started programatically. This is to avoid accidentally starting unnecessary sampling threads. The samples are accumulated into a circular buffer, so make sure to set the buffer length long enough for your points of interest to be captured.
+
+```python
+from cloudfiles.monitoring import IOSampler
+
+sampler = IOSampler(buffer_sec=600, interval=0.25)
+sampler.start_sampling()
+
+...
+
+
+sampler.stop_sampling()
+sampler.plot_histogram()
+```
+
 ## Credits
 
 CloudFiles is derived from the [CloudVolume.Storage](https://github.com/seung-lab/cloud-volume/tree/master/cloudvolume/storage) system.  

{cloud_files-5.5.0 → cloud_files-5.6.1}/README.md

@@ -48,8 +48,20 @@ cf.touch([ "example", "example2" ])
 cf = CloudFile("gs://bucket/file1")
 info = cf.head()
 binary = cf.get()
+obj = cf.get_json()
 cf.put(binary)
+cf.put_json()
 cf[:30] # get first 30 bytes of file
+
+num_bytes = cf.size() # get size in bytes (also in head)
+exists = cf.exists() # true or false
+cf.delete() # deletes the file
+cf.touch() # create the file if it doesn't exist
+cf.move("gs://example/destination/directory") # copy then delete source
+cf.transfer_from("gs://example/source/file.txt") # copies file efficiently
+cf.transfer_to("gs://example/dest/file.txt") # copies file efficiently
+
+path = cf.join([ path1, path2, path3 ]) # use the appropriate path separator
 ```
 
 CloudFiles was developed to access files from object storage without ever touching disk. The goal was to reliably and rapidly access a petabyte of image data broken down into tens to hundreds of millions of files being accessed in parallel across thousands of cores. CloudFiles has been used to processes dozens of images, many of which were in the hundreds of terabyte range. It has reliably read and written tens of billions of files to date.
@@ -75,6 +87,7 @@ CloudFiles was developed to access files from object storage without ever touchi
 ```bash
 pip install cloud-files
 pip install cloud-files[test] # to enable testing with pytest
+pip install cloud-files[monitoring] # enable plotting network performance
 ```
 
 If you run into trouble installing dependenies, make sure you're using at least Python3.6 and you have updated pip. On Linux, some dependencies require manylinux2010 or manylinux2014 binaries which earlier versions of pip do not search for. MacOS, Linux, and Windows are supported platforms.
@@ -127,7 +140,9 @@ You can create the `google-secret.json` file [here](https://console.cloud.google
 
 ## API Documentation  
 
-Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations.  
+Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations. 
+
+`CloudFile` is a more intuitive version of `CloudFiles` designed for managing single files instead of groups of files. See examples above. There is an analogus method for each `CloudFiles` method (where it makes sense).
 
 ### Constructor
 ```python
@@ -187,6 +202,10 @@ binary = cf['filename'][0:5] # same result, fetches 11 bytes
 >> b'hello' # represents byte range 0-4 inclusive of filename
 
 binaries = cf[:100] # download the first 100 files
+
+# Get the TransmissionMonitor object that records
+# the flight time of each file.
+binaries, tm = cf.get(..., return_recording=True)
 ```
 
 `get` supports several different styles of input. The simplest takes a scalar filename and returns the contents of that file. However, you can also specify lists of filenames, a byte range request, and lists of byte range requests. You can provide a generator or iterator as input as well. Order is not guaranteed.
@@ -216,6 +235,10 @@ cf.puts([{
 cf.puts([ (path, content), (path, content) ], compression='gzip')
 cf.put_jsons(...)
 
+# Get the TransmissionMonitor object that records the
+# flight times of each object.
+_, tm = cf.puts(..., return_recording=True)
+
 # Definition of put, put_json is identical
 def put(
     self, 
@@ -420,6 +443,12 @@ cloudfiles -p 2 cp --progress -r s3://bkt/ gs://bkt2/
 cloudfiles cp -c br s3://bkt/file.txt gs://bkt2/
 # decompress
 cloudfiles cp -c none s3://bkt/file.txt gs://bkt2/
+# save chart of file flight times
+cloudfiles cp --flight-time s3://bkt/file.txt gs://bkt2/
+# save a chart of estimated bandwidth usage from these files alone
+cloudfiles cp --io-rate s3://bkt/file.txt gs://bkt2/
+# save a chart of measured bandwidth usage for the machine
+cloudfiles cp --machine-io-rate s3://bkt/file.txt gs://bkt2/
 # move or rename files
 cloudfiles mv s3://bkt/file.txt gs://bkt2/
 # create an empty file if not existing
@@ -479,6 +508,40 @@ cloudfiles alias rm example # remove example://
 
 The alias file is only accessed (and cached) if CloudFiles encounters an unknown protocol. If you stick to default protocols and use the syntax `s3://https://example.com/` for alternative endpoints, you can still use CloudFiles in environments without filesystem access.
 
+## Performance Monitoring
+
+CloudFiles now comes with two tools inside of `cloudfiles.monitoring` for measuring the performance of transfer operations both via the CLI and the programatic interface.
+
+A `TransmissionMonitor` object is created during each download or upload (e.g. `cf.get` or `cf.puts`) call. You can access this object by using the `return_recording=True` flag. This object saves the flight times of each object along with its size in an interval tree datastructure. It comes with methods for estimating the peak bits per a second and can plot both time of flight and the estimated transfer rates (assuming the transfer is evenly divided over the flight of an object, an assumption that is not always true). This allows you to estimate the contribution of a given CloudFiles operation to a machine's network IO.
+
+```python
+from cloudfiles import CloudFiles
+
+... 
+
+results, tm = cf.get([ ... some files ... ], return_recording=True)
+
+value = tm.peak_Mbps() # estimated peak transfer rate
+value = tm.total_Mbps() # estimated average transfer rate
+tm.plot_gantt() # time of flight chart
+tm.plot_histogram() # transfer rate chart
+```
+
+A second object, `IOSampler`, can sample the OS network counters using a background thread and provides a global view of the machine's network performance during the life of the transfer. It is enabled on the CLI for the `cp` command when the `--machine-io-rate` flag is enabled, but must be manually started programatically. This is to avoid accidentally starting unnecessary sampling threads. The samples are accumulated into a circular buffer, so make sure to set the buffer length long enough for your points of interest to be captured.
+
+```python
+from cloudfiles.monitoring import IOSampler
+
+sampler = IOSampler(buffer_sec=600, interval=0.25)
+sampler.start_sampling()
+
+...
+
+
+sampler.stop_sampling()
+sampler.plot_histogram()
+```
+
 ## Credits
 
 CloudFiles is derived from the [CloudVolume.Storage](https://github.com/seung-lab/cloud-volume/tree/master/cloudvolume/storage) system.  

{cloud_files-5.5.0 → cloud_files-5.6.1}/automated_test.py

@@ -4,6 +4,13 @@ import re
 import shutil
 import time
 
+
+import uuid
+from typing import Optional
+from unittest.mock import patch, MagicMock
+import numpy as np
+from cloudfiles.monitoring import TransmissionMonitor, IOSampler, IOEnum
+
 from moto import mock_aws
 
 COMPRESSION_TYPES = [ 
@@ -1259,3 +1266,117 @@ def test_touch(s3, protocol):
   cf.touch([ str(i) for i in range(20) ])
 
   assert sorted(list(cf)) == sorted([ str(i) for i in range(20) ])
+
+class TestTransmissionMonitor:
+  @pytest.fixture
+  def tx_monitor(self):
+    return TransmissionMonitor(IOEnum.TX)
+  
+  @pytest.fixture
+  def rx_monitor(self):
+    return TransmissionMonitor(IOEnum.RX)
+  
+  def test_init(self, tx_monitor):
+    assert tx_monitor._direction == IOEnum.TX
+    assert tx_monitor._total_bytes_landed == 0
+    assert len(tx_monitor._in_flight) == 0
+    assert len(tx_monitor._errors) == 0
+    assert tx_monitor._in_flight_bytes == 0
+    
+  def test_start_io(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    assert isinstance(flight_id, uuid.UUID)
+    assert len(tx_monitor._in_flight) == 1
+    assert tx_monitor._in_flight_bytes == 100
+    
+  def test_end_io(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    time.sleep(0.01) 
+    tx_monitor.end_io(flight_id, 100)
+    
+    assert len(tx_monitor._in_flight) == 0
+    assert tx_monitor._in_flight_bytes == 0
+    assert tx_monitor._total_bytes_landed == 100
+    assert len(tx_monitor._intervaltree) == 1
+    
+  def test_end_error(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    tx_monitor.end_error(flight_id)
+    
+    assert flight_id in tx_monitor._errors
+    assert len(tx_monitor._in_flight) == 1  # Still in flight
+    
+  def test_total_bytes(self, tx_monitor):
+    flight_id1 = tx_monitor.start_io(100)
+    flight_id2 = tx_monitor.start_io(200)
+    tx_monitor.end_io(flight_id1, 100)
+    tx_monitor.end_io(flight_id2, 200)
+    
+    assert tx_monitor.total_bytes() == 300
+    
+  def test_total_bps(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    time.sleep(1.0)
+    tx_monitor.end_io(flight_id, 100)
+    
+    bps = tx_monitor.total_bps()
+    assert pytest.approx(bps, rel=0.1) == 800  # 100 bytes * 8 bits/byte / 1 sec
+    
+  def test_current_bps(self, tx_monitor):
+    # Test with lookback window
+    flight_id = tx_monitor.start_io(100)
+    time.sleep(1.0)
+    tx_monitor.end_io(flight_id, 100)
+    
+    bps = tx_monitor.current_bps(look_back_sec=0.5)
+    assert bps > 0
+    
+  def test_peak_bps(self, tx_monitor):
+    flight_id1 = tx_monitor.start_io(100)
+    time.sleep(0.5)
+    tx_monitor.end_io(flight_id1, 100)
+    
+    flight_id2 = tx_monitor.start_io(200)
+    time.sleep(0.5)
+    tx_monitor.end_io(flight_id2, 200)
+    
+    peak = tx_monitor.peak_bps()
+    assert peak > 0
+    
+  def test_histogram(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    time.sleep(1.0)
+    tx_monitor.end_io(flight_id, 100)
+    
+    hist = tx_monitor.histogram(resolution=1.0)
+    assert len(hist) > 0
+    assert np.sum(hist) == 100
+    
+  def test_merge(self, tx_monitor):
+    monitor1 = TransmissionMonitor(IOEnum.TX)
+    monitor2 = TransmissionMonitor(IOEnum.TX)
+    
+    flight_id1 = monitor1.start_io(100)
+    time.sleep(0.1)
+    monitor1.end_io(flight_id1, 100)
+    
+    flight_id2 = monitor2.start_io(200)
+    time.sleep(0.1)
+    monitor2.end_io(flight_id2, 200)
+    
+    merged = TransmissionMonitor.merge([monitor1, monitor2])
+    
+    assert merged.total_bytes() == 300
+    assert len(merged._intervaltree) == 2
+  
+  def test_serialization(self, tx_monitor):
+    flight_id = tx_monitor.start_io(100)
+    time.sleep(0.1)
+    tx_monitor.end_io(flight_id, 100)
+    
+    import pickle
+    data = pickle.dumps(tx_monitor)
+    new_monitor = pickle.loads(data)
+    
+    assert new_monitor.total_bytes() == 100
+    assert hasattr(new_monitor, '_lock')

{cloud_files-5.5.0 → cloud_files-5.6.1}/cloud_files.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cloud-files
-Version: 5.5.0
+Version: 5.6.1
 Summary: Fast access to cloud storage and local FS.
 Home-page: https://github.com/seung-lab/cloud-files/
 Author: William Silversmith
@@ -30,6 +30,8 @@ Requires-Dist: google-auth>=1.10.0
 Requires-Dist: google-cloud-core>=1.1.0
 Requires-Dist: google-cloud-storage>=1.31.1
 Requires-Dist: google-crc32c>=1.0.0
+Requires-Dist: intervaltree
+Requires-Dist: numpy
 Requires-Dist: orjson
 Requires-Dist: pathos
 Requires-Dist: protobuf>=3.3.0
@@ -46,6 +48,12 @@ Requires-Dist: pytest; extra == "test"
 Requires-Dist: moto>=5; extra == "test"
 Provides-Extra: numpy
 Requires-Dist: numpy; extra == "numpy"
+Provides-Extra: monitoring
+Requires-Dist: psutil; extra == "monitoring"
+Requires-Dist: intervaltree; extra == "monitoring"
+Requires-Dist: matplotlib; extra == "monitoring"
+Provides-Extra: apache
+Requires-Dist: lxml; extra == "apache"
 
 [![PyPI version](https://badge.fury.io/py/cloud-files.svg)](https://badge.fury.io/py/cloud-files) [![Test Suite](https://github.com/seung-lab/cloud-files/workflows/Test%20Suite/badge.svg)](https://github.com/seung-lab/cloud-files/actions?query=workflow%3A%22Test+Suite%22)
 
@@ -97,8 +105,20 @@ cf.touch([ "example", "example2" ])
 cf = CloudFile("gs://bucket/file1")
 info = cf.head()
 binary = cf.get()
+obj = cf.get_json()
 cf.put(binary)
+cf.put_json()
 cf[:30] # get first 30 bytes of file
+
+num_bytes = cf.size() # get size in bytes (also in head)
+exists = cf.exists() # true or false
+cf.delete() # deletes the file
+cf.touch() # create the file if it doesn't exist
+cf.move("gs://example/destination/directory") # copy then delete source
+cf.transfer_from("gs://example/source/file.txt") # copies file efficiently
+cf.transfer_to("gs://example/dest/file.txt") # copies file efficiently
+
+path = cf.join([ path1, path2, path3 ]) # use the appropriate path separator
 ```
 
 CloudFiles was developed to access files from object storage without ever touching disk. The goal was to reliably and rapidly access a petabyte of image data broken down into tens to hundreds of millions of files being accessed in parallel across thousands of cores. CloudFiles has been used to processes dozens of images, many of which were in the hundreds of terabyte range. It has reliably read and written tens of billions of files to date.
@@ -124,6 +144,7 @@ CloudFiles was developed to access files from object storage without ever touchi
 ```bash
 pip install cloud-files
 pip install cloud-files[test] # to enable testing with pytest
+pip install cloud-files[monitoring] # enable plotting network performance
 ```
 
 If you run into trouble installing dependenies, make sure you're using at least Python3.6 and you have updated pip. On Linux, some dependencies require manylinux2010 or manylinux2014 binaries which earlier versions of pip do not search for. MacOS, Linux, and Windows are supported platforms.
@@ -176,7 +197,9 @@ You can create the `google-secret.json` file [here](https://console.cloud.google
 
 ## API Documentation  
 
-Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations.  
+Note that the "Cloud Costs" mentioned below are current as of June 2020 and are subject to change. As of this writing, S3 and Google use identical cost structures for these operations. 
+
+`CloudFile` is a more intuitive version of `CloudFiles` designed for managing single files instead of groups of files. See examples above. There is an analogus method for each `CloudFiles` method (where it makes sense).
 
 ### Constructor
 ```python
@@ -236,6 +259,10 @@ binary = cf['filename'][0:5] # same result, fetches 11 bytes
 >> b'hello' # represents byte range 0-4 inclusive of filename
 
 binaries = cf[:100] # download the first 100 files
+
+# Get the TransmissionMonitor object that records
+# the flight time of each file.
+binaries, tm = cf.get(..., return_recording=True)
 ```
 
 `get` supports several different styles of input. The simplest takes a scalar filename and returns the contents of that file. However, you can also specify lists of filenames, a byte range request, and lists of byte range requests. You can provide a generator or iterator as input as well. Order is not guaranteed.
@@ -265,6 +292,10 @@ cf.puts([{
 cf.puts([ (path, content), (path, content) ], compression='gzip')
 cf.put_jsons(...)
 
+# Get the TransmissionMonitor object that records the
+# flight times of each object.
+_, tm = cf.puts(..., return_recording=True)
+
 # Definition of put, put_json is identical
 def put(
     self, 
@@ -469,6 +500,12 @@ cloudfiles -p 2 cp --progress -r s3://bkt/ gs://bkt2/
 cloudfiles cp -c br s3://bkt/file.txt gs://bkt2/
 # decompress
 cloudfiles cp -c none s3://bkt/file.txt gs://bkt2/
+# save chart of file flight times
+cloudfiles cp --flight-time s3://bkt/file.txt gs://bkt2/
+# save a chart of estimated bandwidth usage from these files alone
+cloudfiles cp --io-rate s3://bkt/file.txt gs://bkt2/
+# save a chart of measured bandwidth usage for the machine
+cloudfiles cp --machine-io-rate s3://bkt/file.txt gs://bkt2/
 # move or rename files
 cloudfiles mv s3://bkt/file.txt gs://bkt2/
 # create an empty file if not existing
@@ -528,6 +565,40 @@ cloudfiles alias rm example # remove example://
 
 The alias file is only accessed (and cached) if CloudFiles encounters an unknown protocol. If you stick to default protocols and use the syntax `s3://https://example.com/` for alternative endpoints, you can still use CloudFiles in environments without filesystem access.
 
+## Performance Monitoring
+
+CloudFiles now comes with two tools inside of `cloudfiles.monitoring` for measuring the performance of transfer operations both via the CLI and the programatic interface.
+
+A `TransmissionMonitor` object is created during each download or upload (e.g. `cf.get` or `cf.puts`) call. You can access this object by using the `return_recording=True` flag. This object saves the flight times of each object along with its size in an interval tree datastructure. It comes with methods for estimating the peak bits per a second and can plot both time of flight and the estimated transfer rates (assuming the transfer is evenly divided over the flight of an object, an assumption that is not always true). This allows you to estimate the contribution of a given CloudFiles operation to a machine's network IO.
+
+```python
+from cloudfiles import CloudFiles
+
+... 
+
+results, tm = cf.get([ ... some files ... ], return_recording=True)
+
+value = tm.peak_Mbps() # estimated peak transfer rate
+value = tm.total_Mbps() # estimated average transfer rate
+tm.plot_gantt() # time of flight chart
+tm.plot_histogram() # transfer rate chart
+```
+
+A second object, `IOSampler`, can sample the OS network counters using a background thread and provides a global view of the machine's network performance during the life of the transfer. It is enabled on the CLI for the `cp` command when the `--machine-io-rate` flag is enabled, but must be manually started programatically. This is to avoid accidentally starting unnecessary sampling threads. The samples are accumulated into a circular buffer, so make sure to set the buffer length long enough for your points of interest to be captured.
+
+```python
+from cloudfiles.monitoring import IOSampler
+
+sampler = IOSampler(buffer_sec=600, interval=0.25)
+sampler.start_sampling()
+
+...
+
+
+sampler.stop_sampling()
+sampler.plot_histogram()
+```
+
 ## Credits
 
 CloudFiles is derived from the [CloudVolume.Storage](https://github.com/seung-lab/cloud-volume/tree/master/cloudvolume/storage) system.  

{cloud_files-5.5.0 → cloud_files-5.6.1}/cloud_files.egg-info/SOURCES.txt

@@ -24,6 +24,7 @@ cloudfiles/exceptions.py
 cloudfiles/gcs.py
 cloudfiles/interfaces.py
 cloudfiles/lib.py
+cloudfiles/monitoring.py
 cloudfiles/paths.py
 cloudfiles/resumable_tools.py
 cloudfiles/scheduler.py

cloud_files-5.6.1/cloud_files.egg-info/pbr.json

@@ -0,0 +1 @@
+{"git_version": "5554b92", "is_release": true}

{cloud_files-5.5.0 → cloud_files-5.6.1}/cloud_files.egg-info/requires.txt

@@ -9,6 +9,8 @@ google-auth>=1.10.0
 google-cloud-core>=1.1.0
 google-cloud-storage>=1.31.1
 google-crc32c>=1.0.0
+intervaltree
+numpy
 orjson
 pathos
 protobuf>=3.3.0
@@ -21,6 +23,14 @@ zstandard
 rsa>=4.7.2
 fasteners
 
+[apache]
+lxml
+
+[monitoring]
+psutil
+intervaltree
+matplotlib
+
 [numpy]
 numpy