hsclient 1.0.0__tar.gz → 1.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: hsclient
3
- Version: 1.0.0
3
+ Version: 1.0.2
4
4
  Summary: A python client for managing HydroShare resources
5
5
  Home-page: https://github.com/hydroshare/hsclient
6
6
  Author: Scott Black
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3.9
14
14
  Requires-Python: >=3.9
15
15
  Description-Content-Type: text/markdown
16
16
  License-File: LICENSE.txt
17
- Requires-Dist: hsmodels>=1.0.0
17
+ Requires-Dist: hsmodels>=1.0.3
18
18
  Requires-Dist: requests
19
19
  Requires-Dist: requests_oauthlib
20
20
  Provides-Extra: pandas
@@ -34,20 +34,24 @@ Requires-Dist: rasterio; extra == "all"
34
34
  Requires-Dist: fiona; extra == "all"
35
35
 
36
36
  # hsclient
37
- A python client for interacting with HydroShare in an object oriented way.
37
+ A python client for interacting with HydroShare in an object oriented way. The hsclient Python package can be use to create, modify, and interact with HydroShare resources. It was designed to allow you to write code to do pretty much everything you can do through HydroShare's web user interface.
38
38
 
39
39
  ## Jupyter Notebooks
40
- HydroShare has a resource with example notebooks. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown and select `Cuahsi Jupyterhub` to launch the notebooks into a Jupyter Environment to start using this project.
40
+ HydroShare has a resource with example Jupyter notebooks for using hsclient. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown at the top of the page and select `Cuahsi Jupyterhub` to launch the notebooks into HydroShare's linked Jupyter Environment. [Click here](https://help.hydroshare.org/apps/CUAHSI-JupyterHub/) for information on how to access the HydroShare linked JupyterHub environment.
41
41
 
42
42
 
43
- ## Install the HS RDF HydroShare Python Client
44
- The HS RDF Python Client for HydroShare won't be installed by default, so it has to be installed first before you can work with it. Use the following command to install the Python Client from the GitHub repository. Eventually we will distribute this package via the Python Package Index (PyPi) so that it can be installed via pip from PyPi.
43
+ ## Install the hsclient HydroShare Python Client
44
+ The hsclient package is installed already in HydroShare's linked JupyterHub Python environment. However, if you want to use hsclient outside of that Python environment, hsclient won't be installed by default. You'll need to install it before you can work with it. Use the following command to pip install hsclient from the Python Package Index (PyPI). You can also use the package manager for your interactive development environment to do this.
45
45
 
46
46
  ```bash
47
47
  pip install hsclient
48
48
  ```
49
49
 
50
- ## Authenticate with HydroShare
50
+ ## Getting Started and Documentation
51
+
52
+ The following are quick examples of how to get started with hsclient. Refer to the Jupyter notebooks above for more extensive examples. [Click here](https://hydroshare.github.io/hsclient/) to access documentation for hsclient.
53
+
54
+ ### Authenticate with HydroShare
51
55
  Before you start interacting with resources in HydroShare you will need to authenticate.
52
56
  ```python
53
57
  from hsclient import HydroShare
@@ -56,7 +60,7 @@ hs = HydroShare()
56
60
  hs.sign_in()
57
61
  ```
58
62
 
59
- ## Create a New Empty Resource
63
+ ### Create a New Empty Resource
60
64
  A "resource" is a container for your content in HydroShare. Think of it as a "working directory" into which you are going to organize the code and/or data you are using and want to share. The following code can be used to create a new, empty resource within which you can create content and metadata.
61
65
 
62
66
  This code creates a new resource in HydroShare. It also creates an in-memory object representation of that resource in your local environmment that you can then manipulate with further code.
@@ -72,12 +76,12 @@ print('The HydroShare Identifier for your new resource is: ' + resIdentifier)
72
76
  print('Your new resource is available at: ' + new_resource.metadata.url)
73
77
  ```
74
78
 
75
- # Creating and Editing Resource Metadata Elements
79
+ ### Creating and Editing Resource Metadata Elements
76
80
  Editing metadata elements for a resource can be done in an object oriented way. You can specify all of the metadata elements in code, which will set their values in memory in your local environment. Values of metadata elements can be edited, removed, or replaced by setting them to a new value, appending new values (in the case where the metadata element accepts a list), or by removing the value entirely.
77
81
 
78
82
  When you are ready to save edits to metadata elements from your local environment to the resource in HydroShare, you can call the save() function on your resource and all of the new metadata values you created/edited will be saved to the resource in HydroShare.
79
83
 
80
- ## Resource Title and Abstract
84
+ ### Resource Title and Abstract
81
85
  The Title and Abstract metadata elements can be specified as text strings. To modify the Title or Abstract after it has been set, just set them to a different value.
82
86
 
83
87
  ```python
@@ -99,3 +103,8 @@ new_resource.save()
99
103
  print('Title: ' + new_resource.metadata.title)
100
104
  print('Abstract: ' + new_resource.metadata.abstract)
101
105
  ```
106
+
107
+ ## Funding and Acknowledgements
108
+
109
+ This material is based upon work supported by the National Science Foundation (NSF) under awards [1931278](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931278) and [1931297](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931297). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
110
+
@@ -1,18 +1,22 @@
1
1
  # hsclient
2
- A python client for interacting with HydroShare in an object oriented way.
2
+ A python client for interacting with HydroShare in an object oriented way. The hsclient Python package can be use to create, modify, and interact with HydroShare resources. It was designed to allow you to write code to do pretty much everything you can do through HydroShare's web user interface.
3
3
 
4
4
  ## Jupyter Notebooks
5
- HydroShare has a resource with example notebooks. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown and select `Cuahsi Jupyterhub` to launch the notebooks into a Jupyter Environment to start using this project.
5
+ HydroShare has a resource with example Jupyter notebooks for using hsclient. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown at the top of the page and select `Cuahsi Jupyterhub` to launch the notebooks into HydroShare's linked Jupyter Environment. [Click here](https://help.hydroshare.org/apps/CUAHSI-JupyterHub/) for information on how to access the HydroShare linked JupyterHub environment.
6
6
 
7
7
 
8
- ## Install the HS RDF HydroShare Python Client
9
- The HS RDF Python Client for HydroShare won't be installed by default, so it has to be installed first before you can work with it. Use the following command to install the Python Client from the GitHub repository. Eventually we will distribute this package via the Python Package Index (PyPi) so that it can be installed via pip from PyPi.
8
+ ## Install the hsclient HydroShare Python Client
9
+ The hsclient package is installed already in HydroShare's linked JupyterHub Python environment. However, if you want to use hsclient outside of that Python environment, hsclient won't be installed by default. You'll need to install it before you can work with it. Use the following command to pip install hsclient from the Python Package Index (PyPI). You can also use the package manager for your interactive development environment to do this.
10
10
 
11
11
  ```bash
12
12
  pip install hsclient
13
13
  ```
14
14
 
15
- ## Authenticate with HydroShare
15
+ ## Getting Started and Documentation
16
+
17
+ The following are quick examples of how to get started with hsclient. Refer to the Jupyter notebooks above for more extensive examples. [Click here](https://hydroshare.github.io/hsclient/) to access documentation for hsclient.
18
+
19
+ ### Authenticate with HydroShare
16
20
  Before you start interacting with resources in HydroShare you will need to authenticate.
17
21
  ```python
18
22
  from hsclient import HydroShare
@@ -21,7 +25,7 @@ hs = HydroShare()
21
25
  hs.sign_in()
22
26
  ```
23
27
 
24
- ## Create a New Empty Resource
28
+ ### Create a New Empty Resource
25
29
  A "resource" is a container for your content in HydroShare. Think of it as a "working directory" into which you are going to organize the code and/or data you are using and want to share. The following code can be used to create a new, empty resource within which you can create content and metadata.
26
30
 
27
31
  This code creates a new resource in HydroShare. It also creates an in-memory object representation of that resource in your local environmment that you can then manipulate with further code.
@@ -37,12 +41,12 @@ print('The HydroShare Identifier for your new resource is: ' + resIdentifier)
37
41
  print('Your new resource is available at: ' + new_resource.metadata.url)
38
42
  ```
39
43
 
40
- # Creating and Editing Resource Metadata Elements
44
+ ### Creating and Editing Resource Metadata Elements
41
45
  Editing metadata elements for a resource can be done in an object oriented way. You can specify all of the metadata elements in code, which will set their values in memory in your local environment. Values of metadata elements can be edited, removed, or replaced by setting them to a new value, appending new values (in the case where the metadata element accepts a list), or by removing the value entirely.
42
46
 
43
47
  When you are ready to save edits to metadata elements from your local environment to the resource in HydroShare, you can call the save() function on your resource and all of the new metadata values you created/edited will be saved to the resource in HydroShare.
44
48
 
45
- ## Resource Title and Abstract
49
+ ### Resource Title and Abstract
46
50
  The Title and Abstract metadata elements can be specified as text strings. To modify the Title or Abstract after it has been set, just set them to a different value.
47
51
 
48
52
  ```python
@@ -64,3 +68,8 @@ new_resource.save()
64
68
  print('Title: ' + new_resource.metadata.title)
65
69
  print('Abstract: ' + new_resource.metadata.abstract)
66
70
  ```
71
+
72
+ ## Funding and Acknowledgements
73
+
74
+ This material is based upon work supported by the National Science Foundation (NSF) under awards [1931278](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931278) and [1931297](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931297). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
75
+
@@ -7,5 +7,6 @@ from hsclient.hydroshare import (
7
7
  TimeseriesAggregation,
8
8
  GeoRasterAggregation,
9
9
  GeoFeatureAggregation,
10
+ CSVAggregation
10
11
  )
11
12
  from hsclient.oauth2_model import Token
@@ -199,6 +199,7 @@ class Aggregation:
199
199
  AggregationType.TimeSeriesAggregation: TimeseriesAggregation,
200
200
  AggregationType.GeographicRasterAggregation: GeoRasterAggregation,
201
201
  AggregationType.GeographicFeatureAggregation: GeoFeatureAggregation,
202
+ AggregationType.CSVFileAggregation: CSVAggregation
202
203
  }
203
204
  for aggr in aggregations_copy:
204
205
  typed_aggr_cls = typed_aggregation_classes.get(aggr.metadata.type, None)
@@ -427,14 +428,14 @@ class DataObjectSupportingAggregation(Aggregation):
427
428
  def _validate_aggregation_path(self, agg_path: str, for_save_data: bool = False) -> str:
428
429
  return self._get_file_path(agg_path)
429
430
 
430
- def _get_data_object(self, agg_path: str, func: Callable) -> \
431
+ def _get_data_object(self, agg_path: str, func: Callable, **func_kwargs) -> \
431
432
  Union['pandas.DataFrame', 'fiona.Collection', 'rasterio.DatasetReader', 'xarray.Dataset']:
432
433
 
433
434
  if self._data_object is not None and self.metadata.type != AggregationType.TimeSeriesAggregation:
434
435
  return self._data_object
435
436
 
436
437
  file_path = self._validate_aggregation_path(agg_path)
437
- data_object = func(file_path)
438
+ data_object = func(file_path, **func_kwargs)
438
439
  if self.metadata.type == AggregationType.MultidimensionalAggregation:
439
440
  data_object.load()
440
441
  data_object.close()
@@ -859,6 +860,72 @@ class GeoRasterAggregation(DataObjectSupportingAggregation):
859
860
  return aggr
860
861
 
861
862
 
863
+ class CSVAggregation(DataObjectSupportingAggregation):
864
+ """Represents a CSV Aggregation in HydroShare"""
865
+ @classmethod
866
+ def create(cls, base_aggr):
867
+ return super().create(aggr_cls=cls, base_aggr=base_aggr)
868
+
869
+ def as_data_object(self, agg_path: str) -> 'pandas.DataFrame':
870
+ """
871
+ Loads the CSV aggregation to a pandas DataFrame object
872
+ :param agg_path: the path to the Time Series aggregation
873
+ :return: the CSV aggregation as a pandas DataFrame object
874
+ """
875
+ if pandas is None:
876
+ raise Exception("pandas package not found")
877
+
878
+ return self._get_data_object(agg_path=agg_path, func=pandas.read_csv, comment="#", dtype="string",
879
+ engine="python")
880
+
881
+ def save_data_object(self, resource: 'Resource', agg_path: str, as_new_aggr: bool = False,
882
+ destination_path: str = "") -> 'Aggregation':
883
+
884
+ """
885
+ Saves the pandas DataFrame object to the CSV aggregation
886
+ :param resource: the resource containing the aggregation
887
+ :param agg_path: the path to the CSV aggregation
888
+ :param as_new_aggr: Defaults False, set to True to create a new CSV aggregation
889
+ :param destination_path: the destination path in Hydroshare to save the new aggregation
890
+ :return: the updated or new CSV aggregation
891
+ """
892
+ self._validate_aggregation_for_update(resource, AggregationType.CSVFileAggregation)
893
+ file_path = self._validate_aggregation_path(agg_path, for_save_data=True)
894
+ self._data_object.to_csv(file_path, index=False)
895
+ aggr_main_file_path = self.main_file_path
896
+ data_object = self._data_object
897
+ if not as_new_aggr:
898
+ # cache some of the metadata fields of the original aggregation to update the metadata of the
899
+ # updated aggregation
900
+ keywords = self.metadata.subjects
901
+ additional_meta = self.metadata.additional_metadata
902
+ title = self.metadata.title
903
+
904
+ # upload the updated aggregation files to the temp folder - to create the updated aggregation
905
+ self._update_aggregation(resource, file_path)
906
+ # retrieve the updated aggregation
907
+ aggr = resource.aggregation(file__path=aggr_main_file_path)
908
+
909
+ # update metadata
910
+ for kw in keywords:
911
+ if kw not in aggr.metadata.subjects:
912
+ aggr.metadata.subjects.append(kw)
913
+ aggr.metadata.additional_metadata = additional_meta
914
+ aggr.metadata.title = title
915
+ aggr.save()
916
+ else:
917
+ # creating a new aggregation by uploading the updated data files
918
+ resource.file_upload(file_path, destination_path=destination_path)
919
+
920
+ # retrieve the new aggregation
921
+ agg_path = urljoin(destination_path, os.path.basename(aggr_main_file_path))
922
+ aggr = resource.aggregation(file__path=agg_path)
923
+ data_object = None
924
+
925
+ aggr._data_object = data_object
926
+ return aggr
927
+
928
+
862
929
  class Resource(Aggregation):
863
930
  """Represents a Resource in HydroShare"""
864
931
 
@@ -1128,7 +1195,10 @@ class Resource(Aggregation):
1128
1195
  if agg_type == AggregationType.SingleFileAggregation:
1129
1196
  type_value = 'SingleFile'
1130
1197
  if agg_type == AggregationType.FileSetAggregation:
1131
- relative_path = dirname(path)
1198
+ if '/' in path:
1199
+ relative_path = dirname(path)
1200
+ else:
1201
+ relative_path = path
1132
1202
  data = {"folder_path": relative_path}
1133
1203
 
1134
1204
  url = urljoin(self._hsapi_path, "functions", "set-file-type", path, type_value)
@@ -1,5 +1,5 @@
1
1
  from datetime import datetime
2
- from typing import Dict, List, Tuple
2
+ from typing import Dict, List, Tuple, Optional
3
3
 
4
4
  from hsmodels.schemas.enums import UserIdentifierType
5
5
  from pydantic import AnyUrl, BaseModel, HttpUrl, field_validator
@@ -12,7 +12,7 @@ class User(BaseModel):
12
12
  phone: str = None
13
13
  address: str = None
14
14
  organization: str = None
15
- website: HttpUrl = None
15
+ website: Optional[HttpUrl] = None
16
16
  identifiers: Dict[UserIdentifierType, AnyUrl] = {}
17
17
  type: str = None
18
18
  subject_areas: List[str] = []
@@ -20,7 +20,17 @@ class User(BaseModel):
20
20
 
21
21
  @field_validator("subject_areas", mode='before')
22
22
  def split_subject_areas(cls, value):
23
- return value.split(", ") if value else []
23
+ if isinstance(value, str):
24
+ return value.split(", ")
25
+ if value is None:
26
+ return []
27
+ return value
28
+
29
+ @field_validator("website", mode='before')
30
+ def handle_empty_website(cls, v):
31
+ if v == "":
32
+ v = None
33
+ return v
24
34
 
25
35
 
26
36
  class ResourcePreview(BaseModel):
@@ -1,8 +1,13 @@
1
+ from collections import namedtuple
1
2
  from os.path import splitext
2
3
  from urllib.request import pathname2url
3
4
 
4
5
  from hsmodels.schemas.enums import AggregationType
5
6
 
7
+ CSVColumnDataType = namedtuple('CSVColumnDataType', ['string', 'number', 'datetime', 'boolean'])(
8
+ 'string', 'number', 'datetime', 'boolean'
9
+ )
10
+
6
11
 
7
12
  def is_aggregation(path):
8
13
  return path.endswith('#aggregation')
@@ -19,6 +24,8 @@ def main_file_type(type: AggregationType):
19
24
  return ".refts.json"
20
25
  if type == AggregationType.TimeSeriesAggregation:
21
26
  return ".sqlite"
27
+ if type == AggregationType.CSVFileAggregation:
28
+ return ".csv"
22
29
  return None
23
30
 
24
31
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: hsclient
3
- Version: 1.0.0
3
+ Version: 1.0.2
4
4
  Summary: A python client for managing HydroShare resources
5
5
  Home-page: https://github.com/hydroshare/hsclient
6
6
  Author: Scott Black
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3.9
14
14
  Requires-Python: >=3.9
15
15
  Description-Content-Type: text/markdown
16
16
  License-File: LICENSE.txt
17
- Requires-Dist: hsmodels>=1.0.0
17
+ Requires-Dist: hsmodels>=1.0.3
18
18
  Requires-Dist: requests
19
19
  Requires-Dist: requests_oauthlib
20
20
  Provides-Extra: pandas
@@ -34,20 +34,24 @@ Requires-Dist: rasterio; extra == "all"
34
34
  Requires-Dist: fiona; extra == "all"
35
35
 
36
36
  # hsclient
37
- A python client for interacting with HydroShare in an object oriented way.
37
+ A python client for interacting with HydroShare in an object oriented way. The hsclient Python package can be use to create, modify, and interact with HydroShare resources. It was designed to allow you to write code to do pretty much everything you can do through HydroShare's web user interface.
38
38
 
39
39
  ## Jupyter Notebooks
40
- HydroShare has a resource with example notebooks. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown and select `Cuahsi Jupyterhub` to launch the notebooks into a Jupyter Environment to start using this project.
40
+ HydroShare has a resource with example Jupyter notebooks for using hsclient. Click [here](https://www.hydroshare.org/resource/7561aa12fd824ebb8edbee05af19b910/) then click the blue `Open with...` dropdown at the top of the page and select `Cuahsi Jupyterhub` to launch the notebooks into HydroShare's linked Jupyter Environment. [Click here](https://help.hydroshare.org/apps/CUAHSI-JupyterHub/) for information on how to access the HydroShare linked JupyterHub environment.
41
41
 
42
42
 
43
- ## Install the HS RDF HydroShare Python Client
44
- The HS RDF Python Client for HydroShare won't be installed by default, so it has to be installed first before you can work with it. Use the following command to install the Python Client from the GitHub repository. Eventually we will distribute this package via the Python Package Index (PyPi) so that it can be installed via pip from PyPi.
43
+ ## Install the hsclient HydroShare Python Client
44
+ The hsclient package is installed already in HydroShare's linked JupyterHub Python environment. However, if you want to use hsclient outside of that Python environment, hsclient won't be installed by default. You'll need to install it before you can work with it. Use the following command to pip install hsclient from the Python Package Index (PyPI). You can also use the package manager for your interactive development environment to do this.
45
45
 
46
46
  ```bash
47
47
  pip install hsclient
48
48
  ```
49
49
 
50
- ## Authenticate with HydroShare
50
+ ## Getting Started and Documentation
51
+
52
+ The following are quick examples of how to get started with hsclient. Refer to the Jupyter notebooks above for more extensive examples. [Click here](https://hydroshare.github.io/hsclient/) to access documentation for hsclient.
53
+
54
+ ### Authenticate with HydroShare
51
55
  Before you start interacting with resources in HydroShare you will need to authenticate.
52
56
  ```python
53
57
  from hsclient import HydroShare
@@ -56,7 +60,7 @@ hs = HydroShare()
56
60
  hs.sign_in()
57
61
  ```
58
62
 
59
- ## Create a New Empty Resource
63
+ ### Create a New Empty Resource
60
64
  A "resource" is a container for your content in HydroShare. Think of it as a "working directory" into which you are going to organize the code and/or data you are using and want to share. The following code can be used to create a new, empty resource within which you can create content and metadata.
61
65
 
62
66
  This code creates a new resource in HydroShare. It also creates an in-memory object representation of that resource in your local environmment that you can then manipulate with further code.
@@ -72,12 +76,12 @@ print('The HydroShare Identifier for your new resource is: ' + resIdentifier)
72
76
  print('Your new resource is available at: ' + new_resource.metadata.url)
73
77
  ```
74
78
 
75
- # Creating and Editing Resource Metadata Elements
79
+ ### Creating and Editing Resource Metadata Elements
76
80
  Editing metadata elements for a resource can be done in an object oriented way. You can specify all of the metadata elements in code, which will set their values in memory in your local environment. Values of metadata elements can be edited, removed, or replaced by setting them to a new value, appending new values (in the case where the metadata element accepts a list), or by removing the value entirely.
77
81
 
78
82
  When you are ready to save edits to metadata elements from your local environment to the resource in HydroShare, you can call the save() function on your resource and all of the new metadata values you created/edited will be saved to the resource in HydroShare.
79
83
 
80
- ## Resource Title and Abstract
84
+ ### Resource Title and Abstract
81
85
  The Title and Abstract metadata elements can be specified as text strings. To modify the Title or Abstract after it has been set, just set them to a different value.
82
86
 
83
87
  ```python
@@ -99,3 +103,8 @@ new_resource.save()
99
103
  print('Title: ' + new_resource.metadata.title)
100
104
  print('Abstract: ' + new_resource.metadata.abstract)
101
105
  ```
106
+
107
+ ## Funding and Acknowledgements
108
+
109
+ This material is based upon work supported by the National Science Foundation (NSF) under awards [1931278](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931278) and [1931297](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931297). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
110
+
@@ -1,4 +1,4 @@
1
- hsmodels>=1.0.0
1
+ hsmodels>=1.0.3
2
2
  requests
3
3
  requests_oauthlib
4
4
 
@@ -5,11 +5,11 @@ README = (pathlib.Path(__file__).parent / "README.md").read_text()
5
5
 
6
6
  setup(
7
7
  name='hsclient',
8
- version='1.0.0',
8
+ version='1.0.2',
9
9
  packages=find_packages(include=['hsclient', 'hsclient.*'],
10
10
  exclude=("tests",)),
11
11
  install_requires=[
12
- 'hsmodels>=1.0.0',
12
+ 'hsmodels>=1.0.3',
13
13
  'requests',
14
14
  'requests_oauthlib',
15
15
  ],
@@ -9,7 +9,13 @@ from fiona.model import to_dict
9
9
  from hsmodels.schemas.enums import AggregationType
10
10
  from rasterio.windows import Window
11
11
 
12
- from hsclient import GeoFeatureAggregation, GeoRasterAggregation, NetCDFAggregation, TimeseriesAggregation
12
+ from hsclient import (
13
+ GeoFeatureAggregation,
14
+ GeoRasterAggregation,
15
+ NetCDFAggregation,
16
+ TimeseriesAggregation,
17
+ CSVAggregation,
18
+ )
13
19
 
14
20
 
15
21
  @pytest.mark.parametrize("search_by", ["type", "file_path"])
@@ -318,3 +324,75 @@ def test_geofeature_save_data_object(resource_with_geofeature_aggr, as_new_aggr)
318
324
  assert ft_dict['properties']['Id'] < 5
319
325
  # need to close the data object so that the tmp directory can be cleaned up
320
326
  fn_collection.close()
327
+
328
+
329
+ @pytest.mark.parametrize("search_by", ["type", "file_path"])
330
+ def test_csv_as_data_object(resource_with_csv_aggr, search_by):
331
+ resource_with_csv_aggr.refresh()
332
+ if search_by == "type":
333
+ aggr = resource_with_csv_aggr.aggregation(type=AggregationType.CSVFileAggregation)
334
+ else:
335
+ file_path = "ecoregions.csv"
336
+ aggr = resource_with_csv_aggr.aggregation(file__path=file_path)
337
+ assert aggr is not None
338
+ assert type(aggr) is CSVAggregation
339
+ with tempfile.TemporaryDirectory() as tmp:
340
+ # download
341
+ unzip_to = os.path.join(tmp, "unzipped_aggr")
342
+ os.makedirs(unzip_to)
343
+ agg_path = resource_with_csv_aggr.aggregation_download(aggregation=aggr, save_path=tmp, unzip_to=unzip_to)
344
+ # load csv data to data object as pandas dataframe
345
+ pd_df = aggr.as_data_object(agg_path=agg_path)
346
+ # test number of rows in pandas dataframe
347
+ assert len(pd_df) == 2923
348
+ # test number of columns
349
+ assert len(pd_df.columns) == 4
350
+
351
+
352
+ @pytest.mark.parametrize("as_new_aggr", [False, True])
353
+ def test_csv_save_data_object(resource_with_csv_aggr, as_new_aggr):
354
+ resource_with_csv_aggr.refresh()
355
+ file_path = "ecoregions.csv"
356
+ aggr = resource_with_csv_aggr.aggregation(file__path=file_path)
357
+ assert aggr is not None
358
+ assert type(aggr) is CSVAggregation
359
+ with tempfile.TemporaryDirectory() as tmp:
360
+ # download
361
+ unzip_to = os.path.join(tmp, "unzipped_aggr")
362
+ os.makedirs(unzip_to)
363
+ agg_path = resource_with_csv_aggr.aggregation_download(aggregation=aggr, save_path=tmp, unzip_to=unzip_to)
364
+ # load csv data to data object as pandas dataframe
365
+ pd_df = aggr.as_data_object(agg_path=agg_path)
366
+ # test number of rows in pandas dataframe
367
+ assert len(pd_df) == 2923
368
+ # test number of columns
369
+ assert len(pd_df.columns) == 4
370
+ # edit the dataframe - remove the last column in the dataframe
371
+ pd_df.drop(pd_df.columns[-1], axis=1, inplace=True)
372
+ assert len(pd_df.columns) == 3
373
+ dst_path = ""
374
+ if as_new_aggr:
375
+ dst_path = "csv_aggr_folder"
376
+ resource_with_csv_aggr.folder_create(dst_path)
377
+ # save the dataframe to a new csv file
378
+ aggr = aggr.save_data_object(resource=resource_with_csv_aggr, agg_path=agg_path, as_new_aggr=as_new_aggr,
379
+ destination_path=dst_path)
380
+ assert aggr is not None
381
+ assert type(aggr) is CSVAggregation
382
+ if as_new_aggr:
383
+ assert aggr.data_object is None
384
+ else:
385
+ assert aggr.data_object is not None
386
+
387
+ # check the updated csv aggregation
388
+ with tempfile.TemporaryDirectory() as tmp:
389
+ # download
390
+ unzip_to = os.path.join(tmp, "unzipped_aggr")
391
+ os.makedirs(unzip_to)
392
+ agg_path = resource_with_csv_aggr.aggregation_download(aggregation=aggr, save_path=tmp, unzip_to=unzip_to)
393
+ # load csv data to data object as pandas dataframe
394
+ pd_df = aggr.as_data_object(agg_path=agg_path)
395
+ # test number of rows in pandas dataframe
396
+ assert len(pd_df) == 2923
397
+ # test number of columns
398
+ assert len(pd_df.columns) == 3
File without changes
File without changes