spio 0.0.0__tar.gz

spio-0.0.0/LICENSE.md

@@ -0,0 +1,19 @@
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spio-0.0.0/PKG-INFO

@@ -0,0 +1,286 @@
+Metadata-Version: 2.1
+Name: spio
+Version: 0.0.0
+Summary: Magical Marvin SharePoint Library
+Author: Roeland Maes
+Author-email: roeland.maes@vito.be
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: full
+Provides-Extra: pandas
+Provides-Extra: pandas-parquet
+Provides-Extra: pandas-xlsx
+Requires-Dist: Office365-REST-Python-Client (>=2.3,<3.0)
+Requires-Dist: msal_extensions (>=0.3,<0.4)
+Description-Content-Type: text/markdown
+
+
+# Spio
+
+## Description
+
+Spio is a Python package that provides functionalities for reading and writing data to a SharePoint.
+It can also handle various types of spatial and non-spatial data formats such as GeoTIFF, GeoPackage, NetCDF, CSV, Excel, and more.
+
+## Installation
+
+To install the package, you can use the following command:
+
+```bash
+    pip install spio 
+```
+
+If you want to install the package with all the dependencies, you can use the following command:
+
+```bash
+    pip install spio[full]
+```
+
+## Building from Source
+
+To build the package from source you can use the following commands:
+
+```bash
+    git clone https://git.vito.be/projects/MARVIN/repos/sharepoint_tools
+    cd sharepoint_tools
+    conda create -f conda_env.yml
+    conda activate spio 
+    poetry install
+```
+
+## Configuration
+
+Before using the package, you need to configure the settings to match your environment. 
+The configuration is managed through a `settings` module which should include all the necessary configurations.
+
+### **Settings Configuration**:
+   Ensure you have a `settings` file (e.g., `.secrets.yaml` or `settings.yaml`) that provides the necessary configuration options. Example:
+
+```python
+    from dynaconf import Dynaconf
+
+    settings = Dynaconf(
+        settings_files=['settings.yaml', '.secrets.yaml']
+    )
+```
+   
+   The settings file should include the following configurations:
+    
+```yaml
+    user_account: your.name@vito.be
+```
+
+### **Initialize Spio with Settings**:
+   You need to initialize Spio with your settings before using any functionalities.
+
+```python
+    from config import settings
+    from marvin.tools import spio
+
+    spio.init_spio(settings)
+```
+
+## Usage
+
+Here are some example usages of the package:
+
+### Reading Data
+
+- **Read GeoTIFF using Rasterio**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import rasterio
+    
+    geotiff_file = SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geotiff.tiff')
+    with rasterio.open(geotiff_file.copy_bytes_io()) as src:
+        # Read the dataset's metadata
+        print("Metadata:", src.meta)
+        print("CRS:", src.crs)
+        print("Bounds:", src.bounds)
+        data = src.read()
+        print("data: ", data.shape)
+```
+
+- **Read GeoTIFF using GDAL**:
+
+```python
+    from marvin.tools.spio import SPIO
+    from osgeo import gdal
+    
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geotiff.tiff') as spio:
+        dataset = gdal.Open(spio.copy_file())
+        band = dataset.GetRasterBand(1)
+        data = band.ReadAsArray()
+        print("data: ", data.shape)
+        # Close the dataset
+        del dataset
+```
+
+- **Read GeoPackage**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg'))
+    print("gdf: ", gdf)
+```
+
+- **Read NetCDF (h5netcdf)**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+
+    ds = xr.load_dataset(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/netcdf.nc'))  # use engine='h5netcdf' if not detected correctly
+    print("ds: ", ds)
+```
+
+- **Read NetCDF (netcdf4)**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/netcdf.nc') as spio:
+        ds = xr.load_dataset(spio.copy_file(), engine='netcdf4')
+        print("ds: ", ds)
+```
+
+- **Read GRIB**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+     
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/grib.grb') as spio:
+        grb_ds = xr.load_dataset(spio.copy_file())  # engine='cfgrib'
+        print("grb_ds: ", grb_ds)
+```
+
+- **Read Parquet**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+     
+    df = pd.read_parquet(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.parquet'))
+    print("df: ", df)
+```
+
+- **Read CSV**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+  
+    df = pd.read_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv'))
+    print(f"df: , df")
+
+    # read first lines
+    df_top = pd.read_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv', read_chunks=SPIO.DEFAULT_CHUNK_SIZE), sep=';', nrows=10)
+    print(f"df_top: , df_top")
+```
+
+- **Read Excel**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+  
+    df = pd.read_excel(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.xlsx'))
+    print("df: ", df)
+```
+
+
+### Writing Data
+
+- **Write CSV**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+
+    df = pd.read_csv('path/to/your/data.csv')
+    df.to_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv'))
+```
+
+- **Write Excel**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+
+    df = pd.read_csv('path/to/your/data.csv')
+    df.to_excel(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.xlsx'), engine='xlsxwriter')
+```
+
+- **Write Text**:
+
+```python
+    from marvin.tools.spio import SPIO
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.txt') as spio:
+        spio.write_lines(['Hello, SPIO!'])
+```
+
+
+- **Write GeoTiff**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import rasterio
+    
+    with rasterio.open('path/to/your/map.tiff') as src:
+        data = src.read()[0]
+        profile = src.profile
+       
+        with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/map.tiff') as spio:  # rasterio will not flush or close the SPIO file, so we need to do it ourselves
+            with rasterio.open(spio, 'w', **profile) as dst:
+                dst.write(data, 1)
+```
+
+
+- **Write GeoPackage Fiona**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file('path/to/your/geopackage.gpkg')
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg') as spio:  # fiona will not flush or close the IO-object, so we need to do it ourselves
+        gdf.to_file(spio, layer='mylayer', driver='GPKG', engine='fiona')
+```
+
+- **Write GeoPackage Fiona**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file('path/to/your/geopackage.gpkg')
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg') as spio:  # pyogrio will not flush the IO-object
+        gdf.to_file(spio.io_delegate(), layer='mylayer', driver='GPKG', engine='pyogrio')  # pyogrio can only write to a real BytesIO object.
+```
+
+### Additional Functionalities
+
+Spio also provides additional functions to handle different data formats and processes.
+You can explore these in the https://git.vito.be/projects/MARVIN/repos/sharepoint_tools/browse/test/tst_spio.py.
+
+
+## Contributing
+
+If you want to contribute to spio, please follow the standard contributing guidelines and push your changes to a new branch in
+https://git.vito.be/projects/MARVIN/repos/sharepoint_tools
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE.md file for details.
+

spio-0.0.0/README.md

@@ -0,0 +1,266 @@
+
+# Spio
+
+## Description
+
+Spio is a Python package that provides functionalities for reading and writing data to a SharePoint.
+It can also handle various types of spatial and non-spatial data formats such as GeoTIFF, GeoPackage, NetCDF, CSV, Excel, and more.
+
+## Installation
+
+To install the package, you can use the following command:
+
+```bash
+    pip install spio 
+```
+
+If you want to install the package with all the dependencies, you can use the following command:
+
+```bash
+    pip install spio[full]
+```
+
+## Building from Source
+
+To build the package from source you can use the following commands:
+
+```bash
+    git clone https://git.vito.be/projects/MARVIN/repos/sharepoint_tools
+    cd sharepoint_tools
+    conda create -f conda_env.yml
+    conda activate spio 
+    poetry install
+```
+
+## Configuration
+
+Before using the package, you need to configure the settings to match your environment. 
+The configuration is managed through a `settings` module which should include all the necessary configurations.
+
+### **Settings Configuration**:
+   Ensure you have a `settings` file (e.g., `.secrets.yaml` or `settings.yaml`) that provides the necessary configuration options. Example:
+
+```python
+    from dynaconf import Dynaconf
+
+    settings = Dynaconf(
+        settings_files=['settings.yaml', '.secrets.yaml']
+    )
+```
+   
+   The settings file should include the following configurations:
+    
+```yaml
+    user_account: your.name@vito.be
+```
+
+### **Initialize Spio with Settings**:
+   You need to initialize Spio with your settings before using any functionalities.
+
+```python
+    from config import settings
+    from marvin.tools import spio
+
+    spio.init_spio(settings)
+```
+
+## Usage
+
+Here are some example usages of the package:
+
+### Reading Data
+
+- **Read GeoTIFF using Rasterio**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import rasterio
+    
+    geotiff_file = SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geotiff.tiff')
+    with rasterio.open(geotiff_file.copy_bytes_io()) as src:
+        # Read the dataset's metadata
+        print("Metadata:", src.meta)
+        print("CRS:", src.crs)
+        print("Bounds:", src.bounds)
+        data = src.read()
+        print("data: ", data.shape)
+```
+
+- **Read GeoTIFF using GDAL**:
+
+```python
+    from marvin.tools.spio import SPIO
+    from osgeo import gdal
+    
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geotiff.tiff') as spio:
+        dataset = gdal.Open(spio.copy_file())
+        band = dataset.GetRasterBand(1)
+        data = band.ReadAsArray()
+        print("data: ", data.shape)
+        # Close the dataset
+        del dataset
+```
+
+- **Read GeoPackage**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg'))
+    print("gdf: ", gdf)
+```
+
+- **Read NetCDF (h5netcdf)**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+
+    ds = xr.load_dataset(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/netcdf.nc'))  # use engine='h5netcdf' if not detected correctly
+    print("ds: ", ds)
+```
+
+- **Read NetCDF (netcdf4)**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/netcdf.nc') as spio:
+        ds = xr.load_dataset(spio.copy_file(), engine='netcdf4')
+        print("ds: ", ds)
+```
+
+- **Read GRIB**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import xarray as xr
+     
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/grib.grb') as spio:
+        grb_ds = xr.load_dataset(spio.copy_file())  # engine='cfgrib'
+        print("grb_ds: ", grb_ds)
+```
+
+- **Read Parquet**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+     
+    df = pd.read_parquet(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.parquet'))
+    print("df: ", df)
+```
+
+- **Read CSV**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+  
+    df = pd.read_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv'))
+    print(f"df: , df")
+
+    # read first lines
+    df_top = pd.read_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv', read_chunks=SPIO.DEFAULT_CHUNK_SIZE), sep=';', nrows=10)
+    print(f"df_top: , df_top")
+```
+
+- **Read Excel**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+  
+    df = pd.read_excel(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.xlsx'))
+    print("df: ", df)
+```
+
+
+### Writing Data
+
+- **Write CSV**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+
+    df = pd.read_csv('path/to/your/data.csv')
+    df.to_csv(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.csv'))
+```
+
+- **Write Excel**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import pandas as pd
+
+    df = pd.read_csv('path/to/your/data.csv')
+    df.to_excel(SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.xlsx'), engine='xlsxwriter')
+```
+
+- **Write Text**:
+
+```python
+    from marvin.tools.spio import SPIO
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/data.txt') as spio:
+        spio.write_lines(['Hello, SPIO!'])
+```
+
+
+- **Write GeoTiff**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import rasterio
+    
+    with rasterio.open('path/to/your/map.tiff') as src:
+        data = src.read()[0]
+        profile = src.profile
+       
+        with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/map.tiff') as spio:  # rasterio will not flush or close the SPIO file, so we need to do it ourselves
+            with rasterio.open(spio, 'w', **profile) as dst:
+                dst.write(data, 1)
+```
+
+
+- **Write GeoPackage Fiona**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file('path/to/your/geopackage.gpkg')
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg') as spio:  # fiona will not flush or close the IO-object, so we need to do it ourselves
+        gdf.to_file(spio, layer='mylayer', driver='GPKG', engine='fiona')
+```
+
+- **Write GeoPackage Fiona**:
+
+```python
+    from marvin.tools.spio import SPIO
+    import geopandas as gpd
+
+    gdf = gpd.read_file('path/to/your/geopackage.gpkg')
+
+    with SPIO('https://yourdomain.sharepoint.com/:i:/r/sites/your-site/path/to/your/geopackage.gpkg') as spio:  # pyogrio will not flush the IO-object
+        gdf.to_file(spio.io_delegate(), layer='mylayer', driver='GPKG', engine='pyogrio')  # pyogrio can only write to a real BytesIO object.
+```
+
+### Additional Functionalities
+
+Spio also provides additional functions to handle different data formats and processes.
+You can explore these in the https://git.vito.be/projects/MARVIN/repos/sharepoint_tools/browse/test/tst_spio.py.
+
+
+## Contributing
+
+If you want to contribute to spio, please follow the standard contributing guidelines and push your changes to a new branch in
+https://git.vito.be/projects/MARVIN/repos/sharepoint_tools
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE.md file for details.

spio-0.0.0/pyproject.toml

@@ -0,0 +1,26 @@
+
+[tool.poetry]
+name = "spio"
+version = "0.0.0"
+description = "Magical Marvin SharePoint Library"
+authors = ["Roeland Maes <roeland.maes@vito.be>", "Thomas Danckaert <thomas.danckaert@vito.be>"]
+packages = [
+    { include = "src/marvin" }
+]
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = "^3.10"
+msal_extensions = "^0.3"
+Office365-REST-Python-Client = "^2.3"
+
+[tool.poetry.extras]
+pandas = ["pandas"]
+pandas_xlsx = ["pandas", "xlsxwriter"]
+pandas_parquet = ["pandas", "pyarrow"]
+full = ["numpy", "pyproj", "gdal", "cfgrib", "dynaconf", "pandas", "netcdf4", "h5netcdf", "pyarrow", "geopandas", "fiona", "rasterio", "matplotlib", "openpyxl", "xlsxwriter", "pyexcelerate"]
+
+[tool.poetry-git-version-plugin]
+make_alpha_version = true
+
+

spio-0.0.0/src/marvin/tools/default_config.ini

@@ -0,0 +1,11 @@
+
+[default]
+graph_url=https://graph.microsoft.com/
+sharepoint_host=yourhost.sharepoint.com
+user_account=some.user@mail.abc
+
+[msal]
+client_id=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+authority=https://login.microsoftonline.com/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+auth_method=device
+scopes=["Files.ReadWrite", "User.ReadWrite"]