pyogrio 0.10.0__tar.gz → 0.11.1__tar.gz

{pyogrio-0.10.0/pyogrio.egg-info → pyogrio-0.11.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: pyogrio
-Version: 0.10.0
+Version: 0.11.1
 Summary: Vectorized spatial vector file format I/O using GDAL/OGR
 Author: pyogrio contributors
 Author-email: "Brendan C. Ward" <bcward@astutespruce.com>
@@ -50,39 +50,34 @@ Provides-Extra: benchmark
 Requires-Dist: pytest-benchmark; extra == "benchmark"
 Provides-Extra: geopandas
 Requires-Dist: geopandas; extra == "geopandas"
+Dynamic: license-file
 
-# pyogrio - Vectorized spatial vector file format I/O using GDAL/OGR
+# pyogrio - bulk-oriented spatial vector file I/O using GDAL/OGR
 
-Pyogrio provides a
-[GeoPandas](https://github.com/geopandas/geopandas)-oriented API to OGR vector
-data sources, such as ESRI Shapefile, GeoPackage, and GeoJSON. Vector data sources
-have geometries, such as points, lines, or polygons, and associated records
-with potentially many columns worth of data.
+Pyogrio provides fast, bulk-oriented read and write access to 
+[GDAL/OGR](https://gdal.org/en/latest/drivers/vector/index.html) vector data
+sources, such as ESRI Shapefile, GeoPackage, GeoJSON, and several others.
+Vector data sources typically have geometries, such as points, lines, or
+polygons, and associated records with potentially many columns worth of data.
 
-Pyogrio uses a vectorized approach for reading and writing GeoDataFrames to and
-from OGR vector data sources in order to give you faster interoperability. It
-uses pre-compiled bindings for GDAL/OGR so that the performance is primarily
-limited by the underlying I/O speed of data source drivers in GDAL/OGR rather
-than multiple steps of converting to and from Python data types within Python.
+The typical use is to read or write these data sources to/from
+[GeoPandas](https://github.com/geopandas/geopandas) `GeoDataFrames`. Because
+the geometry column is optional, reading or writing only non-spatial data is
+also possible. Hence, GeoPackage attribute tables, DBF files, or CSV files are
+also supported.
 
-We have seen \>5-10x speedups reading files and \>5-20x speedups writing files
-compared to using non-vectorized approaches (Fiona and current I/O support in
-GeoPandas).
-
-You can read these data sources into
-`GeoDataFrames`, read just the non-geometry columns into Pandas `DataFrames`,
-or even read non-spatial data sources that exist alongside vector data sources,
-such as tables in a ESRI File Geodatabase, or antiquated DBF files.
+Pyogrio is fast because it uses pre-compiled bindings for GDAL/OGR to read and
+write the data records in bulk. This approach avoids multiple steps of
+converting to and from Python data types within Python, so performance becomes
+primarily limited by the underlying I/O speed of data source drivers in
+GDAL/OGR.
 
-Pyogrio also enables you to write `GeoDataFrames` to at least a few different
-OGR vector data source formats.
+We have seen \>5-10x speedups reading files and \>5-20x speedups writing files
+compared to using row-per-row approaches (e.g. Fiona).
 
 Read the documentation for more information:
 [https://pyogrio.readthedocs.io](https://pyogrio.readthedocs.io/en/latest/).
 
-WARNING: Pyogrio is still at an early version and the API is subject to
-substantial change. Please see [CHANGES](CHANGES.md).
-
 ## Requirements
 
 Supports Python 3.9 - 3.13 and GDAL 3.4.x - 3.9.x.
@@ -105,9 +100,9 @@ for more information.
 
 ## Supported vector formats
 
-Pyogrio supports some of the most common vector data source formats (provided
-they are also supported by GDAL/OGR), including ESRI Shapefile, GeoPackage,
-GeoJSON, and FlatGeobuf.
+Pyogrio supports most common vector data source formats (provided they are also
+supported by GDAL/OGR), including ESRI Shapefile, GeoPackage, GeoJSON, and
+FlatGeobuf.
 
 Please see the [list of supported formats](https://pyogrio.readthedocs.io/en/latest/supported_formats.html)
 for more information.
@@ -117,7 +112,7 @@ for more information.
 Please read the [introduction](https://pyogrio.readthedocs.io/en/latest/supported_formats.html)
 for more information and examples to get started using Pyogrio.
 
-You can also check out the the [API documentation](https://pyogrio.readthedocs.io/en/latest/api.html)
+You can also check out the [API documentation](https://pyogrio.readthedocs.io/en/latest/api.html)
 for full details on using the API.
 
 ## Credits

pyogrio-0.11.1/README.md

@@ -0,0 +1,70 @@
+# pyogrio - bulk-oriented spatial vector file I/O using GDAL/OGR
+
+Pyogrio provides fast, bulk-oriented read and write access to 
+[GDAL/OGR](https://gdal.org/en/latest/drivers/vector/index.html) vector data
+sources, such as ESRI Shapefile, GeoPackage, GeoJSON, and several others.
+Vector data sources typically have geometries, such as points, lines, or
+polygons, and associated records with potentially many columns worth of data.
+
+The typical use is to read or write these data sources to/from
+[GeoPandas](https://github.com/geopandas/geopandas) `GeoDataFrames`. Because
+the geometry column is optional, reading or writing only non-spatial data is
+also possible. Hence, GeoPackage attribute tables, DBF files, or CSV files are
+also supported.
+
+Pyogrio is fast because it uses pre-compiled bindings for GDAL/OGR to read and
+write the data records in bulk. This approach avoids multiple steps of
+converting to and from Python data types within Python, so performance becomes
+primarily limited by the underlying I/O speed of data source drivers in
+GDAL/OGR.
+
+We have seen \>5-10x speedups reading files and \>5-20x speedups writing files
+compared to using row-per-row approaches (e.g. Fiona).
+
+Read the documentation for more information:
+[https://pyogrio.readthedocs.io](https://pyogrio.readthedocs.io/en/latest/).
+
+## Requirements
+
+Supports Python 3.9 - 3.13 and GDAL 3.4.x - 3.9.x.
+
+Reading to GeoDataFrames requires `geopandas>=0.12` with `shapely>=2`.
+
+Additionally, installing `pyarrow` in combination with GDAL 3.6+ enables
+a further speed-up when specifying `use_arrow=True`.
+
+## Installation
+
+Pyogrio is currently available on
+[conda-forge](https://anaconda.org/conda-forge/pyogrio)
+and [PyPI](https://pypi.org/project/pyogrio/)
+for Linux, MacOS, and Windows.
+
+Please read the
+[installation documentation](https://pyogrio.readthedocs.io/en/latest/install.html)
+for more information.
+
+## Supported vector formats
+
+Pyogrio supports most common vector data source formats (provided they are also
+supported by GDAL/OGR), including ESRI Shapefile, GeoPackage, GeoJSON, and
+FlatGeobuf.
+
+Please see the [list of supported formats](https://pyogrio.readthedocs.io/en/latest/supported_formats.html)
+for more information.
+
+## Getting started
+
+Please read the [introduction](https://pyogrio.readthedocs.io/en/latest/supported_formats.html)
+for more information and examples to get started using Pyogrio.
+
+You can also check out the [API documentation](https://pyogrio.readthedocs.io/en/latest/api.html)
+for full details on using the API.
+
+## Credits
+
+This project is made possible by the tremendous efforts of the GDAL, Fiona, and
+Geopandas communities.
+
+-   Core I/O methods and supporting functions adapted from [Fiona](https://github.com/Toblerity/Fiona)
+-   Inspired by [Fiona PR](https://github.com/Toblerity/Fiona/pull/540/files)

{pyogrio-0.10.0 → pyogrio-0.11.1}/pyogrio/__init__.py

@@ -4,7 +4,9 @@ try:
     # we try importing shapely, to ensure it is imported (and it can load its
     # own GEOS copy) before we load GDAL and its linked GEOS
     import shapely
-    import shapely.geos  # noqa: F401
+
+    if shapely.__version__ < "2.0.0":
+        import shapely.geos
 except Exception:
     pass
 
@@ -32,24 +34,24 @@ __version__ = get_versions()["version"]
 del get_versions
 
 __all__ = [
-    "list_drivers",
+    "__gdal_geos_version__",
+    "__gdal_version__",
+    "__gdal_version_string__",
+    "__version__",
     "detect_write_driver",
-    "list_layers",
-    "read_bounds",
-    "read_info",
-    "set_gdal_config_options",
     "get_gdal_config_option",
     "get_gdal_data_path",
+    "list_drivers",
+    "list_layers",
     "open_arrow",
     "read_arrow",
+    "read_bounds",
     "read_dataframe",
+    "read_info",
+    "set_gdal_config_options",
     "vsi_listtree",
     "vsi_rmtree",
     "vsi_unlink",
     "write_arrow",
     "write_dataframe",
-    "__gdal_version__",
-    "__gdal_version_string__",
-    "__gdal_geos_version__",
-    "__version__",
 ]

{pyogrio-0.10.0 → pyogrio-0.11.1}/pyogrio/_compat.py

@@ -33,15 +33,23 @@ HAS_ARROW_API = __gdal_version__ >= (3, 6, 0)
 HAS_ARROW_WRITE_API = __gdal_version__ >= (3, 8, 0)
 HAS_PYARROW = pyarrow is not None
 HAS_PYPROJ = pyproj is not None
+PYARROW_GE_19 = pyarrow is not None and Version(pyarrow.__version__) >= Version(
+    "19.0.0"
+)
 
 HAS_GEOPANDAS = geopandas is not None
 
 PANDAS_GE_15 = pandas is not None and Version(pandas.__version__) >= Version("1.5.0")
 PANDAS_GE_20 = pandas is not None and Version(pandas.__version__) >= Version("2.0.0")
 PANDAS_GE_22 = pandas is not None and Version(pandas.__version__) >= Version("2.2.0")
+PANDAS_GE_30 = pandas is not None and Version(pandas.__version__) >= Version("3.0.0dev")
 
+GDAL_GE_352 = __gdal_version__ >= (3, 5, 2)
+GDAL_GE_37 = __gdal_version__ >= (3, 7, 0)
 GDAL_GE_38 = __gdal_version__ >= (3, 8, 0)
+GDAL_GE_311 = __gdal_version__ >= (3, 11, 0)
 
 HAS_GDAL_GEOS = __gdal_geos_version__ is not None
 
 HAS_SHAPELY = shapely is not None and Version(shapely.__version__) >= Version("2.0.0")
+SHAPELY_GE_21 = shapely is not None and Version(shapely.__version__) >= Version("2.1.0")

pyogrio-0.11.1/pyogrio/_err.pxd

@@ -0,0 +1,9 @@
+cdef object check_last_error()
+cdef int check_int(int retval) except -1
+cdef void *check_pointer(void *ptr) except NULL
+
+cdef class ErrorHandler:
+    cdef object error_stack
+    cdef int check_int(self, int retval, bint squash_errors) except -1
+    cdef void *check_pointer(self, void *ptr, bint squash_errors) except NULL
+    cdef void _handle_error_stack(self, bint squash_errors)

pyogrio-0.11.1/pyogrio/_err.pyx

@@ -0,0 +1,441 @@
+"""Error handling code for GDAL/OGR.
+
+Ported from fiona::_err.pyx
+"""
+
+import contextlib
+import warnings
+from contextvars import ContextVar
+from itertools import zip_longest
+
+from pyogrio._ogr cimport (
+    CE_Warning, CE_Failure, CE_Fatal, CPLErrorReset,
+    CPLGetLastErrorType, CPLGetLastErrorNo, CPLGetLastErrorMsg,
+    OGRERR_NONE, CPLErr, CPLErrorHandler, CPLDefaultErrorHandler,
+    CPLPopErrorHandler, CPLPushErrorHandler)
+
+_ERROR_STACK = ContextVar("error_stack")
+_ERROR_STACK.set([])
+
+
+class CPLE_BaseError(Exception):
+    """Base CPL error class.
+
+    For internal use within Cython only.
+    """
+
+    def __init__(self, error, errno, errmsg):
+        self.error = error
+        self.errno = errno
+        self.errmsg = errmsg
+
+    def __str__(self):
+        return self.__unicode__()
+
+    def __unicode__(self):
+        return u"{}".format(self.errmsg)
+
+    @property
+    def args(self):
+        return self.error, self.errno, self.errmsg
+
+
+class CPLE_AppDefinedError(CPLE_BaseError):
+    pass
+
+
+class CPLE_OutOfMemoryError(CPLE_BaseError):
+    pass
+
+
+class CPLE_FileIOError(CPLE_BaseError):
+    pass
+
+
+class CPLE_OpenFailedError(CPLE_BaseError):
+    pass
+
+
+class CPLE_IllegalArgError(CPLE_BaseError):
+    pass
+
+
+class CPLE_NotSupportedError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AssertionFailedError(CPLE_BaseError):
+    pass
+
+
+class CPLE_NoWriteAccessError(CPLE_BaseError):
+    pass
+
+
+class CPLE_UserInterruptError(CPLE_BaseError):
+    pass
+
+
+class ObjectNullError(CPLE_BaseError):
+    pass
+
+
+class CPLE_HttpResponseError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSBucketNotFoundError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSObjectNotFoundError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSAccessDeniedError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSInvalidCredentialsError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSSignatureDoesNotMatchError(CPLE_BaseError):
+    pass
+
+
+class CPLE_AWSError(CPLE_BaseError):
+    pass
+
+
+class NullPointerError(CPLE_BaseError):
+    """
+    Returned from check_pointer when a NULL pointer is passed, but no GDAL
+    error was raised.
+    """
+    pass
+
+
+class CPLError(CPLE_BaseError):
+    """
+    Returned from check_int when a error code is returned, but no GDAL
+    error was set.
+    """
+    pass
+
+
+# Map of GDAL error numbers to the Python exceptions.
+exception_map = {
+    1: CPLE_AppDefinedError,
+    2: CPLE_OutOfMemoryError,
+    3: CPLE_FileIOError,
+    4: CPLE_OpenFailedError,
+    5: CPLE_IllegalArgError,
+    6: CPLE_NotSupportedError,
+    7: CPLE_AssertionFailedError,
+    8: CPLE_NoWriteAccessError,
+    9: CPLE_UserInterruptError,
+    10: ObjectNullError,
+
+    # error numbers 11-16 are introduced in GDAL 2.1. See
+    # https://github.com/OSGeo/gdal/pull/98.
+    11: CPLE_HttpResponseError,
+    12: CPLE_AWSBucketNotFoundError,
+    13: CPLE_AWSObjectNotFoundError,
+    14: CPLE_AWSAccessDeniedError,
+    15: CPLE_AWSInvalidCredentialsError,
+    16: CPLE_AWSSignatureDoesNotMatchError,
+    17: CPLE_AWSError
+}
+
+
+cdef inline object check_last_error():
+    """Checks if the last GDAL error was a fatal or non-fatal error.
+
+    When a non-fatal error is found, an appropriate exception is raised.
+
+    When a fatal error is found, SystemExit is called.
+
+    Returns
+    -------
+    An Exception, SystemExit, or None
+    """
+    err_type = CPLGetLastErrorType()
+    err_no = CPLGetLastErrorNo()
+    err_msg = clean_error_message(CPLGetLastErrorMsg())
+    if err_msg == "":
+        err_msg = "No error message."
+
+    if err_type == CE_Failure:
+        CPLErrorReset()
+        return exception_map.get(
+            err_no, CPLE_BaseError)(err_type, err_no, err_msg)
+
+    if err_type == CE_Fatal:
+        return SystemExit("Fatal error: {0}".format((err_type, err_no, err_msg)))
+
+
+cdef clean_error_message(const char* err_msg):
+    """Cleans up error messages from GDAL.
+
+    Parameters
+    ----------
+    err_msg : const char*
+        The error message to clean up.
+
+    Returns
+    -------
+    str
+        The cleaned up error message or empty string
+    """
+    if err_msg != NULL:
+        # Reformat message.
+        msg_b = err_msg
+        try:
+            msg = msg_b.decode("utf-8")
+            msg = msg.replace("`", "'")
+            msg = msg.replace("\n", " ")
+        except UnicodeDecodeError as exc:
+            msg = f"Could not decode error message to UTF-8. Raw error: {msg_b}"
+
+    else:
+        msg = ""
+
+    return msg
+
+
+cdef void *check_pointer(void *ptr) except NULL:
+    """Check the pointer returned by a GDAL/OGR function.
+
+    If `ptr` is `NULL`, an exception inheriting from CPLE_BaseError is raised.
+    When the last error registered by GDAL/OGR was a non-fatal error, the
+    exception raised will be customized appropriately. Otherwise a
+    NullPointerError is raised.
+    """
+    if ptr == NULL:
+        exc = check_last_error()
+        if exc:
+            raise exc
+        else:
+            # null pointer was passed, but no error message from GDAL
+            raise NullPointerError(-1, -1, "NULL pointer error")
+
+    return ptr
+
+
+cdef int check_int(int err) except -1:
+    """Check the CPLErr (int) value returned by a GDAL/OGR function.
+
+    If `err` is not OGRERR_NONE, an exception inheriting from CPLE_BaseError is raised.
+    When the last error registered by GDAL/OGR was a non-fatal error, the
+    exception raised will be customized appropriately. Otherwise a CPLError is
+    raised.
+    """
+    if err != OGRERR_NONE:
+        exc = check_last_error()
+        if exc:
+            raise exc
+        else:
+            # no error message from GDAL
+            raise CPLError(-1, -1, "Unspecified OGR / GDAL error")
+
+    return err
+
+
+cdef void error_handler(
+    CPLErr err_class, int err_no, const char* err_msg
+) noexcept nogil:
+    """Custom CPL error handler to match the Python behaviour.
+
+    For non-fatal errors (CE_Failure), error printing to stderr (behaviour of
+    the default GDAL error handler) is suppressed, because we already raise a
+    Python exception that includes the error message.
+
+    Warnings are converted to Python warnings.
+    """
+    if err_class == CE_Fatal:
+        # If the error class is CE_Fatal, we want to have a message issued
+        # because the CPL support code does an abort() before any exception
+        # can be generated
+        CPLDefaultErrorHandler(err_class, err_no, err_msg)
+        return
+
+    if err_class == CE_Failure:
+        # For Failures, do nothing as those are explicitly caught
+        # with error return codes and translated into Python exceptions
+        return
+
+    if err_class == CE_Warning:
+        with gil:
+            warnings.warn(clean_error_message(err_msg), RuntimeWarning)
+        return
+
+    # Fall back to the default handler for non-failure messages since
+    # they won't be translated into exceptions.
+    CPLDefaultErrorHandler(err_class, err_no, err_msg)
+
+
+def _register_error_handler():
+    CPLPushErrorHandler(<CPLErrorHandler>error_handler)
+
+
+cdef class ErrorHandler:
+
+    def __init__(self, error_stack=None):
+        self.error_stack = error_stack or {}
+
+    cdef int check_int(self, int err, bint squash_errors) except -1:
+        """Check the CPLErr (int) value returned by a GDAL/OGR function.
+
+        If `err` is not OGRERR_NONE, an exception inheriting from CPLE_BaseError is
+        raised.
+        When a non-fatal GDAL/OGR error was captured in the error stack, the
+        exception raised will be customized appropriately. Otherwise, a
+        CPLError is raised.
+
+        Parameters
+        ----------
+        err : int
+            The CPLErr returned by a GDAL/OGR function.
+        squash_errors : bool
+            True to squash all errors captured to one error with the exception type of
+            the last error and all error messages concatenated.
+
+        Returns
+        -------
+        int
+            The `err` input parameter if it is OGRERR_NONE. Otherwise an exception is
+            raised.
+
+        """
+        if err != OGRERR_NONE:
+            if self.error_stack.get():
+                self._handle_error_stack(squash_errors)
+            else:
+                raise CPLError(CE_Failure, err, "Unspecified OGR / GDAL error")
+
+        return err
+
+    cdef void *check_pointer(self, void *ptr, bint squash_errors) except NULL:
+        """Check the pointer returned by a GDAL/OGR function.
+
+        If `ptr` is `NULL`, an exception inheriting from CPLE_BaseError is
+        raised.
+        When a non-fatal GDAL/OGR error was captured in the error stack, the
+        exception raised will be customized appropriately. Otherwise, a
+        NullPointerError is raised.
+
+        Parameters
+        ----------
+        ptr : pointer
+            The pointer returned by a GDAL/OGR function.
+        squash_errors : bool
+            True to squash all errors captured to one error with the exception type of
+            the last error and all error messages concatenated.
+
+        Returns
+        -------
+        pointer
+            The `ptr` input parameter if it is not `NULL`. Otherwise an exception is
+            raised.
+
+        """
+        if ptr == NULL:
+            if self.error_stack.get():
+                self._handle_error_stack(squash_errors)
+            else:
+                raise NullPointerError(-1, -1, "NULL pointer error")
+
+        return ptr
+
+    cdef void _handle_error_stack(self, bint squash_errors):
+        """Handle the errors in `error_stack`."""
+        stack = self.error_stack.get()
+        for error, cause in zip_longest(stack[::-1], stack[::-1][1:]):
+            if error is not None and cause is not None:
+                error.__cause__ = cause
+
+        last = stack.pop()
+        if last is not None:
+            if squash_errors:
+                # Concatenate all error messages, and raise a single exception
+                errmsg = str(last)
+                inner = last.__cause__
+                while inner is not None:
+                    errmsg = f"{errmsg}; {inner}"
+                    inner = inner.__cause__
+
+                if errmsg == "":
+                    errmsg = "No error message."
+
+                raise type(last)(-1, -1, errmsg)
+
+            raise last
+
+
+cdef void stacking_error_handler(
+    CPLErr err_class,
+    int err_no,
+    const char* err_msg
+) noexcept nogil:
+    """Custom CPL error handler that adds non-fatal errors to a stack.
+
+    All non-fatal errors (CE_Failure) are not printed to stderr (behaviour
+    of the default GDAL error handler), but they are converted to python
+    exceptions and added to a stack, so they can be dealt with afterwards.
+
+    Warnings are converted to Python warnings.
+    """
+    if err_class == CE_Fatal:
+        # If the error class is CE_Fatal, we want to have a message issued
+        # because the CPL support code does an abort() before any exception
+        # can be generated
+        CPLDefaultErrorHandler(err_class, err_no, err_msg)
+        return
+
+    if err_class == CE_Failure:
+        # For Failures, add them to the error exception stack
+        with gil:
+            stack = _ERROR_STACK.get()
+            stack.append(
+                exception_map.get(err_no, CPLE_BaseError)(
+                    err_class, err_no, clean_error_message(err_msg)
+                ),
+            )
+            _ERROR_STACK.set(stack)
+
+        return
+
+    if err_class == CE_Warning:
+        with gil:
+            warnings.warn(clean_error_message(err_msg), RuntimeWarning)
+        return
+
+    # Fall back to the default handler for non-failure messages since
+    # they won't be translated into exceptions.
+    CPLDefaultErrorHandler(err_class, err_no, err_msg)
+
+
+@contextlib.contextmanager
+def capture_errors():
+    """A context manager that captures all GDAL non-fatal errors occuring.
+
+    It adds all errors to a single stack, so it assumes that no more than one
+    GDAL function is called.
+
+    Yields an ErrorHandler object that can be used to handle the errors
+    if any were captured.
+    """
+    CPLErrorReset()
+    _ERROR_STACK.set([])
+
+    # stacking_error_handler records GDAL errors in the order they occur and
+    # converts them to exceptions.
+    CPLPushErrorHandler(<CPLErrorHandler>stacking_error_handler)
+
+    # Run code in the `with` block.
+    yield ErrorHandler(_ERROR_STACK)
+
+    CPLPopErrorHandler()
+    _ERROR_STACK.set([])
+    CPLErrorReset()

{pyogrio-0.10.0 → pyogrio-0.11.1}/pyogrio/_geometry.pxd

@@ -1,4 +1,4 @@
 from pyogrio._ogr cimport *
 
 cdef str get_geometry_type(void *ogr_layer)
-cdef OGRwkbGeometryType get_geometry_type_code(str geometry_type) except *
+cdef OGRwkbGeometryType get_geometry_type_code(str geometry_type) except *