xml2db 0.12.0__tar.gz → 0.12.2__tar.gz

{xml2db-0.12.0/src/xml2db.egg-info → xml2db-0.12.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: xml2db
-Version: 0.12.0
+Version: 0.12.2
 Summary: Import complex XML files to a relational database
 Author-email: Commission de régulation de l'énergie <opensource@cre.fr>
 Project-URL: Documentation, https://cre-dev.github.io/xml2db
@@ -13,19 +13,18 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: sqlalchemy>1.4
-Requires-Dist: xmlschema==3.1.0
+Requires-Dist: xmlschema==3.3.2
 Requires-Dist: lxml==5.1.0
 Provides-Extra: docs
-Requires-Dist: mkdocs-material==9.5.23; extra == "docs"
-Requires-Dist: mkdocstrings-python==1.10.2; extra == "docs"
+Requires-Dist: mkdocs-material==9.5.34; extra == "docs"
+Requires-Dist: mkdocstrings-python==1.11.1; extra == "docs"
 Provides-Extra: tests
 Requires-Dist: pytest>=7.0; extra == "tests"
 
-# Loading complex XML files to a relational database
+# Loading XML files into a relational database
 
-`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It is designed to 
-handle complex XML files which cannot be denormalized to flat tables. It works out of the box, without any custom 
-mapping rules.
+`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It handles complex 
+XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules.
 
 It can be used within an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) data pipeline pattern as it 
 allows loading XML files into a relational data model which is very close from the source data, yet easy to work with.
@@ -52,7 +51,7 @@ document = data_model.parse_xml(
 document.insert_into_target_tables()
 ```
 
-The resulting data model will adhere closely to the XSD schema. However, `xml2db` will perform a few systematic 
+The data model created by `xml2db` will be close to the XSD schema. However, `xml2db` will perform a few systematic 
 simplifications aimed at limiting the complexity of the resulting data model and the storage footprint. The resulting 
 data model can be configured, but the above code will work out of the box, with reasonable defaults.
 
@@ -60,9 +59,9 @@ The raw data loaded into the database can then be processed if need be, using fo
 SQL views or stored procedures aimed at extracting, correcting and formatting the data into more user-friendly tables.
 
 This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. 
-Automated integration tests run against PostgreSQL, MySQL and MS SQL Server. `xml2db` does not work with SQLite. You may
-have to install additional packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL or 
-`pyodbc` for MS SQL Server).
+Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional 
+packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL, `pyodbc` for MS SQL Server or
+`duckdb_engine` for DuckDB).
 
 **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!**
 
@@ -97,7 +96,8 @@ pytest -m "not dbtest"
 
 ## Contributing
 
-`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex XML data.
+`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex 
+XML data.
 
 Contributions are welcome, as well as bug reports, starting on the project's 
 [issue page](https://github.com/cre-dev/xml2db/issues).

{xml2db-0.12.0 → xml2db-0.12.2}/README.md

@@ -1,8 +1,7 @@
-# Loading complex XML files to a relational database
+# Loading XML files into a relational database
 
-`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It is designed to 
-handle complex XML files which cannot be denormalized to flat tables. It works out of the box, without any custom 
-mapping rules.
+`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It handles complex 
+XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules.
 
 It can be used within an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) data pipeline pattern as it 
 allows loading XML files into a relational data model which is very close from the source data, yet easy to work with.
@@ -29,7 +28,7 @@ document = data_model.parse_xml(
 document.insert_into_target_tables()
 ```
 
-The resulting data model will adhere closely to the XSD schema. However, `xml2db` will perform a few systematic 
+The data model created by `xml2db` will be close to the XSD schema. However, `xml2db` will perform a few systematic 
 simplifications aimed at limiting the complexity of the resulting data model and the storage footprint. The resulting 
 data model can be configured, but the above code will work out of the box, with reasonable defaults.
 
@@ -37,9 +36,9 @@ The raw data loaded into the database can then be processed if need be, using fo
 SQL views or stored procedures aimed at extracting, correcting and formatting the data into more user-friendly tables.
 
 This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. 
-Automated integration tests run against PostgreSQL, MySQL and MS SQL Server. `xml2db` does not work with SQLite. You may
-have to install additional packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL or 
-`pyodbc` for MS SQL Server).
+Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional 
+packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL, `pyodbc` for MS SQL Server or
+`duckdb_engine` for DuckDB).
 
 **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!**
 
@@ -74,7 +73,8 @@ pytest -m "not dbtest"
 
 ## Contributing
 
-`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex XML data.
+`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex 
+XML data.
 
 Contributions are welcome, as well as bug reports, starting on the project's 
 [issue page](https://github.com/cre-dev/xml2db/issues).

{xml2db-0.12.0 → xml2db-0.12.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "xml2db"
-version = "0.12.0"
+version = "0.12.2"
 authors = [
   { name="Commission de régulation de l'énergie", email="opensource@cre.fr" },
 ]
@@ -18,12 +18,12 @@ classifiers = [
 ]
 dependencies = [
     "sqlalchemy>1.4",
-    "xmlschema==3.1.0",
+    "xmlschema==3.3.2",
     "lxml==5.1.0",
 ]
 
 [project.optional-dependencies]
-docs = ["mkdocs-material==9.5.23", "mkdocstrings-python==1.10.2"]
+docs = ["mkdocs-material==9.5.34", "mkdocstrings-python==1.11.1"]
 tests = ["pytest>=7.0"]
 
 [project.urls]

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db/document.py

@@ -2,7 +2,7 @@ import csv
 import datetime
 import logging
 from io import BytesIO
-from typing import Union, TYPE_CHECKING, Dict
+from typing import Union, TYPE_CHECKING
 from zoneinfo import ZoneInfo
 from sqlalchemy import Column, Table, text, select
 from sqlalchemy.engine import Connection
@@ -12,7 +12,6 @@ from lxml import etree
 if TYPE_CHECKING:
     from .model import DataModel
 
-from .exceptions import DataModelConfigError
 from .xml_converter import XMLConverter
 
 logger = logging.getLogger(__name__)

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db/model.py

@@ -8,8 +8,10 @@ import hashlib
 
 import xmlschema
 import sqlalchemy
+from lxml import etree
 from sqlalchemy import MetaData, create_engine, inspect
 from sqlalchemy.sql.ddl import CreateIndex, CreateTable
+from sqlalchemy.exc import ProgrammingError
 from graphlib import TopologicalSorter
 
 from .document import Document
@@ -49,6 +51,7 @@ class DataModel:
 
     Attributes:
         xml_schema: The `xmlschema.XMLSchema` object associated with this data model
+        lxml_schema: The `lxml.etree.XMLSchema` object associated with this data model
         data_flow_name: A short identifier used for the data model (`short_name` argument value)
         data_flow_long_name: A longer for the data model (`long_name` argument value)
         db_schema: A database schema name to store the database tables
@@ -72,22 +75,22 @@ class DataModel:
         base_url: str = None,
         model_config: dict = None,
         connection_string: str = None,
-        db_engine: str = None,
+        db_engine: sqlalchemy.Engine = None,
         db_type: str = None,
         db_schema: str = None,
         temp_prefix: str = None,
     ):
         self.model_config = self._validate_config(model_config)
-        self.tables_config = model_config.get("tables", {})
-
-        self.xml_schema = xmlschema.XMLSchema(
-            os.path.basename(xsd_file) if base_url is None else xsd_file,
-            base_url=(
-                base_url
-                if base_url is not None
-                else os.path.normpath(os.path.dirname(xsd_file))
-            ),
-        )
+        self.tables_config = model_config.get("tables", {}) if model_config else {}
+
+        xsd_file_name = xsd_file
+        if base_url is None:
+            base_url = os.path.normpath(os.path.dirname(xsd_file))
+            xsd_file_name = os.path.basename(xsd_file)
+
+        self.xml_schema = xmlschema.XMLSchema(xsd_file_name, base_url=base_url)
+        self.lxml_schema = etree.XMLSchema(etree.parse(xsd_file))
+
         self.xml_converter = XMLConverter(data_model=self)
         self.data_flow_name = short_name
         self.data_flow_long_name = long_name
@@ -104,10 +107,12 @@ class DataModel:
             else:
                 engine_options = {}
                 if "mssql" in connection_string:
-                    engine_options = {"fast_executemany": True}
+                    engine_options = {
+                        "fast_executemany": True,
+                        "isolation_level": "SERIALIZABLE",
+                    }
                 self.engine = create_engine(
                     connection_string,
-                    isolation_level="SERIALIZABLE",
                     **engine_options,
                 )
             self.db_type = self.engine.dialect.name
@@ -647,13 +652,24 @@ class DataModel:
         You do not have to call this method explicitly when using
             [`Document.insert_into_target_tables()`](document.md#xml2db.document.Document.insert_into_target_tables).
         """
+
+        def do_create_schema():
+            with self.engine.connect() as conn:
+                conn.execute(sqlalchemy.schema.CreateSchema(self.db_schema))
+                conn.commit()
+
         if self.db_schema is not None:
-            inspector = inspect(self.engine)
-            if self.db_schema not in inspector.get_schema_names():
-                with self.engine.connect() as conn:
-                    conn.execute(sqlalchemy.schema.CreateSchema(self.db_schema))
-                    conn.commit()
-                logger.info(f"Created schema: {self.db_schema}")
+            if self.db_type == "duckdb":
+                try:
+                    do_create_schema()
+                except ProgrammingError:
+                    pass
+            else:
+                inspector = inspect(self.engine)
+                if self.db_schema not in inspector.get_schema_names():
+                    do_create_schema()
+
+            logger.info(f"Created schema: {self.db_schema}")
 
     def drop_all_tables(self):
         """Drop the data model target (unprefixed) tables.

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db/table/duplicated_table.py

@@ -9,6 +9,7 @@ from sqlalchemy import (
     Boolean,
     select,
     and_,
+    Sequence,
 )
 
 from .transformed_table import DataModelTableTransformed
@@ -83,10 +84,23 @@ class DataModelTableDuplicated(DataModelTableTransformed):
             if callable(self.config.get("extra_args", []))
             else self.config.get("extra_args", [])
         )
+        if self.data_model.db_type == "duckdb":
+            pk_sequence = Sequence(f"pk_sequ_{self.name}")
+            pk_col = Column(
+                f"pk_{self.name}",
+                Integer,
+                pk_sequence,
+                server_default=pk_sequence.next_value(),
+                primary_key=True,
+            )
+        else:
+            pk_col = Column(
+                f"pk_{self.name}", Integer, primary_key=True, autoincrement=True
+            )
         self.table = Table(
             self.name,
             self.metadata,
-            Column(f"pk_{self.name}", Integer, primary_key=True, autoincrement=True),
+            pk_col,
             PrimaryKeyConstraint(
                 name=f"cx_pk_{self.name}",
                 mssql_clustered=not self.config["as_columnstore"],

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db/table/reused_table.py

@@ -7,6 +7,7 @@ from sqlalchemy import (
     UniqueConstraint,
     Boolean,
     select,
+    Sequence,
 )
 
 from .transformed_table import DataModelTableTransformed
@@ -86,10 +87,25 @@ class DataModelTableReused(DataModelTableTransformed):
             if callable(self.config.get("extra_args", []))
             else self.config.get("extra_args", [])
         )
+
+        if self.data_model.db_type == "duckdb":
+            pk_sequence = Sequence(f"pk_sequ_{self.name}")
+            pk_col = Column(
+                f"pk_{self.name}",
+                Integer,
+                pk_sequence,
+                server_default=pk_sequence.next_value(),
+                primary_key=True,
+            )
+        else:
+            pk_col = Column(
+                f"pk_{self.name}", Integer, primary_key=True, autoincrement=True
+            )
+
         self.table = Table(
             self.name,
             self.metadata,
-            Column(f"pk_{self.name}", Integer, primary_key=True, autoincrement=True),
+            pk_col,
             PrimaryKeyConstraint(
                 name=f"cx_pk_{self.name}",
                 mssql_clustered=not self.config["as_columnstore"],

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db/xml_converter.py

@@ -6,8 +6,6 @@ from lxml import etree
 from io import BytesIO
 from itertools import zip_longest
 
-from .exceptions import DataModelConfigError
-
 
 if typing.TYPE_CHECKING:
     from .model import DataModel
@@ -76,7 +74,7 @@ class XMLConverter:
             logger.info("Skipping XML file validation")
         else:
             logger.info("Validating XML file against the schema")
-            if not self.model.xml_schema.is_valid(xt if xt else xml_file):
+            if not self.model.lxml_schema.validate(xt if xt else etree.parse(xml_file)):
                 logger.error(f"XML file {file_path} does not conform with the schema")
                 raise ValueError(
                     f"XML file {file_path} does not conform with the schema"

{xml2db-0.12.0 → xml2db-0.12.2/src/xml2db.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: xml2db
-Version: 0.12.0
+Version: 0.12.2
 Summary: Import complex XML files to a relational database
 Author-email: Commission de régulation de l'énergie <opensource@cre.fr>
 Project-URL: Documentation, https://cre-dev.github.io/xml2db
@@ -13,19 +13,18 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: sqlalchemy>1.4
-Requires-Dist: xmlschema==3.1.0
+Requires-Dist: xmlschema==3.3.2
 Requires-Dist: lxml==5.1.0
 Provides-Extra: docs
-Requires-Dist: mkdocs-material==9.5.23; extra == "docs"
-Requires-Dist: mkdocstrings-python==1.10.2; extra == "docs"
+Requires-Dist: mkdocs-material==9.5.34; extra == "docs"
+Requires-Dist: mkdocstrings-python==1.11.1; extra == "docs"
 Provides-Extra: tests
 Requires-Dist: pytest>=7.0; extra == "tests"
 
-# Loading complex XML files to a relational database
+# Loading XML files into a relational database
 
-`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It is designed to 
-handle complex XML files which cannot be denormalized to flat tables. It works out of the box, without any custom 
-mapping rules.
+`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It handles complex 
+XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules.
 
 It can be used within an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) data pipeline pattern as it 
 allows loading XML files into a relational data model which is very close from the source data, yet easy to work with.
@@ -52,7 +51,7 @@ document = data_model.parse_xml(
 document.insert_into_target_tables()
 ```
 
-The resulting data model will adhere closely to the XSD schema. However, `xml2db` will perform a few systematic 
+The data model created by `xml2db` will be close to the XSD schema. However, `xml2db` will perform a few systematic 
 simplifications aimed at limiting the complexity of the resulting data model and the storage footprint. The resulting 
 data model can be configured, but the above code will work out of the box, with reasonable defaults.
 
@@ -60,9 +59,9 @@ The raw data loaded into the database can then be processed if need be, using fo
 SQL views or stored procedures aimed at extracting, correcting and formatting the data into more user-friendly tables.
 
 This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. 
-Automated integration tests run against PostgreSQL, MySQL and MS SQL Server. `xml2db` does not work with SQLite. You may
-have to install additional packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL or 
-`pyodbc` for MS SQL Server).
+Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional 
+packages to connect to your database (e.g. `psycopg2` for PostgreSQL, `pymysql` for MySQL, `pyodbc` for MS SQL Server or
+`duckdb_engine` for DuckDB).
 
 **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!**
 
@@ -97,7 +96,8 @@ pytest -m "not dbtest"
 
 ## Contributing
 
-`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex XML data.
+`xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex 
+XML data.
 
 Contributions are welcome, as well as bug reports, starting on the project's 
 [issue page](https://github.com/cre-dev/xml2db/issues).

{xml2db-0.12.0 → xml2db-0.12.2}/src/xml2db.egg-info/SOURCES.txt

@@ -20,4 +20,5 @@ src/xml2db/table/table.py
 src/xml2db/table/transformed_table.py
 tests/test_conversions.py
 tests/test_models_output.py
-tests/test_roundtrip.py
+tests/test_roundtrip.py
+tests/test_validation.py

xml2db-0.12.2/src/xml2db.egg-info/requires.txt

@@ -0,0 +1,10 @@
+sqlalchemy>1.4
+xmlschema==3.3.2
+lxml==5.1.0
+
+[docs]
+mkdocs-material==9.5.34
+mkdocstrings-python==1.11.1
+
+[tests]
+pytest>=7.0

{xml2db-0.12.0 → xml2db-0.12.2}/tests/test_roundtrip.py

@@ -1,8 +1,9 @@
 import os
+
 import pytest
 from lxml import etree
-from xml2db.xml_converter import XMLConverter, remove_record_hash
 
+from xml2db.xml_converter import XMLConverter, remove_record_hash
 from .fixtures import setup_db_model, conn_string
 from .sample_models import models
 

xml2db-0.12.2/tests/test_validation.py

@@ -0,0 +1,85 @@
+import xml.etree.ElementTree
+
+import lxml.etree
+import pytest
+
+from xml2db import DataModel
+from .sample_models import models
+
+
+@pytest.mark.parametrize(
+    "args",
+    [
+        ("invalid", True, False, ValueError),
+        ("invalid", True, True, ValueError),
+        ("invalid", False, False, ValueError),
+        ("invalid", False, True, ValueError),
+        ("malformed_recover", True, False, lxml.etree.XMLSyntaxError),
+        ("malformed_recover", True, True, None),
+        ("malformed_recover", False, False, lxml.etree.XMLSyntaxError),
+        ("malformed_recover", False, True, None),
+        ("malformed_no_recover", True, False, lxml.etree.XMLSyntaxError),
+        ("malformed_no_recover", True, True, ValueError),
+        ("malformed_no_recover", False, False, lxml.etree.XMLSyntaxError),
+        ("malformed_no_recover", False, True, ValueError),
+    ],
+)
+def test_invalid_xml(args: tuple):
+
+    file_name, iterparse, recover, exception = args
+    data_model = DataModel(models[0]["xsd_path"])
+
+    if exception is None:
+        data_model.parse_xml(
+            f"tests/sample_models/orders/invalid_xml/{file_name}.xml",
+            skip_validation=False,
+            iterparse=iterparse,
+            recover=recover,
+        )
+    else:
+        with pytest.raises(exception):
+            data_model.parse_xml(
+                f"tests/sample_models/orders/invalid_xml/{file_name}.xml",
+                skip_validation=False,
+                iterparse=iterparse,
+                recover=recover,
+            )
+
+
+@pytest.mark.parametrize(
+    "args",
+    [
+        ("invalid", True, False, IndexError),
+        ("invalid", True, True, IndexError),
+        ("invalid", False, False, None),
+        ("invalid", False, True, None),
+        ("malformed_recover", True, False, lxml.etree.XMLSyntaxError),
+        ("malformed_recover", True, True, None),
+        ("malformed_recover", False, False, lxml.etree.XMLSyntaxError),
+        ("malformed_recover", False, True, None),
+        ("malformed_no_recover", True, False, lxml.etree.XMLSyntaxError),
+        ("malformed_no_recover", True, True, IndexError),
+        ("malformed_no_recover", False, False, lxml.etree.XMLSyntaxError),
+        ("malformed_no_recover", False, True, None),
+    ],
+)
+def test_invalid_xml_skip_verify(args: tuple):
+
+    file_name, iterparse, recover, exception = args
+    data_model = DataModel(models[0]["xsd_path"])
+
+    if exception is None:
+        data_model.parse_xml(
+            f"tests/sample_models/orders/invalid_xml/{file_name}.xml",
+            skip_validation=True,
+            iterparse=iterparse,
+            recover=recover,
+        )
+    else:
+        with pytest.raises(exception):
+            data_model.parse_xml(
+                f"tests/sample_models/orders/invalid_xml/{file_name}.xml",
+                skip_validation=True,
+                iterparse=iterparse,
+                recover=recover,
+            )

xml2db-0.12.0/src/xml2db.egg-info/requires.txt

@@ -1,10 +0,0 @@
-sqlalchemy>1.4
-xmlschema==3.1.0
-lxml==5.1.0
-
-[docs]
-mkdocs-material==9.5.23
-mkdocstrings-python==1.10.2
-
-[tests]
-pytest>=7.0

{xml2db-0.12.0 → xml2db-0.12.2}/tests/test_conversions.py

@@ -1,10 +1,10 @@
 import os
+
 import pytest
 from lxml import etree
 
 from xml2db import DataModel
 from xml2db.xml_converter import XMLConverter, remove_record_hash
-
 from .sample_models import models
 
 

{xml2db-0.12.0 → xml2db-0.12.2}/tests/test_models_output.py

@@ -1,9 +1,9 @@
 import os
+
 import pytest
 from sqlalchemy.dialects import postgresql, mssql, mysql
 
 from xml2db import DataModel
-
 from .sample_models import models