omniload 0.2.0__tar.gz → 0.3.0__tar.gz

{omniload-0.2.0 → omniload-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omniload
-Version: 0.2.0
+Version: 0.3.0
 Summary: omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.
 Author: Mustafa Ersan, Sanju Shah Gupta, Saravjeet 'Aman' Singh
 Author-email: Burak Karakan <burak.karakan@getbruin.com>
@@ -65,7 +65,7 @@ License-File: NOTICE
 Requires-Dist: aiohttp<4
 Requires-Dist: asana<3.3
 Requires-Dist: click<8.5
-Requires-Dist: clickhouse-connect<1,>=0.8
+Requires-Dist: clickhouse-connect<2,>=0.8
 Requires-Dist: clickhouse-driver<0.3
 Requires-Dist: clickhouse-sqlalchemy<0.4
 Requires-Dist: confluent-kafka<2.15
@@ -73,7 +73,7 @@ Requires-Dist: couchbase<5,>=4
 Requires-Dist: databricks-sql-connector<5
 Requires-Dist: databricks-sqlalchemy<3
 Requires-Dist: dataclasses-json<0.7
-Requires-Dist: dlt<1.22
+Requires-Dist: dlt<1.29
 Requires-Dist: dlt-cratedb<0.2
 Requires-Dist: duckdb<1.6
 Requires-Dist: duckdb-engine<0.18
@@ -128,7 +128,7 @@ Requires-Dist: poethepoet<1; extra == "develop"
 Requires-Dist: pyproject-fmt<3; extra == "develop"
 Requires-Dist: pytest-repeat<1; extra == "develop"
 Requires-Dist: ruff<0.16; extra == "develop"
-Requires-Dist: ty==0.0.51; extra == "develop"
+Requires-Dist: ty==0.0.52; extra == "develop"
 Requires-Dist: validate-pyproject<1; extra == "develop"
 Provides-Extra: full
 Requires-Dist: omniload[ibm-db,odbc,oracle]; extra == "full"
@@ -143,6 +143,7 @@ Provides-Extra: release
 Requires-Dist: build<2; extra == "release"
 Requires-Dist: twine<7; extra == "release"
 Provides-Extra: test
+Requires-Dist: boto3<2; extra == "test"
 Requires-Dist: pendulum[test]<4,>=3; extra == "test"
 Requires-Dist: pytest<10; extra == "test"
 Requires-Dist: pytest-asyncio<2; extra == "test"
@@ -167,34 +168,49 @@ Dynamic: license-file
 
 ## About
 
-omniload is a polyglot data loader framework based on dlt.
-It allows you to load data from any source into any destination,
-either using a concise CLI from your shell,
-or the Python API from your own applications.
+omniload is a polyglot data loader framework based on [dlt], the
+open-source Python library for building data pipelines. dlt does the
+heavy lifting (schema inference, normalization, and incremental load
+state); omniload wraps it behind a single CLI and a URI scheme, so you
+can copy data between any source and any destination without writing
+pipeline code, either from your shell or the Python API.
 
 omniload provides the same efficient incremental data loading modes
-inherited by [dlt]: `append`, `merge`, and `delete+insert`.
+inherited by dlt: `append`, `merge`, and `delete+insert`.
 
 ## Install
-We recommend using [uv] to install or run `omniload`.
+
+### Container
+
+For running `omniload` in a container environment, use the provided OCI
+images.
 
 ```shell
-pip install uv
+docker run --rm ghcr.io/panodata/omniload:latest version
+```
+
+### Package
+
+For running or installing the `omniload` Python package,
+we are recommending to use [uv]. [^1][^2]
+```shell
 uvx omniload
 ```
 
-Alternatively, if you'd like to install it globally:
+Alternatively, if you like to install it account-wide on your system:
 ```shell
-uv pip install --system omniload
+uv tool install omniload
 ```
 
-While installation with vanilla `pip` is possible, it's an order of magnitude slower.
+[^1]: You can install `uv` using `pip install uv`, or another method that matches your operating system.
+[^2]: While installing `omniload` with vanilla `pip` is possible, it is an order of magnitude slower.
 
 ## Synopsis
 
-The next command instructs omniload to read the table `public.some_data` from
-your PostgreSQL instance, and to write the data to your BigQuery warehouse
-under the schema `omniload` and table `some_data`.
+### CLI
+
+Instruct omniload to read from the PostgreSQL table `public.some_data`,
+and write to the BigQuery warehouse table `omniload.some_data`.
 
 ```shell
 omniload ingest \
@@ -204,6 +220,27 @@ omniload ingest \
     --dest-table 'omniload.some_data'
 ```
 
+### Python API
+
+Using `omniload.run_ingest`, you can run the same
+ingestion from your own application. The keyword arguments map one-to-one to the
+CLI options, and it returns the dlt `LoadInfo` for the run (or `None` for a dry run).
+
+```python
+from omniload import run_ingest
+
+info = run_ingest(
+    source_uri="sqlite:///./source.db",
+    dest_uri="duckdb:///./warehouse.duckdb",
+    source_table="main.some_table",
+    dest_table="public.some_table",
+)
+print(info)
+```
+
+See the [Python API documentation] for dry runs, error handling, and passing
+enum options as strings.
+
 ## Handbook
 
 Please visit the [quickstart documentation], or inspect the
@@ -240,6 +277,7 @@ the way down. Kudos.
 
 [dlt]: https://github.com/dlt-hub/dlt
 [quickstart documentation]: https://omniload.readthedocs.io/getting-started/quickstart.html
+[Python API documentation]: https://omniload.readthedocs.io/getting-started/python-api.html
 [ingestr]: https://bruin-data.github.io/ingestr/
 [Issues]: https://github.com/panodata/omniload/issues
 [LICENSE]: https://github.com/panodata/omniload/blob/main/LICENSE

{omniload-0.2.0 → omniload-0.3.0}/README.md

@@ -12,34 +12,49 @@
 
 ## About
 
-omniload is a polyglot data loader framework based on dlt.
-It allows you to load data from any source into any destination,
-either using a concise CLI from your shell,
-or the Python API from your own applications.
+omniload is a polyglot data loader framework based on [dlt], the
+open-source Python library for building data pipelines. dlt does the
+heavy lifting (schema inference, normalization, and incremental load
+state); omniload wraps it behind a single CLI and a URI scheme, so you
+can copy data between any source and any destination without writing
+pipeline code, either from your shell or the Python API.
 
 omniload provides the same efficient incremental data loading modes
-inherited by [dlt]: `append`, `merge`, and `delete+insert`.
+inherited by dlt: `append`, `merge`, and `delete+insert`.
 
 ## Install
-We recommend using [uv] to install or run `omniload`.
+
+### Container
+
+For running `omniload` in a container environment, use the provided OCI
+images.
 
 ```shell
-pip install uv
+docker run --rm ghcr.io/panodata/omniload:latest version
+```
+
+### Package
+
+For running or installing the `omniload` Python package,
+we are recommending to use [uv]. [^1][^2]
+```shell
 uvx omniload
 ```
 
-Alternatively, if you'd like to install it globally:
+Alternatively, if you like to install it account-wide on your system:
 ```shell
-uv pip install --system omniload
+uv tool install omniload
 ```
 
-While installation with vanilla `pip` is possible, it's an order of magnitude slower.
+[^1]: You can install `uv` using `pip install uv`, or another method that matches your operating system.
+[^2]: While installing `omniload` with vanilla `pip` is possible, it is an order of magnitude slower.
 
 ## Synopsis
 
-The next command instructs omniload to read the table `public.some_data` from
-your PostgreSQL instance, and to write the data to your BigQuery warehouse
-under the schema `omniload` and table `some_data`.
+### CLI
+
+Instruct omniload to read from the PostgreSQL table `public.some_data`,
+and write to the BigQuery warehouse table `omniload.some_data`.
 
 ```shell
 omniload ingest \
@@ -49,6 +64,27 @@ omniload ingest \
     --dest-table 'omniload.some_data'
 ```
 
+### Python API
+
+Using `omniload.run_ingest`, you can run the same
+ingestion from your own application. The keyword arguments map one-to-one to the
+CLI options, and it returns the dlt `LoadInfo` for the run (or `None` for a dry run).
+
+```python
+from omniload import run_ingest
+
+info = run_ingest(
+    source_uri="sqlite:///./source.db",
+    dest_uri="duckdb:///./warehouse.duckdb",
+    source_table="main.some_table",
+    dest_table="public.some_table",
+)
+print(info)
+```
+
+See the [Python API documentation] for dry runs, error handling, and passing
+enum options as strings.
+
 ## Handbook
 
 Please visit the [quickstart documentation], or inspect the
@@ -85,6 +121,7 @@ the way down. Kudos.
 
 [dlt]: https://github.com/dlt-hub/dlt
 [quickstart documentation]: https://omniload.readthedocs.io/getting-started/quickstart.html
+[Python API documentation]: https://omniload.readthedocs.io/getting-started/python-api.html
 [ingestr]: https://bruin-data.github.io/ingestr/
 [Issues]: https://github.com/panodata/omniload/issues
 [LICENSE]: https://github.com/panodata/omniload/blob/main/LICENSE

{omniload-0.2.0 → omniload-0.3.0}/docs/conf.py

@@ -97,6 +97,7 @@ linkcheck_ignore = [
     r"https://support.appsflyer.com/",
     r"https://help.docebo.com/",
     r"https://www.reddit.com/",
+    r"https://www.linkedin.com/developers.*",
 ]
 linkcheck_anchors_ignore_for_url = [
     r"https://developers.zoom.us/",

omniload-0.3.0/omniload/__init__.py

@@ -0,0 +1,37 @@
+from importlib.metadata import PackageNotFoundError, version
+from unittest import mock
+
+from omniload.api import (
+    IncrementalStrategy,
+    LoaderFileFormat,
+    Progress,
+    SchemaNaming,
+    SqlBackend,
+    SqlReflectionLevel,
+    run_ingest,
+)
+from omniload.src.errors import IngestJobError, ValidationError
+
+__appname__ = "omniload"
+
+
+try:
+    __version__ = version(__appname__)
+except PackageNotFoundError:  # pragma: no cover
+    __version__ = "0.0.0-dev"
+
+# Turn off dlt's telemetry to tame three more requests per invocation.
+mock.patch("dlt.common.runtime.telemetry._TELEMETRY_STARTED", True)
+
+__all__ = [
+    "IncrementalStrategy",
+    "IngestJobError",
+    "LoaderFileFormat",
+    "Progress",
+    "SchemaNaming",
+    "SqlBackend",
+    "SqlReflectionLevel",
+    "ValidationError",
+    "__version__",
+    "run_ingest",
+]