sqlite2duckdb 0.3.0__tar.gz → 0.5.0__tar.gz

{sqlite2duckdb-0.3.0 → sqlite2duckdb-0.5.0}/.gitignore

@@ -158,3 +158,6 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Test databases generated at the repo root
+/*.db

sqlite2duckdb-0.5.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.5
+Name: sqlite2duckdb
+Version: 0.5.0
+Summary: A tool to convert sqlite database to duckdb database
+Project-URL: Homepage, https://github.com/dridk/sqlite2duckdb
+Project-URL: Issues, https://github.com/dridk/sqlite2duckdb/issues
+Author-email: Sacha Schutz <sacha.schutz@pm.me>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: database,duckdb,olap,oltp,sqlite
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.9
+Requires-Dist: duckdb>=1.1.0
+Description-Content-Type: text/markdown
+
+# sqlite2duckdb
+
+![CI](https://github.com/dridk/sqlite2duckdb/actions/workflows/ci.yml/badge.svg)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/sqlite2duckdb)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/sqlite2duckdb)
+
+A tool for converting a [sqlite](https://www.sqlite.org/) database into a [duckdb](https://duckdb.org/) database
+
+## Description
+
+Sqlite is an embedded online database designed for transactional reading and writing.
+Duckdb is also an embedded database, but column-oriented, designed for analytical process with a very high reading efficiency.
+
+For more details [Medium post](https://towardsdatascience.com/forget-about-sqlite-use-duckdb-instead-and-thank-me-later-df76ee9bb777)
+
+
+## Installation
+
+With [uv](https://docs.astral.sh/uv/), no installation is required. `uvx` downloads and runs the tool in one go:
+
+```bash
+uvx sqlite2duckdb source.db target.db
+```
+
+To keep it around:
+
+```bash
+uv tool install sqlite2duckdb
+```
+
+Or with pip:
+
+```bash
+pip install sqlite2duckdb
+```
+
+## Usage
+
+### As a command line
+
+```
+usage: sqlite2duckdb [-f] <sqlite_path> <duckdb_path>
+
+Convert Sqlite database to Duckdb database
+
+positional arguments:
+  sqlite_path    sqlite file path
+  duckdb_path    duckdb file path
+
+options:
+  -h, --help     show this help message and exit
+  -f, --force    overwrite the duckdb file if it already exists
+  -q, --quiet    only report errors
+  --verbose      report every step
+  -v, --version  show program's version number and exit
+```
+
+### Examples
+
+```bash
+uvx sqlite2duckdb source.db target.db
+uvx sqlite2duckdb --force source.db target.db   # overwrite target.db without asking
+```
+
+### From python
+
+```python
+from sqlite2duckdb import sqlite_to_duckdb
+
+result = sqlite_to_duckdb("source.sqlite", "target.duckdb")
+print(result.tables, result.views, result.elapsed)
+```
+
+## What is converted
+
+| | |
+|---|---|
+| Tables and data | ✅ |
+| Primary keys, NOT NULL and UNIQUE constraints | ✅ |
+| Indexes | ✅ |
+| Views | ✅ best effort |
+| FOREIGN KEY and CHECK constraints | ❌ |
+
+A view whose SQL uses something duckdb has no equivalent for (`MATCH`, `julianday()`) is
+skipped with a warning instead of failing the conversion.
+
+FOREIGN KEY and CHECK are not copied, though not for lack of support: duckdb accepts and
+enforces both in `CREATE TABLE`. It checks foreign keys row by row, so a self-referencing
+table cannot be bulk loaded, and there is no `ALTER TABLE ADD CONSTRAINT` to add them once
+the data is in.
+
+
+## Contributing
+
+The project uses [uv](https://docs.astral.sh/uv/) for everything:
+
+```bash
+make dev     # uv sync — installs duckdb plus the test deps (pytest, faker, ruff)
+make test    # uv run pytest
+make lint    # uv run ruff check . && uv run ruff format --check .
+make build   # uv build
+make publish # uv publish (PyPI trusted publishing, also run on tags by CI)
+```
+
+### See also
+
+- [Harlequin](https://github.com/tconbeer/harlequin): A nice duckdb IDE for your terminal

sqlite2duckdb-0.5.0/README.md

@@ -0,0 +1,107 @@
+# sqlite2duckdb
+
+![CI](https://github.com/dridk/sqlite2duckdb/actions/workflows/ci.yml/badge.svg)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/sqlite2duckdb)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/sqlite2duckdb)
+
+A tool for converting a [sqlite](https://www.sqlite.org/) database into a [duckdb](https://duckdb.org/) database
+
+## Description
+
+Sqlite is an embedded online database designed for transactional reading and writing.
+Duckdb is also an embedded database, but column-oriented, designed for analytical process with a very high reading efficiency.
+
+For more details [Medium post](https://towardsdatascience.com/forget-about-sqlite-use-duckdb-instead-and-thank-me-later-df76ee9bb777)
+
+
+## Installation
+
+With [uv](https://docs.astral.sh/uv/), no installation is required. `uvx` downloads and runs the tool in one go:
+
+```bash
+uvx sqlite2duckdb source.db target.db
+```
+
+To keep it around:
+
+```bash
+uv tool install sqlite2duckdb
+```
+
+Or with pip:
+
+```bash
+pip install sqlite2duckdb
+```
+
+## Usage
+
+### As a command line
+
+```
+usage: sqlite2duckdb [-f] <sqlite_path> <duckdb_path>
+
+Convert Sqlite database to Duckdb database
+
+positional arguments:
+  sqlite_path    sqlite file path
+  duckdb_path    duckdb file path
+
+options:
+  -h, --help     show this help message and exit
+  -f, --force    overwrite the duckdb file if it already exists
+  -q, --quiet    only report errors
+  --verbose      report every step
+  -v, --version  show program's version number and exit
+```
+
+### Examples
+
+```bash
+uvx sqlite2duckdb source.db target.db
+uvx sqlite2duckdb --force source.db target.db   # overwrite target.db without asking
+```
+
+### From python
+
+```python
+from sqlite2duckdb import sqlite_to_duckdb
+
+result = sqlite_to_duckdb("source.sqlite", "target.duckdb")
+print(result.tables, result.views, result.elapsed)
+```
+
+## What is converted
+
+| | |
+|---|---|
+| Tables and data | ✅ |
+| Primary keys, NOT NULL and UNIQUE constraints | ✅ |
+| Indexes | ✅ |
+| Views | ✅ best effort |
+| FOREIGN KEY and CHECK constraints | ❌ |
+
+A view whose SQL uses something duckdb has no equivalent for (`MATCH`, `julianday()`) is
+skipped with a warning instead of failing the conversion.
+
+FOREIGN KEY and CHECK are not copied, though not for lack of support: duckdb accepts and
+enforces both in `CREATE TABLE`. It checks foreign keys row by row, so a self-referencing
+table cannot be bulk loaded, and there is no `ALTER TABLE ADD CONSTRAINT` to add them once
+the data is in.
+
+
+## Contributing
+
+The project uses [uv](https://docs.astral.sh/uv/) for everything:
+
+```bash
+make dev     # uv sync — installs duckdb plus the test deps (pytest, faker, ruff)
+make test    # uv run pytest
+make lint    # uv run ruff check . && uv run ruff format --check .
+make build   # uv build
+make publish # uv publish (PyPI trusted publishing, also run on tags by CI)
+```
+
+### See also
+
+- [Harlequin](https://github.com/tconbeer/harlequin): A nice duckdb IDE for your terminal

sqlite2duckdb-0.5.0/pyproject.toml

@@ -0,0 +1,60 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+
+[project]
+name = "sqlite2duckdb"
+version = "0.5.0"
+authors = [{name="Sacha Schutz", email="sacha.schutz@pm.me"}]
+description = "A tool to convert sqlite database to duckdb database"
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["sqlite", "duckdb", "database", "olap", "oltp"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    'duckdb >= 1.1.0'
+]
+
+
+[project.urls]
+Homepage = "https://github.com/dridk/sqlite2duckdb"
+Issues = "https://github.com/dridk/sqlite2duckdb/issues"
+
+[project.scripts]
+sqlite2duckdb = "sqlite2duckdb.__main__:main_cli"
+
+[dependency-groups]
+dev = [
+    "pytest >= 7.0",
+    "faker",
+    "ruff",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["sqlite2duckdb"]
+
+# Allow list rather than a deny list, so that anything new landing in the repo
+# stays out of the distribution unless it is explicitly wanted.
+[tool.hatch.build.targets.sdist]
+include = ["sqlite2duckdb", "tests", "README.md"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py39"
+
+[tool.ruff.lint]
+extend-select = ["I", "UP"]

sqlite2duckdb-0.5.0/sqlite2duckdb/__init__.py

@@ -0,0 +1,14 @@
+import importlib.metadata
+
+from sqlite2duckdb.sqlite_to_duckdb import ConversionResult, sqlite_to_duckdb
+
+try:
+    __version__ = importlib.metadata.version("sqlite2duckdb")
+except importlib.metadata.PackageNotFoundError:
+    # Running from a source checkout that was never installed.
+    __version__ = "0.0.0.dev0"
+
+# Deprecated alias, kept so that existing imports keep working.
+__VERSION__ = __version__
+
+__all__ = ["ConversionResult", "__version__", "sqlite_to_duckdb"]

sqlite2duckdb-0.5.0/sqlite2duckdb/__main__.py

@@ -0,0 +1,76 @@
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+
+from sqlite2duckdb import __version__, sqlite_to_duckdb
+
+
+def main_cli() -> int:
+    parser = argparse.ArgumentParser(
+        prog="sqlite2duckdb",
+        description="Convert Sqlite database to Duckdb database",
+        usage="sqlite2duckdb [-f] <sqlite_path> <duckdb_path>",
+    )
+
+    parser.add_argument("sqlite_path", type=str, help="sqlite file path")
+    parser.add_argument("duckdb_path", type=str, help="duckdb file path")
+    parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        help="overwrite the duckdb file if it already exists",
+    )
+    parser.add_argument("-q", "--quiet", action="store_true", help="only report errors")
+    parser.add_argument("--verbose", action="store_true", help="report every step")
+    parser.add_argument(
+        "-v", "--version", action="version", version=f"sqlite2duckdb {__version__}"
+    )
+
+    args = parser.parse_args()
+
+    if args.quiet:
+        level = logging.WARNING
+    elif args.verbose:
+        level = logging.DEBUG
+    else:
+        level = logging.INFO
+    # Progress goes to stderr so that stdout stays free for pipelines.
+    logging.basicConfig(level=level, format="%(message)s", stream=sys.stderr)
+
+    overwrite = args.force
+    if not overwrite and os.path.exists(args.duckdb_path):
+        if not sys.stdin.isatty():
+            print(
+                f"{args.duckdb_path} already exists. Use --force to overwrite it.",
+                file=sys.stderr,
+            )
+            return 1
+        try:
+            answer = (
+                input(
+                    f"{args.duckdb_path} already exists. do you want to delete this file ? (yes/no): "
+                )
+                .strip()
+                .lower()
+            )
+        except EOFError:
+            print(f"{args.duckdb_path} already exists.", file=sys.stderr)
+            return 1
+        if answer not in ("yes", "y"):
+            return 1
+        overwrite = True
+
+    try:
+        sqlite_to_duckdb(args.sqlite_path, args.duckdb_path, overwrite=overwrite)
+    except (FileNotFoundError, FileExistsError) as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main_cli())

sqlite2duckdb-0.5.0/sqlite2duckdb/sqlite_to_duckdb.py

@@ -0,0 +1,267 @@
+from __future__ import annotations
+
+import contextlib
+import logging
+import os
+import sqlite3
+import time
+from dataclasses import dataclass
+
+import duckdb
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ConversionResult:
+    """What a successful conversion produced."""
+
+    target: str
+    tables: int
+    elapsed: float
+    """Wall clock duration of the conversion, in seconds."""
+    views: int = 0
+
+
+def _quote_identifier(name: str) -> str:
+    return '"' + name.replace('"', '""') + '"'
+
+
+def _format_duration(seconds: float) -> str:
+    if seconds < 1:
+        return f"{seconds * 1000:.2f} ms"
+    return f"{seconds:.2f} s"
+
+
+def _remove_quietly(path: str) -> None:
+    for candidate in (path, path + ".wal"):
+        try:
+            os.remove(candidate)
+        except OSError:
+            pass
+
+
+def _brackets_to_quotes(sql: str) -> str:
+    """Rewrite sqlite's [bracket] identifiers into standard "double quoted" ones.
+
+    Quoted runs are copied verbatim so that a `[` inside a string literal survives.
+    """
+
+    out = []
+    index = 0
+    length = len(sql)
+
+    while index < length:
+        char = sql[index]
+
+        if char in "'\"`":
+            end = index + 1
+            while end < length:
+                if sql[end] == char:
+                    if end + 1 < length and sql[end + 1] == char:
+                        end += 2
+                        continue
+                    break
+                end += 1
+            out.append(sql[index : end + 1])
+            index = end + 1
+
+        elif char == "[":
+            end = sql.find("]", index)
+            if end == -1:
+                out.append(char)
+                index += 1
+            else:
+                out.append(_quote_identifier(sql[index + 1 : end]))
+                index = end + 1
+
+        else:
+            out.append(char)
+            index += 1
+
+    return "".join(out)
+
+
+def _copy_indexes(conn: duckdb.DuckDBPyConnection, sqlite_path: str) -> None:
+    """Recreate the source indexes, translating sqlite quoting on the way."""
+
+    with contextlib.closing(sqlite3.connect(sqlite_path)) as source:
+        indexes = source.execute(
+            "SELECT name, sql FROM sqlite_master WHERE type = 'index' AND sql IS NOT NULL"
+        ).fetchall()
+
+    for name, sql in indexes:
+        try:
+            conn.sql(_brackets_to_quotes(sql))
+        except duckdb.Error as error:
+            logger.warning("Could not recreate index %s: %s", name, error)
+
+
+def _copy_unique_constraints(
+    conn: duckdb.DuckDBPyConnection, sqlite_path: str, table_names: list[str]
+) -> None:
+    """Replay the UNIQUE constraints that sqlite records without any SQL.
+
+    A column or table level UNIQUE becomes an autoindex whose sqlite_master row
+    has a NULL sql, so _copy_indexes cannot see it. Duckdb has no ALTER TABLE ADD
+    CONSTRAINT either, so a unique index is how the guarantee is carried over.
+    """
+
+    with contextlib.closing(sqlite3.connect(sqlite_path)) as source:
+        for table in table_names:
+            indexes = source.execute(
+                f"PRAGMA index_list({_quote_identifier(table)})"
+            ).fetchall()
+
+            for _, index_name, unique, origin, _partial in indexes:
+                # 'c' indexes carry their own SQL and are handled by _copy_indexes,
+                # and 'pk' is already part of the table DDL.
+                if not unique or origin != "u":
+                    continue
+
+                columns = [
+                    row[2]
+                    for row in source.execute(
+                        f"PRAGMA index_info({_quote_identifier(index_name)})"
+                    ).fetchall()
+                ]
+                if any(column is None for column in columns):
+                    logger.warning(
+                        "Skipping unique index %s: it is built on an expression",
+                        index_name,
+                    )
+                    continue
+
+                targets = ", ".join(_quote_identifier(column) for column in columns)
+                try:
+                    conn.sql(
+                        f"CREATE UNIQUE INDEX {_quote_identifier(index_name)} "
+                        f"ON {_quote_identifier(table)} ({targets})"
+                    )
+                except duckdb.Error as error:
+                    logger.warning(
+                        "Could not recreate unique index %s: %s", index_name, error
+                    )
+
+
+def _copy_views(conn: duckdb.DuckDBPyConnection, sqlite_path: str) -> int:
+    """Recreate the source views, and return how many made it across."""
+
+    with contextlib.closing(sqlite3.connect(sqlite_path)) as source:
+        views = source.execute(
+            "SELECT name, sql FROM sqlite_master WHERE type = 'view' AND sql IS NOT NULL"
+        ).fetchall()
+
+    pending = [(name, _brackets_to_quotes(sql)) for name, sql in views]
+    errors: dict[str, duckdb.Error] = {}
+    created = 0
+
+    # A view can sit on top of another one and sqlite_master does not guarantee
+    # dependency order, so keep retrying while a pass still makes progress.
+    while pending:
+        failed = []
+        for name, statement in pending:
+            try:
+                conn.sql(statement)
+            except duckdb.Error as error:
+                errors[name] = error
+                failed.append((name, statement))
+            else:
+                created += 1
+
+        if len(failed) == len(pending):
+            break
+        pending = failed
+
+    for name, _ in pending:
+        logger.warning("Could not recreate view %s: %s", name, errors[name])
+
+    return created
+
+
+def _copy_tables(
+    conn: duckdb.DuckDBPyConnection, tables: list[tuple[str, str]]
+) -> None:
+    """Recreate each table then fill it from the attached database."""
+
+    for table_name, ddl in tables:
+        # This DDL is regenerated by duckdb from the attached catalog, so unlike
+        # sqlite's own it always parses, and it still carries the primary keys
+        # and the NOT NULL constraints.
+        conn.sql(ddl)
+        quoted = _quote_identifier(table_name)
+        conn.sql(f"INSERT INTO {quoted} SELECT * FROM __other.{quoted}")
+
+
+def sqlite_to_duckdb(
+    sqlite_db: str | os.PathLike[str],
+    duck_db: str | os.PathLike[str],
+    *,
+    overwrite: bool = False,
+) -> ConversionResult:
+    """Copy a sqlite database into a new duckdb database.
+
+    Tables, data, views, primary keys, NOT NULL and UNIQUE constraints and
+    indexes are copied. FOREIGN KEY and CHECK constraints are not: duckdb checks
+    foreign keys row by row, so a self-referencing table cannot be bulk loaded,
+    and there is no ALTER TABLE ADD CONSTRAINT to add them once the data is in.
+
+    A view duckdb cannot bind is skipped with a warning rather than failing the
+    whole conversion.
+
+    Raises FileNotFoundError if `sqlite_db` does not exist, and FileExistsError
+    if `duck_db` already exists and `overwrite` is False.
+    """
+
+    sqlite_path = os.fspath(sqlite_db)
+    duck_path = os.fspath(duck_db)
+
+    if not os.path.exists(sqlite_path):
+        raise FileNotFoundError(f"File {sqlite_path} doesn't exist")
+
+    if os.path.exists(duck_path):
+        if not overwrite:
+            raise FileExistsError(f"Database {duck_path} already exists")
+        _remove_quietly(duck_path)
+
+    logger.info("Create %s database", duck_path)
+    start_time = time.perf_counter()
+
+    conn = duckdb.connect(duck_path)
+    try:
+        ## Install sqlite
+        conn.sql("INSTALL sqlite; LOAD sqlite;")
+
+        # Bound parameters are not allowed in ATTACH, so escape the quotes ourselves
+        source_path = sqlite_path.replace("'", "''")
+        conn.sql(f"ATTACH '{source_path}' AS __other (TYPE SQLITE, READ_ONLY)")
+
+        ## Get sqlite Names, along with the CREATE statement duckdb derives for them
+        tables = conn.sql(
+            "SELECT table_name, sql FROM duckdb_tables WHERE database_name = '__other'"
+        ).fetchall()
+        logger.info("%d table(s) found", len(tables))
+
+        _copy_tables(conn, tables)
+        _copy_indexes(conn, sqlite_path)
+        _copy_unique_constraints(conn, sqlite_path, [name for name, _ in tables])
+        views = _copy_views(conn, sqlite_path)
+        if views:
+            logger.info("%d view(s) copied", views)
+
+        conn.sql("DETACH __other")
+    except BaseException:
+        # Never leave a half-written database behind: it would trip the
+        # "already exists" guard on the next run.
+        conn.close()
+        _remove_quietly(duck_path)
+        raise
+
+    conn.close()
+
+    elapsed = time.perf_counter() - start_time
+    logger.info("Done in %s !", _format_duration(elapsed))
+
+    return ConversionResult(
+        target=duck_path, tables=len(tables), elapsed=elapsed, views=views
+    )