sqlite-utils 3.38__tar.gz → 4.0a0__tar.gz

{sqlite_utils-3.38/sqlite_utils.egg-info → sqlite_utils-4.0a0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: sqlite-utils
-Version: 3.38
+Version: 4.0a0
 Summary: CLI tool and Python library for manipulating SQLite databases
 Home-page: https://github.com/simonw/sqlite-utils
 Author: Simon Willison
@@ -16,13 +16,12 @@ Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: End Users/Desktop
 Classifier: Topic :: Database
 Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: sqlite-fts4
@@ -52,8 +51,18 @@ Requires-Dist: types-pluggy; extra == "mypy"
 Requires-Dist: data-science-types; extra == "mypy"
 Provides-Extra: flake8
 Requires-Dist: flake8; extra == "flake8"
-Provides-Extra: tui
-Requires-Dist: trogon; extra == "tui"
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # sqlite-utils
 

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/docs/changelog.rst

@@ -4,6 +4,17 @@
  Changelog
 ===========
 
+.. _v4_0a0:
+
+4.0a0 (2025-05-08)
+------------------
+
+- Upsert operations now use SQLite's ``INSERT ... ON CONFLICT SET`` syntax on all SQLite versions later than 3.23.1. This is a very slight breaking change for apps that depend on the previous ``INSERT OR IGNORE`` followed by ``UPDATE`` behavior. (:issue:`652`)
+- Python library users can opt-in to the previous implementation by passing ``use_old_upsert=True`` to the ``Database()`` constructor, see :ref:`python_api_old_upsert`.
+- Dropped support for Python 3.8, added support for Python 3.13. (:issue:`646`)
+- ``sqlite-utils tui`` is now provided by the `sqlite-utils-tui <https://github.com/simonw/sqlite-utils-tui>`__ plugin. (:issue:`648`)
+- Test suite now also runs against SQLite 3.23.1, the last version (from 2018-04-10) before the new ``INSERT ... ON CONFLICT SET`` syntax was added. (:issue:`654`)
+
 .. _v3_38:
 
 3.38 (2024-11-23)
@@ -111,7 +122,7 @@ This release introduces a new :ref:`plugin system <plugins>`. Read more about th
 3.32 (2023-05-21)
 -----------------
 
-- New experimental ``sqlite-utils tui`` interface for interactively building command-line invocations, powered by `Trogon <https://github.com/Textualize/trogon>`__. This requires an optional dependency, installed using ``sqlite-utils install trogon``. There is a screenshot :ref:`in the documentation <cli_tui>`. (:issue:`545`)
+- New experimental ``sqlite-utils tui`` interface for interactively building command-line invocations, powered by `Trogon <https://github.com/Textualize/trogon>`__. This requires an optional dependency, installed using ``sqlite-utils install trogon``. (:issue:`545`)
 - ``sqlite-utils analyze-tables`` command (:ref:`documentation <cli_analyze_tables>`) now has a ``--common-limit 20`` option for changing the number of common/least-common values shown for each column. (:issue:`544`)
 - ``sqlite-utils analyze-tables --no-most`` and ``--no-least`` options for disabling calculation of most-common and least-common values.
 - If a column contains only ``null`` values, ``analyze-tables`` will no longer attempt to calculate the most common and least common values for that column. (:issue:`547`)

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/docs/cli-reference.rst

@@ -65,7 +65,6 @@ This page lists the ``--help`` for every ``sqlite-utils`` CLI sub-command.
         "create-spatial-index": "cli_spatialite_indexes",
         "install": "cli_install",
         "uninstall": "cli_uninstall",
-        "tui": "cli_tui",
     }
     commands.sort(key = lambda command: go_first.index(command) if command in go_first else 999)
     cog.out("\n")
@@ -1045,23 +1044,6 @@ disable-fts
       -h, --help             Show this message and exit.
 
 
-.. _cli_ref_tui:
-
-tui
-===
-
-See :ref:`cli_tui`.
-
-::
-
-    Usage: sqlite-utils tui [OPTIONS]
-
-      Open Textual TUI.
-
-    Options:
-      -h, --help  Show this message and exit.
-
-
 .. _cli_ref_optimize:
 
 optimize

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/docs/cli.rst

@@ -1128,7 +1128,7 @@ You can insert binary data into a BLOB column by first encoding it using base64
 Inserting newline-delimited JSON
 --------------------------------
 
-You can also import `newline-delimited JSON <http://ndjson.org/>`__ using the ``--nl`` option:
+You can also import newline-delimited JSON (see `JSON Lines <https://jsonlines.org/>`__) using the ``--nl`` option:
 
 .. code-block:: bash
 
@@ -1285,7 +1285,7 @@ You can set the ``SQLITE_UTILS_DETECT_TYPES`` environment variable if you want `
 
 If a CSV or TSV file includes empty cells, like this one:
 
-.. code-block:: csv
+::
 
     name,age,weight
     Cleo,6,
@@ -2782,29 +2782,3 @@ You can uninstall packages that were installed using ``sqlite-utils install`` wi
     sqlite-utils uninstall beautifulsoup4
 
 Use ``-y`` to skip the request for confirmation.
-
-.. _cli_tui:
-
-Experimental TUI
-================
-
-A TUI is a "text user interface" (or "terminal user interface") - a keyboard and mouse driven graphical interface running in your terminal.
-
-``sqlite-utils`` has experimental support for a TUI for building command-line invocations, built on top of the `Trogon <https://github.com/Textualize/trogon/>`__ TUI library.
-
-To enable this feature you will need to install the ``trogon`` dependency. You can do that like so:
-
-.. code-block:: bash
-
-    sqlite-utils install trogon
-
-Once installed, running the ``sqlite-utils tui`` command will launch the TUI interface:
-
-.. code-block:: bash
-
-    sqlite-utils tui
-
-You can then construct a command by selecting options from the menus, and execute it using ``Ctrl+R``.
-
-.. image:: _static/img/tui.png
-    :alt: A TUI interface for sqlite-utils - the left column shows a list of commands, while the right panel has a form for constructing arguments to the add-column command.

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/docs/python-api.rst

@@ -927,6 +927,13 @@ An ``upsert_all()`` method is also available, which behaves like ``insert_all()`
 .. note::
     ``.upsert()`` and ``.upsert_all()`` in sqlite-utils 1.x worked like ``.insert(..., replace=True)`` and ``.insert_all(..., replace=True)`` do in 2.x. See `issue #66 <https://github.com/simonw/sqlite-utils/issues/66>`__ for details of this change.
 
+.. _python_api_old_upsert:
+
+Alternative upserts using INSERT OR IGNORE
+------------------------------------------
+
+Upserts use ``INSERT INTO ... ON CONFLICT SET``. Prior to ``sqlite-utils 4.0`` these used a sequence of ``INSERT OR IGNORE`` followed by an ``UPDATE``. This older method is still used for SQLite 3.23.1 and earlier. You can force the older implementation by passing ``use_old_upsert=True`` to the ``Database()`` constructor.
+
 .. _python_api_convert:
 
 Converting data in columns
@@ -2711,7 +2718,7 @@ By default, the name of the Python function will be used as the name of the SQL
 
     print(db.execute('select rev("hello")').fetchone()[0])
 
-Python 3.8 added the ability to register `deterministic SQLite functions <https://sqlite.org/deterministic.html>`__, allowing you to indicate that a function will return the exact same result for any given inputs and hence allowing SQLite to apply some performance optimizations. You can mark a function as deterministic using ``deterministic=True``, like this:
+If a function will return the exact same result for any given inputs you can register it as a `deterministic SQLite function <https://sqlite.org/deterministic.html>`__ allowing SQLite to apply some performance optimizations:
 
 .. code-block:: python
 
@@ -2719,8 +2726,6 @@ Python 3.8 added the ability to register `deterministic SQLite functions <https:
     def reverse_string(s):
         return "".join(reversed(list(s)))
 
-If you run this on a version of Python prior to 3.8 your code will still work, but the ``deterministic=True`` parameter will be ignored.
-
 By default registering a function with the same name and number of arguments will have no effect - the ``Database`` instance keeps track of functions that have already been registered and skips registering them if ``@db.register_function`` is called a second time.
 
 If you want to deliberately replace the registered function with a new implementation, use the ``replace=True`` argument:

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
 import io
 import os
 
-VERSION = "3.38"
+VERSION = "4.0a0"
 
 
 def get_long_description():
@@ -50,7 +50,6 @@ setup(
             "data-science-types",
         ],
         "flake8": ["flake8"],
-        "tui": ["trogon"],
     },
     entry_points="""
         [console_scripts]
@@ -64,7 +63,7 @@ setup(
         "Issues": "https://github.com/simonw/sqlite-utils/issues",
         "CI": "https://github.com/simonw/sqlite-utils/actions",
     },
-    python_requires=">=3.8",
+    python_requires=">=3.9",
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Intended Audience :: Developers",
@@ -72,7 +71,6 @@ setup(
         "Intended Audience :: End Users/Desktop",
         "Topic :: Database",
         "License :: OSI Approved :: Apache Software License",
-        "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
         "Programming Language :: Python :: 3.11",

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/sqlite_utils/cli.py

@@ -35,11 +35,6 @@ from .utils import (
     TypeTracker,
 )
 
-try:
-    import trogon  # type: ignore
-except ImportError:
-    trogon = None
-
 
 CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"])
 
@@ -127,10 +122,6 @@ def cli():
     pass
 
 
-if trogon is not None:
-    cli = trogon.tui()(cli)
-
-
 @cli.command()
 @click.argument(
     "path",
@@ -1107,7 +1098,9 @@ def insert_upsert_implementation(
             if (
                 isinstance(e, OperationalError)
                 and e.args
-                and "has no column named" in e.args[0]
+                and (
+                    "has no column named" in e.args[0] or "no such column" in e.args[0]
+                )
             ):
                 raise click.ClickException(
                     "{}\n\nTry using --alter to add additional columns".format(

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/sqlite_utils/db.py

@@ -237,37 +237,30 @@ if pd:
 
 class AlterError(Exception):
     "Error altering table"
-    pass
 
 
 class NoObviousTable(Exception):
     "Could not tell which table this operation refers to"
-    pass
 
 
 class NoTable(Exception):
     "Specified table does not exist"
-    pass
 
 
 class BadPrimaryKey(Exception):
     "Table does not have a single obvious primary key"
-    pass
 
 
 class NotFoundError(Exception):
     "Record not found"
-    pass
 
 
 class PrimaryKeyRequired(Exception):
     "Primary key needs to be specified"
-    pass
 
 
 class InvalidColumns(Exception):
     "Specified columns do not exist"
-    pass
 
 
 class DescIndex(str):
@@ -311,6 +304,8 @@ class Database:
       ``sql, parameters`` every time a SQL query is executed
     :param use_counts_table: set to ``True`` to use a cached counts table, if available. See
       :ref:`python_api_cached_table_counts`
+    :param use_old_upsert: set to ``True`` to force the older upsert implementation. See
+      :ref:`python_api_old_upsert`
     :param strict: Apply STRICT mode to all created tables (unless overridden)
     """
 
@@ -327,10 +322,12 @@ class Database:
         tracer: Optional[Callable] = None,
         use_counts_table: bool = False,
         execute_plugins: bool = True,
+        use_old_upsert: bool = False,
         strict: bool = False,
     ):
         self.memory_name = None
         self.memory = False
+        self.use_old_upsert = use_old_upsert
         assert (filename_or_conn is not None and (not memory and not memory_name)) or (
             filename_or_conn is None and (memory or memory_name)
         ), "Either specify a filename_or_conn or pass memory=True"
@@ -678,16 +675,46 @@ class Database:
     @property
     def supports_strict(self) -> bool:
         "Does this database support STRICT mode?"
-        try:
+        if not hasattr(self, "_supports_strict"):
+            try:
+                table_name = "t{}".format(secrets.token_hex(16))
+                with self.conn:
+                    self.conn.execute(
+                        "create table {} (name text) strict".format(table_name)
+                    )
+                    self.conn.execute("drop table {}".format(table_name))
+                self._supports_strict = True
+            except Exception:
+                self._supports_strict = False
+        return self._supports_strict
+
+    @property
+    def supports_on_conflict(self) -> bool:
+        # SQLite's upsert is implemented as INSERT INTO ... ON CONFLICT DO ...
+        if not hasattr(self, "_supports_on_conflict"):
             table_name = "t{}".format(secrets.token_hex(16))
-            with self.conn:
-                self.conn.execute(
-                    "create table {} (name text) strict".format(table_name)
-                )
-                self.conn.execute("drop table {}".format(table_name))
-            return True
-        except Exception:
-            return False
+            try:
+                with self.conn:
+                    self.conn.execute(
+                        "create table {} (id integer primary key, name text)".format(
+                            table_name
+                        )
+                    )
+                    self.conn.execute(
+                        "insert into {} (id, name) values (1, 'one')".format(table_name)
+                    )
+                    self.conn.execute(
+                        (
+                            "insert into {} (id, name) values (1, 'two') "
+                            "on conflict do update set name = 'two'"
+                        ).format(table_name)
+                    )
+                    self._supports_on_conflict = True
+            except Exception:
+                self._supports_on_conflict = False
+            finally:
+                self.conn.execute("drop table if exists {}".format(table_name))
+        return self._supports_on_conflict
 
     @property
     def sqlite_version(self) -> Tuple[int, ...]:
@@ -2973,13 +3000,19 @@ class Table(Queryable):
         replace,
         ignore,
     ):
-        # values is the list of insert data that is passed to the
-        # .execute() method - but some of them may be replaced by
-        # new primary keys if we are extracting any columns.
-        values = []
+        """
+        Given a list ``chunk`` of records that should be written to *this* table,
+        return a list of ``(sql, parameters)`` 2-tuples which, when executed in
+        order, perform the desired INSERT / UPSERT / REPLACE operation.
+        """
         if hash_id_columns and hash_id is None:
             hash_id = "id"
+
         extracts = resolve_extracts(extracts)
+
+        # Build a row-list ready for executemany-style flattening
+        values = []
+
         for record in chunk:
             record_values = []
             for key in all_columns:
@@ -2999,76 +3032,103 @@ class Table(Queryable):
                 record_values.append(value)
             values.append(record_values)
 
-        queries_and_params = []
-        if upsert:
-            if isinstance(pk, str):
-                pks = [pk]
-            else:
-                pks = pk
-            self.last_pk = None
-            for record_values in values:
-                record = dict(zip(all_columns, record_values))
-                placeholders = list(pks)
-                # Need to populate not-null columns too, or INSERT OR IGNORE ignores
-                # them since it ignores the resulting integrity errors
-                if not_null:
-                    placeholders.extend(not_null)
-                sql = "INSERT OR IGNORE INTO [{table}]({cols}) VALUES({placeholders});".format(
-                    table=self.name,
-                    cols=", ".join(["[{}]".format(p) for p in placeholders]),
-                    placeholders=", ".join(["?" for p in placeholders]),
-                )
-                queries_and_params.append(
-                    (sql, [record[col] for col in pks] + ["" for _ in (not_null or [])])
-                )
-                # UPDATE [book] SET [name] = 'Programming' WHERE [id] = 1001;
-                set_cols = [col for col in all_columns if col not in pks]
-                if set_cols:
-                    sql2 = "UPDATE [{table}] SET {pairs} WHERE {wheres}".format(
-                        table=self.name,
-                        pairs=", ".join(
-                            "[{}] = {}".format(col, conversions.get(col, "?"))
-                            for col in set_cols
-                        ),
-                        wheres=" AND ".join("[{}] = ?".format(pk) for pk in pks),
-                    )
-                    queries_and_params.append(
-                        (
-                            sql2,
-                            [record[col] for col in set_cols]
-                            + [record[pk] for pk in pks],
+        columns_sql = ", ".join(f"[{c}]" for c in all_columns)
+        placeholder_expr = ", ".join(conversions.get(c, "?") for c in all_columns)
+        row_placeholders_sql = ", ".join(f"({placeholder_expr})" for _ in values)
+        flat_params = list(itertools.chain.from_iterable(values))
+
+        # replace=True mean INSERT OR REPLACE INTO
+        if replace:
+            sql = (
+                f"INSERT OR REPLACE INTO [{self.name}] "
+                f"({columns_sql}) VALUES {row_placeholders_sql}"
+            )
+            return [(sql, flat_params)]
+
+        # If not an upsert it's an INSERT, maybe with OR IGNORE
+        if not upsert:
+            or_ignore = ""
+            if ignore:
+                or_ignore = " OR IGNORE"
+            sql = (
+                f"INSERT{or_ignore} INTO [{self.name}] "
+                f"({columns_sql}) VALUES {row_placeholders_sql}"
+            )
+            return [(sql, flat_params)]
+
+        # Everything from here on is for upsert=True
+        pk_cols = [pk] if isinstance(pk, str) else list(pk)
+        non_pk_cols = [c for c in all_columns if c not in pk_cols]
+        conflict_sql = ", ".join(f"[{c}]" for c in pk_cols)
+
+        if self.db.supports_on_conflict and not self.db.use_old_upsert:
+            if non_pk_cols:
+                # DO UPDATE
+                assignments = []
+                for c in non_pk_cols:
+                    if c in conversions:
+                        assignments.append(
+                            f"[{c}] = {conversions[c].replace('?', f'excluded.[{c}]')}"
                         )
-                    )
-                # We can populate .last_pk right here
-                if num_records_processed == 1:
-                    self.last_pk = tuple(record[pk] for pk in pks)
-                    if len(self.last_pk) == 1:
-                        self.last_pk = self.last_pk[0]
+                    else:
+                        assignments.append(f"[{c}] = excluded.[{c}]")
+                do_clause = "DO UPDATE SET " + ", ".join(assignments)
+            else:
+                # All columns are in the PK – nothing to update.
+                do_clause = "DO NOTHING"
 
+            sql = (
+                f"INSERT INTO [{self.name}] ({columns_sql}) "
+                f"VALUES {row_placeholders_sql} "
+                f"ON CONFLICT({conflict_sql}) {do_clause}"
+            )
+            return [(sql, flat_params)]
+
+        # At this point we need compatibility UPSERT for SQLite < 3.24.0
+        # (INSERT OR IGNORE + second UPDATE stage)
+        queries_and_params = []
+        if isinstance(pk, str):
+            pks = [pk]
         else:
-            or_what = ""
-            if replace:
-                or_what = "OR REPLACE "
-            elif ignore:
-                or_what = "OR IGNORE "
-            sql = """
-                INSERT {or_what}INTO [{table}] ({columns}) VALUES {rows};
-            """.strip().format(
-                or_what=or_what,
+            pks = pk
+        self.last_pk = None
+        for record_values in values:
+            record = dict(zip(all_columns, record_values))
+            placeholders = list(pks)
+            # Need to populate not-null columns too, or INSERT OR IGNORE ignores
+            # them since it ignores the resulting integrity errors
+            if not_null:
+                placeholders.extend(not_null)
+            sql = "INSERT OR IGNORE INTO [{table}]({cols}) VALUES({placeholders});".format(
                 table=self.name,
-                columns=", ".join("[{}]".format(c) for c in all_columns),
-                rows=", ".join(
-                    "({placeholders})".format(
-                        placeholders=", ".join(
-                            [conversions.get(col, "?") for col in all_columns]
-                        )
-                    )
-                    for record in chunk
-                ),
+                cols=", ".join(["[{}]".format(p) for p in placeholders]),
+                placeholders=", ".join(["?" for p in placeholders]),
             )
-            flat_values = list(itertools.chain(*values))
-            queries_and_params = [(sql, flat_values)]
-
+            queries_and_params.append(
+                (sql, [record[col] for col in pks] + ["" for _ in (not_null or [])])
+            )
+            # UPDATE [book] SET [name] = 'Programming' WHERE [id] = 1001;
+            set_cols = [col for col in all_columns if col not in pks]
+            if set_cols:
+                sql2 = "UPDATE [{table}] SET {pairs} WHERE {wheres}".format(
+                    table=self.name,
+                    pairs=", ".join(
+                        "[{}] = {}".format(col, conversions.get(col, "?"))
+                        for col in set_cols
+                    ),
+                    wheres=" AND ".join("[{}] = ?".format(pk) for pk in pks),
+                )
+                queries_and_params.append(
+                    (
+                        sql2,
+                        [record[col] for col in set_cols] + [record[pk] for pk in pks],
+                    )
+                )
+            # We can populate .last_pk right here
+            if num_records_processed == 1:
+                self.last_pk = tuple(record[pk] for pk in pks)
+                if len(self.last_pk) == 1:
+                    self.last_pk = self.last_pk[0]
         return queries_and_params
 
     def insert_chunk(
@@ -3086,7 +3146,7 @@ class Table(Queryable):
         num_records_processed,
         replace,
         ignore,
-    ):
+    ) -> Optional[sqlite3.Cursor]:
         queries_and_params = self.build_insert_queries_and_params(
             extracts,
             chunk,
@@ -3101,9 +3161,8 @@ class Table(Queryable):
             replace,
             ignore,
         )
-
+        result = None
         with self.db.conn:
-            result = None
             for query, params in queries_and_params:
                 try:
                     result = self.db.execute(query, params)
@@ -3132,7 +3191,7 @@ class Table(Queryable):
                             ignore,
                         )
 
-                        self.insert_chunk(
+                        result = self.insert_chunk(
                             alter,
                             extracts,
                             second_half,
@@ -3150,20 +3209,7 @@ class Table(Queryable):
 
                     else:
                         raise
-            if num_records_processed == 1 and not upsert:
-                self.last_rowid = result.lastrowid
-                self.last_pk = self.last_rowid
-                # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened
-                if (hash_id or pk) and self.last_rowid:
-                    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
-                    if hash_id:
-                        self.last_pk = row[hash_id]
-                    elif isinstance(pk, str):
-                        self.last_pk = row[pk]
-                    else:
-                        self.last_pk = tuple(row[p] for p in pk)
-
-        return
+        return result
 
     def insert(
         self,
@@ -3203,7 +3249,7 @@ class Table(Queryable):
         :param not_null: Set of strings specifying columns that should be ``NOT NULL``.
         :param defaults: Dictionary specifying default values for specific columns.
         :param hash_id: Name of a column to create and use as a primary key, where the
-          value of thet primary key will be derived as a SHA1 hash of the other column values
+          value of that primary key will be derived as a SHA1 hash of the other column values
           in the record. ``hash_id="id"`` is a common column name used for this.
         :param alter: Boolean, should any missing columns be added automatically?
         :param ignore: Boolean, if a record already exists with this primary key, ignore this insert.
@@ -3283,6 +3329,7 @@ class Table(Queryable):
 
         if upsert and (not pk and not hash_id):
             raise PrimaryKeyRequired("upsert() requires a pk")
+
         assert not (hash_id and pk), "Use either pk= or hash_id="
         if hash_id_columns and (hash_id is None):
             hash_id = "id"
@@ -3314,6 +3361,7 @@ class Table(Queryable):
         self.last_pk = None
         if truncate and self.exists():
             self.db.execute("DELETE FROM [{}];".format(self.name))
+        result = None
         for chunk in chunks(itertools.chain([first_record], records), batch_size):
             chunk = list(chunk)
             num_records_processed += len(chunk)
@@ -3321,6 +3369,12 @@ class Table(Queryable):
                 if not self.exists():
                     # Use the first batch to derive the table names
                     column_types = suggest_column_types(chunk)
+                    if extracts:
+                        for col in extracts:
+                            if col in column_types:
+                                column_types[col] = (
+                                    int  # This will be an integer foreign key
+                                )
                     column_types.update(columns or {})
                     self.create(
                         column_types,
@@ -3348,7 +3402,7 @@ class Table(Queryable):
 
             first = False
 
-            self.insert_chunk(
+            result = self.insert_chunk(
                 alter,
                 extracts,
                 chunk,
@@ -3364,6 +3418,33 @@ class Table(Queryable):
                 ignore,
             )
 
+        # If we only handled a single row populate self.last_pk
+        if num_records_processed == 1:
+            # For an insert we need to use result.lastrowid
+            if not upsert and result is not None:
+                self.last_rowid = result.lastrowid
+                if (hash_id or pk) and self.last_rowid:
+                    # Set self.last_pk to the pk(s) for that rowid
+                    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
+                    if hash_id:
+                        self.last_pk = row[hash_id]
+                    elif isinstance(pk, str):
+                        self.last_pk = row[pk]
+                    else:
+                        self.last_pk = tuple(row[p] for p in pk)
+                else:
+                    self.last_pk = self.last_rowid
+            else:
+                # For an upsert use first_record from earlier
+                if hash_id:
+                    self.last_pk = hash_record(first_record, hash_id_columns)
+                else:
+                    self.last_pk = (
+                        first_record[pk]
+                        if isinstance(pk, str)
+                        else tuple(first_record[p] for p in pk)
+                    )
+
         if analyze:
             self.analyze()
 
@@ -3852,7 +3933,7 @@ def jsonify_if_needed(value):
 
 
 def resolve_extracts(
-    extracts: Optional[Union[Dict[str, str], List[str], Tuple[str]]]
+    extracts: Optional[Union[Dict[str, str], List[str], Tuple[str]]],
 ) -> dict:
     if extracts is None:
         extracts = {}

{sqlite_utils-3.38 → sqlite_utils-4.0a0/sqlite_utils.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: sqlite-utils
-Version: 3.38
+Version: 4.0a0
 Summary: CLI tool and Python library for manipulating SQLite databases
 Home-page: https://github.com/simonw/sqlite-utils
 Author: Simon Willison
@@ -16,13 +16,12 @@ Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: End Users/Desktop
 Classifier: Topic :: Database
 Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: sqlite-fts4
@@ -52,8 +51,18 @@ Requires-Dist: types-pluggy; extra == "mypy"
 Requires-Dist: data-science-types; extra == "mypy"
 Provides-Extra: flake8
 Requires-Dist: flake8; extra == "flake8"
-Provides-Extra: tui
-Requires-Dist: trogon; extra == "tui"
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # sqlite-utils
 

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/sqlite_utils.egg-info/requires.txt

@@ -29,6 +29,3 @@ pytest
 black>=24.1.1
 hypothesis
 cogapp
-
-[tui]
-trogon

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/tests/test_cli.py

@@ -916,7 +916,7 @@ def test_query_json_with_json_cols(db_path):
 
 @pytest.mark.parametrize(
     "content,is_binary",
-    [(b"\x00\x0Fbinary", True), ("this is text", False), (1, False), (1.5, False)],
+    [(b"\x00\x0fbinary", True), ("this is text", False), (1, False), (1.5, False)],
 )
 def test_query_raw(db_path, content, is_binary):
     Database(db_path)["files"].insert({"content": content})
@@ -931,7 +931,7 @@ def test_query_raw(db_path, content, is_binary):
 
 @pytest.mark.parametrize(
     "content,is_binary",
-    [(b"\x00\x0Fbinary", True), ("this is text", False), (1, False), (1.5, False)],
+    [(b"\x00\x0fbinary", True), ("this is text", False), (1, False), (1.5, False)],
 )
 def test_query_raw_lines(db_path, content, is_binary):
     Database(db_path)["files"].insert_all({"content": content} for _ in range(3))
@@ -1116,11 +1116,8 @@ def test_upsert_alter(db_path, tmpdir):
         cli.cli, ["upsert", db_path, "dogs", json_path, "--pk", "id"]
     )
     assert result.exit_code == 1
-    assert (
-        "Error: no such column: age\n\n"
-        "sql = UPDATE [dogs] SET [age] = ? WHERE [id] = ?\n"
-        "parameters = [5, 1]"
-    ) == result.output.strip()
+    # Could be one of two errors depending on SQLite version
+    assert ("Try using --alter to add additional columns") in result.output.strip()
     # Should succeed with --alter
     result = CliRunner().invoke(
         cli.cli, ["upsert", db_path, "dogs", json_path, "--pk", "id", "--alter"]
@@ -2248,7 +2245,7 @@ def test_integer_overflow_error(tmpdir):
     assert result.exit_code == 1
     assert result.output == (
         "Error: Python int too large to convert to SQLite INTEGER\n\n"
-        "sql = INSERT INTO [items] ([bignumber]) VALUES (?);\n"
+        "sql = INSERT INTO [items] ([bignumber]) VALUES (?)\n"
         "parameters = [34223049823094832094802398430298048240]\n"
     )
 

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/tests/test_create.py

@@ -173,14 +173,16 @@ def test_create_table_from_example_with_compound_primary_keys(fresh_db):
 @pytest.mark.parametrize(
     "method_name", ("insert", "upsert", "insert_all", "upsert_all")
 )
-def test_create_table_with_custom_columns(fresh_db, method_name):
-    table = fresh_db["dogs"]
+@pytest.mark.parametrize("use_old_upsert", (False, True))
+def test_create_table_with_custom_columns(method_name, use_old_upsert):
+    db = Database(memory=True, use_old_upsert=use_old_upsert)
+    table = db["dogs"]
     method = getattr(table, method_name)
     record = {"id": 1, "name": "Cleo", "age": "5"}
     if method_name.endswith("_all"):
         record = [record]
     method(record, pk="id", columns={"age": int, "weight": float})
-    assert ["dogs"] == fresh_db.table_names()
+    assert ["dogs"] == db.table_names()
     expected_columns = [
         {"name": "id", "type": "INTEGER"},
         {"name": "name", "type": "TEXT"},

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/tests/test_plugins.py

@@ -1,9 +1,19 @@
 from click.testing import CliRunner
 import click
 import importlib
+import pytest
 from sqlite_utils import cli, Database, hookimpl, plugins
 
 
+def _supports_pragma_function_list():
+    db = Database(memory=True)
+    try:
+        db.execute("select * from pragma_function_list()")
+    except Exception:
+        return False
+    return True
+
+
 def test_register_commands():
     importlib.reload(cli)
     assert plugins.get_plugins() == []
@@ -37,6 +47,10 @@ def test_register_commands():
         assert plugins.get_plugins() == []
 
 
+@pytest.mark.skipif(
+    not _supports_pragma_function_list(),
+    reason="Needs SQLite version that supports pragma_function_list()",
+)
 def test_prepare_connection():
     importlib.reload(cli)
     assert plugins.get_plugins() == []
@@ -54,7 +68,7 @@ def test_prepare_connection():
         return [
             row[0]
             for row in db.execute(
-                "select distinct name from pragma_function_list order by 1"
+                "select distinct name from pragma_function_list() order by 1"
             ).fetchall()
         ]
 

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/tests/test_tracer.py

@@ -18,7 +18,7 @@ def test_tracer():
         ("select name from sqlite_master where type = 'view'", None),
         ("CREATE TABLE [dogs] (\n   [name] TEXT\n);\n        ", None),
         ("select name from sqlite_master where type = 'view'", None),
-        ("INSERT INTO [dogs] ([name]) VALUES (?);", ["Cleopaws"]),
+        ("INSERT INTO [dogs] ([name]) VALUES (?)", ["Cleopaws"]),
         ("select name from sqlite_master where type = 'view'", None),
         (
             "CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n    [name],\n    content=[dogs]\n)",

{sqlite_utils-3.38 → sqlite_utils-4.0a0}/tests/test_upsert.py

@@ -1,9 +1,12 @@
 from sqlite_utils.db import PrimaryKeyRequired
+from sqlite_utils import Database
 import pytest
 
 
-def test_upsert(fresh_db):
-    table = fresh_db["table"]
+@pytest.mark.parametrize("use_old_upsert", (False, True))
+def test_upsert(use_old_upsert):
+    db = Database(memory=True, use_old_upsert=use_old_upsert)
+    table = db["table"]
     table.insert({"id": 1, "name": "Cleo"}, pk="id")
     table.upsert({"id": 1, "age": 5}, pk="id", alter=True)
     assert list(table.rows) == [{"id": 1, "name": "Cleo", "age": 5}]