sqlite-chronicle 0.2__tar.gz → 0.3__tar.gz

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/PKG-INFO

@@ -1,18 +1,19 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: sqlite-chronicle
-Version: 0.2
+Version: 0.3
 Summary: Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite
 Author: Simon Willison
-License: Apache-2.0
+License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/simonw/sqlite-chronicle
 Project-URL: Changelog, https://github.com/simonw/sqlite-chronicle/releases
 Project-URL: Issues, https://github.com/simonw/sqlite-chronicle/issues
 Project-URL: CI, https://github.com/simonw/sqlite-chronicle/actions
-Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Database
+Classifier: Programming Language :: SQL
 Description-Content-Type: text/markdown
 Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
-Requires-Dist: sqlite-utils; extra == "test"
+Requires-Dist: sqlite-utils>=4.0a0; extra == "test"
 
 # sqlite-chronicle
 
@@ -29,25 +30,38 @@ Use triggers to track when rows in a SQLite table were updated or deleted
 pip install sqlite-chronicle
 ```
 
-## enable_chronicle(conn, table_name)
+## Command-line interface
+
+You can enable chronicle for specific tables in a SQLite database using the command-line interface, passing in one or more table names:
+
+```bash
+python -m sqlite_chronicle database.db table_1 table_2
+```
+
+## Python API
+
+This package exposes two Python functions for configuring and using chronicle tables:
+
+### enable_chronicle(conn, table_name)
 
 This module provides a function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
 
 1. Checks if a `_chronicle_{table_name}` table exists already. If so, it does nothing. Otherwise...
-2. Creates that table, with the same primary key columns as the original table plus integer columns `added_ms`, `updated_ms`, `version` and `deleted`
-3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `added_ms` and `updated_ms` to the current timestamp in milliseconds, and `version` column that starts at 1 and increments for each subsequent row
+2. Creates that table, with the same primary key columns as the original table plus integer columns `__added_ms`, `__updated_ms`, `__version` and `__deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `__added_ms` and `__updated_ms` to the current timestamp in milliseconds, and `__version` column that starts at 1 and increments for each subsequent row
 4. Sets up three triggers on the table:
-  - An after insert trigger, which creates a new row in the chronicle table, sets `added_ms` and `updated_ms` to the current time and increments the `version`
-  - An after update trigger, which updates the `updated_ms` timestamp and also updates any primary keys if they have changed (likely extremely rare) plus increments the `version`
-  - An after delete trigger, which updates the `updated_ms`, increments the `version` and places a `1` in the `deleted` column
 
-The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+  - An AFTER INSERT trigger, which creates a new row in the chronicle table, sets `__added_ms` and `__updated_ms` to the current time and sets the `__version` to one higher than the current maximum version for that table
+  - An AFTER UPDATE trigger, which updates the `__updated_ms` timestamp and increments the `__version` - but only if at least one column in the row has changed
+  - An AFTER DELETE trigger, which updates the `__updated_ms`, increments the `__version` and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not exist or if it does not have a single or compound primary key, 
 
-Note that the `version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(version)` + 1 for that entire table.
+Note that the `__version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(__version)` + 1 for that entire table.
 
 The end result is a chronicle table that looks something like this:
 
-|  id |    added_ms  | updated_ms | version | deleted |
+|  id |    __added_ms  | __updated_ms | __version | __deleted |
 |-----|---------------|---------|--------|---------|
 |  47 | 1694408890954 | 1694408890954 | 2 |      0 |
 |  48 | 1694408874863 | 1694408874863 | 3 |      1 |
@@ -55,7 +69,7 @@ The end result is a chronicle table that looks something like this:
 |   2 | 1694408825192 | 1694408825192 | 5 |      0 |
 |   3 | 1694408825192 | 1694408825192 | 6 |      0 |
 
-## updates_since(conn, table_name, since=None, batch_size=1000)
+### updates_since(conn, table_name, since=None, batch_size=1000)
 
 The `sqlite_chronicle.updates_since()` function returns a generator over a list of `Change` objects.
 
@@ -75,7 +89,7 @@ Change(
     updated_ms=1701836971223,
     version=5,
     row={'id': 5, 'name': 'Simon'},
-    deleted=0
+    deleted=False
 )
 ```
 A `Change` is a dataclass with the following properties:
@@ -91,6 +105,11 @@ Any time you call this you should track the last `version` number that you see,
 
 Note that if a row had multiple updates in between calls to this function you will still only see one `Change` object for that row - the `updated_ms` and `version` will reflect the most recent update.
 
+## Implementation notes
+
+- If you run `INSERT OR REPLACE INTO ...` and update an existing record in a way that does not change any of the fields, this system will still treat that record as if it has been updated. Use `INSERT ... ON CONFLICT SET` upserts instead to avoid this problem.
+- Updates to columns that are part of a primary key for the record is not currently supported.
+
 ## Potential applications
 
 Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked" - by looking at the `version` column which has an index to make it fast to answer that question.

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/README.md

@@ -13,25 +13,38 @@ Use triggers to track when rows in a SQLite table were updated or deleted
 pip install sqlite-chronicle
 ```
 
-## enable_chronicle(conn, table_name)
+## Command-line interface
+
+You can enable chronicle for specific tables in a SQLite database using the command-line interface, passing in one or more table names:
+
+```bash
+python -m sqlite_chronicle database.db table_1 table_2
+```
+
+## Python API
+
+This package exposes two Python functions for configuring and using chronicle tables:
+
+### enable_chronicle(conn, table_name)
 
 This module provides a function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
 
 1. Checks if a `_chronicle_{table_name}` table exists already. If so, it does nothing. Otherwise...
-2. Creates that table, with the same primary key columns as the original table plus integer columns `added_ms`, `updated_ms`, `version` and `deleted`
-3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `added_ms` and `updated_ms` to the current timestamp in milliseconds, and `version` column that starts at 1 and increments for each subsequent row
+2. Creates that table, with the same primary key columns as the original table plus integer columns `__added_ms`, `__updated_ms`, `__version` and `__deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `__added_ms` and `__updated_ms` to the current timestamp in milliseconds, and `__version` column that starts at 1 and increments for each subsequent row
 4. Sets up three triggers on the table:
-  - An after insert trigger, which creates a new row in the chronicle table, sets `added_ms` and `updated_ms` to the current time and increments the `version`
-  - An after update trigger, which updates the `updated_ms` timestamp and also updates any primary keys if they have changed (likely extremely rare) plus increments the `version`
-  - An after delete trigger, which updates the `updated_ms`, increments the `version` and places a `1` in the `deleted` column
 
-The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+  - An AFTER INSERT trigger, which creates a new row in the chronicle table, sets `__added_ms` and `__updated_ms` to the current time and sets the `__version` to one higher than the current maximum version for that table
+  - An AFTER UPDATE trigger, which updates the `__updated_ms` timestamp and increments the `__version` - but only if at least one column in the row has changed
+  - An AFTER DELETE trigger, which updates the `__updated_ms`, increments the `__version` and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not exist or if it does not have a single or compound primary key, 
 
-Note that the `version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(version)` + 1 for that entire table.
+Note that the `__version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(__version)` + 1 for that entire table.
 
 The end result is a chronicle table that looks something like this:
 
-|  id |    added_ms  | updated_ms | version | deleted |
+|  id |    __added_ms  | __updated_ms | __version | __deleted |
 |-----|---------------|---------|--------|---------|
 |  47 | 1694408890954 | 1694408890954 | 2 |      0 |
 |  48 | 1694408874863 | 1694408874863 | 3 |      1 |
@@ -39,7 +52,7 @@ The end result is a chronicle table that looks something like this:
 |   2 | 1694408825192 | 1694408825192 | 5 |      0 |
 |   3 | 1694408825192 | 1694408825192 | 6 |      0 |
 
-## updates_since(conn, table_name, since=None, batch_size=1000)
+### updates_since(conn, table_name, since=None, batch_size=1000)
 
 The `sqlite_chronicle.updates_since()` function returns a generator over a list of `Change` objects.
 
@@ -59,7 +72,7 @@ Change(
     updated_ms=1701836971223,
     version=5,
     row={'id': 5, 'name': 'Simon'},
-    deleted=0
+    deleted=False
 )
 ```
 A `Change` is a dataclass with the following properties:
@@ -75,6 +88,11 @@ Any time you call this you should track the last `version` number that you see,
 
 Note that if a row had multiple updates in between calls to this function you will still only see one `Change` object for that row - the `updated_ms` and `version` will reflect the most recent update.
 
+## Implementation notes
+
+- If you run `INSERT OR REPLACE INTO ...` and update an existing record in a way that does not change any of the fields, this system will still treat that record as if it has been updated. Use `INSERT ... ON CONFLICT SET` upserts instead to avoid this problem.
+- Updates to columns that are part of a primary key for the record is not currently supported.
+
 ## Potential applications
 
 Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked" - by looking at the `version` column which has an index to make it fast to answer that question.

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/pyproject.toml

@@ -1,12 +1,13 @@
 [project]
 name = "sqlite-chronicle"
-version = "0.2"
+version = "0.3"
 description = "Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite"
 readme = "README.md"
 authors = [{name = "Simon Willison"}]
-license = {text = "Apache-2.0"}
+license = "Apache-2.0"
 classifiers = [
-    "License :: OSI Approved :: Apache Software License"
+    "Topic :: Database",
+    "Programming Language :: SQL"
 ]
 
 [project.urls]
@@ -16,4 +17,11 @@ Issues = "https://github.com/simonw/sqlite-chronicle/issues"
 CI = "https://github.com/simonw/sqlite-chronicle/actions"
 
 [project.optional-dependencies]
-test = ["pytest", "sqlite-utils"]
+test = ["pytest", "sqlite-utils>=4.0a0"]
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+py-modules = ["sqlite_chronicle"]

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/sqlite_chronicle.egg-info/PKG-INFO

@@ -1,18 +1,19 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: sqlite-chronicle
-Version: 0.2
+Version: 0.3
 Summary: Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite
 Author: Simon Willison
-License: Apache-2.0
+License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/simonw/sqlite-chronicle
 Project-URL: Changelog, https://github.com/simonw/sqlite-chronicle/releases
 Project-URL: Issues, https://github.com/simonw/sqlite-chronicle/issues
 Project-URL: CI, https://github.com/simonw/sqlite-chronicle/actions
-Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Database
+Classifier: Programming Language :: SQL
 Description-Content-Type: text/markdown
 Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
-Requires-Dist: sqlite-utils; extra == "test"
+Requires-Dist: sqlite-utils>=4.0a0; extra == "test"
 
 # sqlite-chronicle
 
@@ -29,25 +30,38 @@ Use triggers to track when rows in a SQLite table were updated or deleted
 pip install sqlite-chronicle
 ```
 
-## enable_chronicle(conn, table_name)
+## Command-line interface
+
+You can enable chronicle for specific tables in a SQLite database using the command-line interface, passing in one or more table names:
+
+```bash
+python -m sqlite_chronicle database.db table_1 table_2
+```
+
+## Python API
+
+This package exposes two Python functions for configuring and using chronicle tables:
+
+### enable_chronicle(conn, table_name)
 
 This module provides a function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
 
 1. Checks if a `_chronicle_{table_name}` table exists already. If so, it does nothing. Otherwise...
-2. Creates that table, with the same primary key columns as the original table plus integer columns `added_ms`, `updated_ms`, `version` and `deleted`
-3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `added_ms` and `updated_ms` to the current timestamp in milliseconds, and `version` column that starts at 1 and increments for each subsequent row
+2. Creates that table, with the same primary key columns as the original table plus integer columns `__added_ms`, `__updated_ms`, `__version` and `__deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `__added_ms` and `__updated_ms` to the current timestamp in milliseconds, and `__version` column that starts at 1 and increments for each subsequent row
 4. Sets up three triggers on the table:
-  - An after insert trigger, which creates a new row in the chronicle table, sets `added_ms` and `updated_ms` to the current time and increments the `version`
-  - An after update trigger, which updates the `updated_ms` timestamp and also updates any primary keys if they have changed (likely extremely rare) plus increments the `version`
-  - An after delete trigger, which updates the `updated_ms`, increments the `version` and places a `1` in the `deleted` column
 
-The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+  - An AFTER INSERT trigger, which creates a new row in the chronicle table, sets `__added_ms` and `__updated_ms` to the current time and sets the `__version` to one higher than the current maximum version for that table
+  - An AFTER UPDATE trigger, which updates the `__updated_ms` timestamp and increments the `__version` - but only if at least one column in the row has changed
+  - An AFTER DELETE trigger, which updates the `__updated_ms`, increments the `__version` and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not exist or if it does not have a single or compound primary key, 
 
-Note that the `version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(version)` + 1 for that entire table.
+Note that the `__version` for a table is a globally incrementing number, so every time it is set it will be set to the current `max(__version)` + 1 for that entire table.
 
 The end result is a chronicle table that looks something like this:
 
-|  id |    added_ms  | updated_ms | version | deleted |
+|  id |    __added_ms  | __updated_ms | __version | __deleted |
 |-----|---------------|---------|--------|---------|
 |  47 | 1694408890954 | 1694408890954 | 2 |      0 |
 |  48 | 1694408874863 | 1694408874863 | 3 |      1 |
@@ -55,7 +69,7 @@ The end result is a chronicle table that looks something like this:
 |   2 | 1694408825192 | 1694408825192 | 5 |      0 |
 |   3 | 1694408825192 | 1694408825192 | 6 |      0 |
 
-## updates_since(conn, table_name, since=None, batch_size=1000)
+### updates_since(conn, table_name, since=None, batch_size=1000)
 
 The `sqlite_chronicle.updates_since()` function returns a generator over a list of `Change` objects.
 
@@ -75,7 +89,7 @@ Change(
     updated_ms=1701836971223,
     version=5,
     row={'id': 5, 'name': 'Simon'},
-    deleted=0
+    deleted=False
 )
 ```
 A `Change` is a dataclass with the following properties:
@@ -91,6 +105,11 @@ Any time you call this you should track the last `version` number that you see,
 
 Note that if a row had multiple updates in between calls to this function you will still only see one `Change` object for that row - the `updated_ms` and `version` will reflect the most recent update.
 
+## Implementation notes
+
+- If you run `INSERT OR REPLACE INTO ...` and update an existing record in a way that does not change any of the fields, this system will still treat that record as if it has been updated. Use `INSERT ... ON CONFLICT SET` upserts instead to avoid this problem.
+- Updates to columns that are part of a primary key for the record is not currently supported.
+
 ## Potential applications
 
 Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked" - by looking at the `version` column which has an index to make it fast to answer that question.

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/sqlite_chronicle.egg-info/SOURCES.txt

@@ -6,5 +6,6 @@ sqlite_chronicle.egg-info/SOURCES.txt
 sqlite_chronicle.egg-info/dependency_links.txt
 sqlite_chronicle.egg-info/requires.txt
 sqlite_chronicle.egg-info/top_level.txt
+tests/test_cli.py
 tests/test_sqlite_chronicle.py
 tests/test_updates_since.py

sqlite_chronicle-0.3/sqlite_chronicle.egg-info/requires.txt

@@ -0,0 +1,4 @@
+
+[test]
+pytest
+sqlite-utils>=4.0a0

sqlite_chronicle-0.3/sqlite_chronicle.py

@@ -0,0 +1,309 @@
+import dataclasses
+import sqlite3
+import textwrap
+from typing import Generator, Optional, List
+
+
+class ChronicleError(Exception):
+    pass
+
+
+@dataclasses.dataclass
+class Change:
+    pks: tuple
+    added_ms: int
+    updated_ms: int
+    version: int
+    row: dict
+    deleted: bool
+
+
+def enable_chronicle(conn: sqlite3.Connection, table_name: str) -> None:
+    """
+    Turn on chronicle tracking for `table_name`.
+
+    - Creates _chronicle_<table> (PK cols + __added_ms, __updated_ms, __version, __deleted)
+    - Populates that table with one row per existing row in the original
+    - AFTER INSERT trigger
+    - AFTER UPDATE trigger (WHEN any OLD<>NEW)
+    - AFTER DELETE trigger
+
+    Requires client code to use UPSERT (INSERT…ON CONFLICT DO UPDATE) for upserts rather
+    than INSERT OR REPLACE, since INSERT OR REPLACE will be incorrectly tracked.
+    """
+    cursor = conn.cursor()
+
+    # If chronicle table exists already, do nothing
+    chronicle_table = f"_chronicle_{table_name}"
+    cursor.execute(
+        "SELECT name FROM sqlite_master WHERE type='table' AND name=?",
+        (chronicle_table,),
+    )
+    if cursor.fetchone():
+        return
+
+    # Gather table schema info
+    cursor.execute(f'PRAGMA table_info("{table_name}")')
+    table_info = cursor.fetchall()
+
+    # Error if no such table
+    if not table_info:
+        raise ChronicleError(f"Table {table_name!r} does not exist")
+
+    # Identify primary key columns and non-PK columns
+    primary_key_columns = [(row[1], row[2]) for row in table_info if row[5]]
+    if not primary_key_columns:
+        raise ChronicleError(f"{table_name!r} has no PRIMARY KEY")
+    non_pk_columns = [row[1] for row in table_info if not row[5]]
+    primary_key_names = [col for col, _ in primary_key_columns]
+
+    # SQL expressions for timestamps and versioning
+    current_timestamp_expr = (
+        "CAST((julianday('now') - 2440587.5) * 86400 * 1000 AS INTEGER)"
+    )
+    next_version_expr = (
+        f'COALESCE((SELECT MAX(__version) FROM "{chronicle_table}"), 0) + 1'
+    )
+
+    # Build trigger WHEN condition: any non-PK column changed
+    if non_pk_columns:
+        update_condition = " OR ".join(
+            f'OLD."{col}" IS NOT NEW."{col}"' for col in non_pk_columns
+        )
+    else:
+        # no non-PK columns → treat any update as a change
+        update_condition = "1"
+
+    # Build PK matching clause for WHERE conditions
+    primary_key_match_clause = " AND ".join(
+        f'"{col}" = NEW."{col}"' for col in primary_key_names
+    )
+
+    # Collect all SQL statements to execute
+    sql_statements: List[str] = []
+
+    # 1) Create chronicle table
+    pk_definitions = ", ".join(
+        f'"{col}" {col_type}' for col, col_type in primary_key_columns
+    )
+    pk_constraint = ", ".join(f'"{col}"' for col in primary_key_names)
+    sql_statements.append(
+        textwrap.dedent(
+            f"""
+            CREATE TABLE "{chronicle_table}" (
+              {pk_definitions},
+              __added_ms   INTEGER,
+              __updated_ms INTEGER,
+              __version    INTEGER,
+              __deleted    INTEGER DEFAULT 0,
+              PRIMARY KEY({pk_constraint})
+            )
+        """
+        ).strip()
+    )
+    sql_statements.append(
+        textwrap.dedent(
+            f"""
+            CREATE INDEX "{chronicle_table}__version_idx"
+              ON "{chronicle_table}"(__version);
+        """
+        ).strip()
+    )
+
+    # 2) Seed chronicle table with existing rows
+    version_expr = (
+        f"ROW_NUMBER() OVER (ORDER BY "
+        + ", ".join(f'"{col}"' for col in primary_key_names)
+        + ")"
+    )
+
+    cols_insert = (
+        ", ".join(f'"{col}"' for col in primary_key_names)
+        + ", __added_ms, __updated_ms, __version, __deleted"
+    )
+    cols_select = (
+        ", ".join(f'"{col}"' for col in primary_key_names)
+        + f", {current_timestamp_expr} AS __added_ms"
+        + f", {current_timestamp_expr} AS __updated_ms"
+        + f", {version_expr} AS __version"
+        + ", 0 AS __deleted"
+    )
+
+    sql_statements.append(
+        f'INSERT INTO "{chronicle_table}" ({cols_insert})\n'
+        f" SELECT {cols_select}\n"
+        f'   FROM "{table_name}";'
+    )
+
+    # 3) AFTER INSERT trigger
+    sql_statements.append(
+        textwrap.dedent(
+            f"""
+            CREATE TRIGGER "chronicle_{table_name}_ai"
+            AFTER INSERT ON "{table_name}"
+            FOR EACH ROW
+            BEGIN
+              INSERT OR IGNORE INTO "{chronicle_table}" (
+                {', '.join(f'"{col}"' for col in primary_key_names)},
+                __added_ms, __updated_ms, __version, __deleted
+              )
+              VALUES (
+                {', '.join(f'NEW."{col}"' for col in primary_key_names)},
+                {current_timestamp_expr}, {current_timestamp_expr},
+                {next_version_expr}, 0
+              );
+            END;
+        """
+        ).strip()
+    )
+
+    # 4) AFTER UPDATE trigger (only if real change)
+    sql_statements.append(
+        textwrap.dedent(
+            f"""
+            CREATE TRIGGER "chronicle_{table_name}_au"
+            AFTER UPDATE ON "{table_name}"
+            FOR EACH ROW
+            WHEN {update_condition}
+            BEGIN
+              UPDATE "{chronicle_table}"
+                SET __updated_ms = {current_timestamp_expr},
+                    __version    = {next_version_expr}
+              WHERE {primary_key_match_clause};
+            END;
+        """
+        ).strip()
+    )
+
+    # 5) AFTER DELETE trigger
+    pk_old_match = " AND ".join(f'"{col}" = OLD."{col}"' for col in primary_key_names)
+    sql_statements.append(
+        textwrap.dedent(
+            f"""
+            CREATE TRIGGER "chronicle_{table_name}_ad"
+            AFTER DELETE ON "{table_name}"
+            FOR EACH ROW
+            BEGIN
+              UPDATE "{chronicle_table}"
+                SET __updated_ms = {current_timestamp_expr},
+                    __version    = {next_version_expr},
+                    __deleted    = 1
+              WHERE {pk_old_match};
+            END;
+        """
+        ).strip()
+    )
+
+    # Execute all statements within a transaction
+    with conn:
+        for stmt in sql_statements:
+            cursor.execute(stmt)
+
+
+def updates_since(
+    conn: sqlite3.Connection,
+    table_name: str,
+    since: Optional[int] = None,
+    batch_size: int = 1000,
+) -> Generator[Change, None, None]:
+    """
+    Yields Change(pks, added_ms, updated_ms, version, row, deleted)
+    for every chronicle.version > since, in ascending order.
+    """
+    cur = conn.cursor()
+    cur.row_factory = sqlite3.Row
+    if since is None:
+        since = 0
+
+    # find PK columns
+    cur.execute(f'PRAGMA table_info("{table_name}")')
+    cols = cur.fetchall()
+    pk_names = [c["name"] for c in cols if c["pk"]]
+    non_pk = [c["name"] for c in cols if not c["pk"]]
+
+    # build select
+    chron = f"_chronicle_{table_name}"
+    pks = ", ".join(f'c."{c}"' for c in pk_names)
+    originals = ", ".join(f't."{c}"' for c in non_pk)
+    select = ", ".join(
+        [pks, originals, "c.__added_ms", "c.__updated_ms", "c.__version", "c.__deleted"]
+    )
+    join = " AND ".join(f'c."{c}" = t."{c}"' for c in pk_names)
+
+    sql = textwrap.dedent(
+        f"""
+        SELECT {select}
+          FROM {chron} AS c
+          LEFT JOIN "{table_name}" AS t
+            ON {join}
+         WHERE c.__version > ?
+         ORDER BY c.__version
+         LIMIT {batch_size}
+        """
+    ).strip()
+
+    while True:
+        rows = cur.execute(sql, (since,)).fetchall()
+        if not rows:
+            break
+        for r in rows:
+            since = r["__version"]
+            # build row dict of original columns
+            row = {
+                c: r[c]
+                for c in r.keys()
+                if c not in ("__added_ms", "__updated_ms", "__version", "__deleted")
+            }
+            yield Change(
+                pks=tuple(r[c] for c in pk_names),
+                added_ms=r["__added_ms"],
+                updated_ms=r["__updated_ms"],
+                version=r["__version"],
+                row=row,
+                deleted=bool(r["__deleted"]),
+            )
+
+
+def cli_main(argv=None) -> int:
+    import argparse
+    import sys
+
+    parser = argparse.ArgumentParser(
+        prog="python -m sqlite_chronicle",
+        description="Enable chronicle tracking on one or more tables in an SQLite DB.",
+    )
+    parser.add_argument("db_path", help="Path to the SQLite database file")
+    parser.add_argument(
+        "tables",
+        nargs="+",
+        help="One or more table names to enable chronicle tracking on",
+    )
+
+    args = parser.parse_args(argv)
+
+    try:
+        conn = sqlite3.connect(args.db_path)
+    except sqlite3.Error as e:
+        print(f"ERROR: cannot open database {args.db_path!r}: {e}", file=sys.stderr)
+        return 1
+
+    any_error = False
+    for tbl in args.tables:
+        try:
+            enable_chronicle(conn, tbl)
+            print(f"- chronicle enabled on table {tbl!r}")
+        except ChronicleError as ce:
+            print(f"ERROR: {ce}", file=sys.stderr)
+            any_error = True
+        except sqlite3.Error as se:
+            print(f"SQL ERROR on table {tbl!r}: {se}", file=sys.stderr)
+            any_error = True
+
+    conn.close()
+    return 1 if any_error else 0
+
+
+if __name__ == "__main__":
+    import sys
+
+    sys.exit(cli_main())

sqlite_chronicle-0.3/tests/test_cli.py

@@ -0,0 +1,51 @@
+import sqlite3
+from sqlite_chronicle import cli_main
+
+
+def test_cli_main_success(tmp_path, capsys):
+    # Setup a simple DB file with a table with PRIMARY KEY
+    db_path = tmp_path / "test.db"
+    conn = sqlite3.connect(str(db_path))
+    conn.execute("CREATE TABLE t1 (id INTEGER PRIMARY KEY, val TEXT)")
+    conn.execute("INSERT INTO t1 (val) VALUES (?)", ("foo",))
+    conn.commit()
+    conn.close()
+
+    # Enable chronicle on the table
+    exit_code = cli_main([str(db_path), "t1"])
+    captured = capsys.readouterr()
+    assert exit_code == 0
+    assert "chronicle enabled on table 't1'" in captured.out
+
+    # Enabling chronicle again on the same table should be no-op (success)
+    exit_code = cli_main([str(db_path), "t1"])
+    captured = capsys.readouterr()
+    assert exit_code == 0
+
+
+def test_cli_main_error_invalid_db(capsys):
+    # Test passing a non-existent DB path
+    fake_path = "/nonexistent/path/to.db"
+    exit_code = cli_main([fake_path, "t1"])
+    captured = capsys.readouterr()
+
+    assert exit_code == 1
+    assert f"ERROR: cannot open database '{fake_path}'" in captured.err
+
+
+def test_cli_main_bad_table(tmp_path, capsys):
+    # Create a table and then drop it to cause SQL error during processing
+    db_path = tmp_path / "test.db"
+    conn = sqlite3.connect(str(db_path))
+    conn.execute("CREATE TABLE t1 (id INTEGER PRIMARY KEY, val TEXT)")
+    conn.commit()
+    # Drop t1 so enable_chronicle fails with SQL error
+    conn.execute("DROP TABLE t1")
+    conn.commit()
+    conn.close()
+
+    exit_code = cli_main([str(db_path), "t1"])
+    captured = capsys.readouterr()
+
+    assert exit_code == 1
+    assert "ERROR: Table 't1' does not exist" in captured.err

sqlite_chronicle-0.3/tests/test_sqlite_chronicle.py

@@ -0,0 +1,143 @@
+import pytest
+import sqlite_utils
+from sqlite_chronicle import enable_chronicle
+import time
+from unittest.mock import ANY
+
+
+@pytest.mark.parametrize("table_name", ("dogs", "dogs and stuff", "weird.table.name"))
+@pytest.mark.parametrize("pks", (["id"], ["id", "name"]))
+def test_enable_chronicle(table_name, pks):
+    chronicle_table = f"_chronicle_{table_name}"
+    db = sqlite_utils.Database(memory=True)
+    db[table_name].insert_all(
+        [
+            {"id": 1, "name": "Cleo", "color": "black"},
+        ],
+        pk=pks[0] if len(pks) == 1 else pks,
+    )
+    enable_chronicle(db.conn, table_name)
+    db[table_name].insert({"id": 2, "name": "Pancakes", "color": "corgi"})
+    # It should have the same primary keys
+    assert db[chronicle_table].pks == pks
+    # Should also have updated_ms and deleted columns
+    assert set(db[chronicle_table].columns_dict.keys()) == set(
+        pks + ["__added_ms", "__updated_ms", "__version", "__deleted"]
+    )
+    # With an index
+    assert db[chronicle_table].indexes[0].columns == ["__version"]
+    if pks == ["id"]:
+        expected = [
+            {
+                "id": 1,
+                "__added_ms": ANY,
+                "__updated_ms": ANY,
+                "__version": 1,
+                "__deleted": 0,
+            },
+            {
+                "id": 2,
+                "__added_ms": ANY,
+                "__updated_ms": ANY,
+                "__version": 2,
+                "__deleted": 0,
+            },
+        ]
+    else:
+        expected = [
+            {
+                "id": 1,
+                "name": "Cleo",
+                "__added_ms": ANY,
+                "__updated_ms": ANY,
+                "__version": 1,
+                "__deleted": 0,
+            },
+            {
+                "id": 2,
+                "name": "Pancakes",
+                "__added_ms": ANY,
+                "__updated_ms": ANY,
+                "__version": 2,
+                "__deleted": 0,
+            },
+        ]
+    rows = list(db[chronicle_table].rows)
+    assert rows == expected
+    for row in rows:
+        assert row["__added_ms"] != 0
+        assert row["__updated_ms"] != 0
+        assert row["__added_ms"] == row["__updated_ms"]
+    # Running it again should do nothing because table exists
+    enable_chronicle(db.conn, table_name)
+    # Insert a row
+    db[table_name].insert({"id": 3, "name": "Mango", "color": "orange"})
+    get_by = 3 if pks == ["id"] else (3, "Mango")
+    row = db[chronicle_table].get(get_by)
+    if pks == ["id"]:
+        assert row == {
+            "id": 3,
+            "__added_ms": ANY,
+            "__updated_ms": ANY,
+            "__version": 3,
+            "__deleted": 0,
+        }
+
+    else:
+        assert row == {
+            "id": 3,
+            "name": "Mango",
+            "__added_ms": ANY,
+            "__updated_ms": ANY,
+            "__version": 3,
+            "__deleted": 0,
+        }
+
+    version = db[chronicle_table].get(get_by)["__version"]
+    updated_ms = db[chronicle_table].get(get_by)["__updated_ms"]
+    time.sleep(0.01)
+    # Update a row
+    db[table_name].update(get_by, {"color": "mango"})
+    assert db[chronicle_table].get(get_by)["__version"] > version
+    assert db[chronicle_table].get(get_by)["__updated_ms"] > updated_ms
+    # Delete a row
+    assert db[table_name].count == 3
+    time.sleep(0.01)
+    db[table_name].delete(get_by)
+    assert db[table_name].count == 2
+    assert db[chronicle_table].get(get_by)["__deleted"] == 1
+    new_version = db[chronicle_table].get(get_by)["__version"]
+    assert new_version > version
+
+
+@pytest.mark.parametrize("pks", (["foo"], ["foo", "bar"]))
+def test_enable_chronicle_alternative_primary_keys(pks):
+    db = sqlite_utils.Database(memory=True)
+    db["dogs"].insert({"foo": 1, "bar": 2, "name": "Cleo", "color": "black"}, pk=pks)
+    enable_chronicle(db.conn, "dogs")
+    assert db["_chronicle_dogs"].pks == pks
+
+
+def test_upsert():
+    db = sqlite_utils.Database(memory=True)
+    dogs = db.table("dogs", pk="id").create(
+        {"id": int, "name": str, "color": str}, pk="id"
+    )
+    enable_chronicle(db.conn, "dogs")
+    dogs.insert({"id": 1, "name": "Cleo", "color": "black"})
+    dogs.upsert({"id": 2, "name": "Pancakes", "color": "corgi"})
+
+    def chronicle_rows():
+        return list(
+            db.query("select id, __version as version from _chronicle_dogs order by id")
+        )
+
+    assert chronicle_rows() == [{"id": 1, "version": 1}, {"id": 2, "version": 2}]
+
+    # Upsert that should update the row
+    dogs.upsert({"id": 1, "name": "Cleo", "color": "brown"})
+    assert chronicle_rows() == [{"id": 1, "version": 3}, {"id": 2, "version": 2}]
+
+    # Upsert that should be a no-op
+    dogs.upsert({"id": 1, "name": "Cleo", "color": "brown"})
+    assert chronicle_rows() == [{"id": 1, "version": 3}, {"id": 2, "version": 2}]

{sqlite-chronicle-0.2 → sqlite_chronicle-0.3}/tests/test_updates_since.py

@@ -59,7 +59,7 @@ def test_updates_since(db):
                 "name": "The fate of the crew on the Mary Celeste",
                 "year": "1872",
             },
-            deleted=0,
+            deleted=False,
         ),
         Change(
             pks=(2,),
@@ -71,7 +71,7 @@ def test_updates_since(db):
                 "name": "The disappearance of the Amber Room",
                 "year": "1941",
             },
-            deleted=0,
+            deleted=False,
         ),
     ]
     last = changes[-1].version
@@ -84,7 +84,7 @@ def test_updates_since(db):
             updated_ms=ANY,
             version=3,
             row={"id": 3, "name": "The lost city of Atlantis", "year": "360 BC"},
-            deleted=0,
+            deleted=False,
         )
     ]
     last2 = new_changes[-1].version
@@ -101,7 +101,7 @@ def test_updates_since(db):
                 "name": "The fate of the crew on the Mary Celeste",
                 "year": "unknown",
             },
-            deleted=0,
+            deleted=False,
         ),
         Change(
             pks=(3,),
@@ -109,7 +109,7 @@ def test_updates_since(db):
             updated_ms=ANY,
             version=5,
             row={"id": 3, "name": "The lost city of Atlantis", "year": "unknown"},
-            deleted=0,
+            deleted=False,
         ),
     ]
     last3 = new_changes2[-1].version
@@ -122,7 +122,7 @@ def test_updates_since(db):
             updated_ms=ANY,
             version=6,
             row={"id": 2, "name": None, "year": None},
-            deleted=1,
+            deleted=True,
         )
     ]
 
@@ -144,12 +144,12 @@ def test_updates_since_more_rows_than_batch_size_in_an_update():
         ({"id": i, "name": "Name {}".format(i)} for i in range(201)), pk="id"
     )
     enable_chronicle(db.conn, "mysteries")
-    since_id = db.execute("select max(version) from _chronicle_mysteries").fetchone()[0]
+    max_v = db.execute("select max(__version) from _chronicle_mysteries").fetchone()[0]
     # Update them all in one go
     with db.conn:
         db.execute("update mysteries set name = 'Updated'")
 
-    changes = list(updates_since(db.conn, "mysteries", batch_size=100, since=since_id))
+    changes = list(updates_since(db.conn, "mysteries", batch_size=100, since=max_v))
     assert len(changes) == 201
     # Each change should have a different version
     assert len(set(c.version for c in changes)) == 201

sqlite-chronicle-0.2/sqlite_chronicle.egg-info/requires.txt

@@ -1,4 +0,0 @@
-
-[test]
-pytest
-sqlite-utils

sqlite-chronicle-0.2/sqlite_chronicle.py

@@ -1,196 +0,0 @@
-import dataclasses
-import sqlite3
-import textwrap
-from typing import Generator, Optional
-
-
-class ChronicleError(Exception):
-    pass
-
-
-@dataclasses.dataclass
-class Change:
-    pks: tuple
-    added_ms: int
-    updated_ms: int
-    version: int
-    row: dict
-    deleted: bool
-
-
-def enable_chronicle(conn: sqlite3.Connection, table_name: str):
-    c = conn.cursor()
-
-    # Check if the _chronicle_ table exists
-    c.execute(
-        f"SELECT name FROM sqlite_master WHERE type='table' AND name='_chronicle_{table_name}';"
-    )
-    if c.fetchone():
-        return
-
-    # Determine primary key columns and their types
-    c.execute(f'PRAGMA table_info("{table_name}");')
-    primary_key_columns = [(row[1], row[2]) for row in c.fetchall() if row[5]]
-    if not primary_key_columns:
-        raise ChronicleError(f"Table {table_name} has no primary keys")
-
-    # Create the _chronicle_ table
-    pk_def = ", ".join(
-        [f'"{col_name}" {col_type}' for col_name, col_type in primary_key_columns]
-    )
-
-    current_time_expr = "CAST((julianday('now') - 2440587.5) * 86400 * 1000 AS INTEGER)"
-    next_version_expr = (
-        f'COALESCE((SELECT MAX(version) FROM "_chronicle_{table_name}"), 0) + 1'
-    )
-
-    with conn:
-        c.execute(
-            f"""
-            CREATE TABLE "_chronicle_{table_name}" (
-                {pk_def},
-                added_ms INTEGER,
-                updated_ms INTEGER,
-                version INTEGER DEFAULT 0,
-                deleted INTEGER DEFAULT 0,
-                PRIMARY KEY ({', '.join([f'"{col[0]}"' for col in primary_key_columns])})
-            );
-            """
-        )
-
-        # Index on version column
-        c.execute(
-            f"CREATE INDEX '_chronicle_{table_name}_version' ON '_chronicle_{table_name}' (version);"
-        )
-
-        # Populate the _chronicle_ table with existing rows from the original table
-        c.execute(
-            f"""
-            INSERT INTO "_chronicle_{table_name}" (
-                {', '.join([col[0] for col in primary_key_columns])},
-                added_ms,
-                updated_ms,
-                version
-            )
-            SELECT
-                {', '.join([f'"{col[0]}"' for col in primary_key_columns])},
-                {current_time_expr},
-                {current_time_expr},
-                ROW_NUMBER() OVER (ORDER BY id)
-            FROM "{table_name}";
-            """
-        )
-
-        # Create the after insert trigger
-        after_insert_sql = textwrap.dedent(
-            f"""
-        CREATE TRIGGER "_chronicle_{table_name}_ai"
-        AFTER INSERT ON "{table_name}"
-        FOR EACH ROW
-        BEGIN
-            INSERT INTO "_chronicle_{table_name}" ({', '.join([f'"{col[0]}"' for col in primary_key_columns])}, added_ms, updated_ms, version)
-            VALUES ({', '.join(['NEW.' + f'"{col[0]}"' for col in primary_key_columns])}, {current_time_expr}, {current_time_expr}, {next_version_expr});
-        END;
-        """
-        )
-        c.execute(after_insert_sql)
-
-        # Create the after update trigger
-        c.execute(
-            f"""
-            CREATE TRIGGER "_chronicle_{table_name}_au"
-            AFTER UPDATE ON "{table_name}"
-            FOR EACH ROW
-            BEGIN
-                UPDATE "_chronicle_{table_name}"
-                SET updated_ms = {current_time_expr},
-                    version = {next_version_expr},
-                    {', '.join([f'"{col[0]}" = NEW."{col[0]}"' for col in primary_key_columns])}
-                WHERE { ' AND '.join([f'"{col[0]}" = OLD."{col[0]}"' for col in primary_key_columns]) };
-            END;
-            """
-        )
-
-        # Create the after delete trigger
-        c.execute(
-            f"""
-            CREATE TRIGGER "_chronicle_{table_name}_ad"
-            AFTER DELETE ON "{table_name}"
-            FOR EACH ROW
-            BEGIN
-                UPDATE "_chronicle_{table_name}"
-                SET updated_ms = {current_time_expr},
-                    version = {next_version_expr},
-                    deleted = 1
-                WHERE { ' AND '.join([f'"{col[0]}" = OLD."{col[0]}"' for col in primary_key_columns]) };
-            END;
-            """
-        )
-
-
-def updates_since(
-    conn: sqlite3.Connection,
-    table_name: str,
-    since: Optional[int] = None,
-    batch_size: int = 1000,
-) -> Generator[Change, None, None]:
-    cursor = conn.cursor()
-    cursor.row_factory = sqlite3.Row
-
-    if since is None:
-        since = 0
-
-    # Find primary keys
-    cursor.execute(f"PRAGMA table_info({table_name})")
-    columns = cursor.fetchall()
-    primary_keys = [col["name"] for col in columns if col["pk"]]
-
-    # Create the join_on clause based on primary keys
-    join_conditions = [f'chronicle."{pk}" = t."{pk}"' for pk in primary_keys]
-    join_on = " AND ".join(join_conditions)
-
-    # Select clause is primary keys from chronicle table, then columns from original table
-    select_clause = ", ".join(
-        [f"chronicle.{pk}" for pk in primary_keys]
-        + [f"t.{col['name']}" for col in columns if col not in primary_keys]
-        + [
-            "chronicle.added_ms as __chronicle_added_ms",
-            "chronicle.updated_ms as __chronicle_updated_ms",
-            "chronicle.version as __chronicle_version",
-            "CASE WHEN t.id IS NULL THEN 1 ELSE 0 END AS __chronicle_deleted",
-        ]
-    )
-
-    # Paginate through ordered by version batch_size at a time
-    while True:
-        sql = textwrap.dedent(
-            f"""
-            SELECT {select_clause}
-            FROM "_chronicle_{table_name}" chronicle
-            LEFT JOIN "{table_name}" t ON {join_on}
-            WHERE chronicle.version > ?
-            ORDER BY chronicle.version
-            LIMIT {batch_size}
-            """
-        )
-        rows = cursor.execute(sql, (since,)).fetchall()
-
-        if not rows:
-            break
-
-        for row in rows:
-            # Need row without __chronicle_ columns
-            row_without = dict(
-                (k, row[k]) for k in row.keys() if not k.startswith("__chronicle_")
-            )
-            added_ms = row["__chronicle_added_ms"]
-            updated_ms = row["__chronicle_updated_ms"]
-            since = row["__chronicle_version"]
-            yield Change(
-                pks=tuple(row[pk] for pk in primary_keys),
-                added_ms=added_ms,
-                updated_ms=updated_ms,
-                version=since,
-                row=row_without,
-                deleted=row["__chronicle_deleted"],
-            )

sqlite-chronicle-0.2/tests/test_sqlite_chronicle.py

@@ -1,117 +0,0 @@
-import pytest
-import sqlite_utils
-from sqlite_chronicle import enable_chronicle
-import time
-from unittest.mock import ANY
-
-
-@pytest.mark.parametrize("table_name", ("dogs", "dogs and stuff", "weird.table.name"))
-@pytest.mark.parametrize("pks", (["id"], ["id", "name"]))
-def test_enable_chronicle(table_name, pks):
-    chronicle_table = f"_chronicle_{table_name}"
-    db = sqlite_utils.Database(memory=True)
-    db[table_name].insert_all(
-        [
-            {"id": 1, "name": "Cleo", "color": "black"},
-            {"id": 2, "name": "Pancakes", "color": "corgi"},
-        ],
-        pk=pks[0] if len(pks) == 1 else pks,
-    )
-    enable_chronicle(db.conn, table_name)
-    # It should have the same primary keys
-    assert db[chronicle_table].pks == pks
-    # Should also have updated_ms and deleted columns
-    assert set(db[chronicle_table].columns_dict.keys()) == set(
-        pks + ["added_ms", "updated_ms", "version", "deleted"]
-    )
-    # With an index
-    assert db[chronicle_table].indexes[0].columns == ["version"]
-    if pks == ["id"]:
-        expected = [
-            {
-                "id": 1,
-                "added_ms": ANY,
-                "updated_ms": ANY,
-                "version": 1,
-                "deleted": 0,
-            },
-            {
-                "id": 2,
-                "added_ms": ANY,
-                "updated_ms": ANY,
-                "version": 2,
-                "deleted": 0,
-            },
-        ]
-    else:
-        expected = [
-            {
-                "id": 1,
-                "name": "Cleo",
-                "added_ms": ANY,
-                "updated_ms": ANY,
-                "version": 1,
-                "deleted": 0,
-            },
-            {
-                "id": 2,
-                "name": "Pancakes",
-                "added_ms": ANY,
-                "updated_ms": ANY,
-                "version": 2,
-                "deleted": 0,
-            },
-        ]
-    rows = list(db[chronicle_table].rows)
-    assert rows == expected
-    # Running it again should do nothing because table exists
-    enable_chronicle(db.conn, table_name)
-    # Insert a row
-    db[table_name].insert({"id": 3, "name": "Mango", "color": "orange"})
-    get_by = 3 if pks == ["id"] else (3, "Mango")
-    row = db[chronicle_table].get(get_by)
-    if pks == ["id"]:
-        assert row == {
-            "id": 3,
-            "added_ms": ANY,
-            "updated_ms": ANY,
-            "version": 3,
-            "deleted": 0,
-        }
-
-    else:
-        assert row == {
-            "id": 3,
-            "name": "Mango",
-            "added_ms": ANY,
-            "updated_ms": ANY,
-            "version": 3,
-            "deleted": 0,
-        }
-
-    version = db[chronicle_table].get(get_by)["updated_ms"]
-    time.sleep(0.01)
-    # Update a row
-    db[table_name].update(get_by, {"color": "mango"})
-    assert db[chronicle_table].get(get_by)["updated_ms"] > version
-    # Delete a row
-    assert db[table_name].count == 3
-    time.sleep(0.01)
-    db[table_name].delete(get_by)
-    assert db[table_name].count == 2
-    assert db[chronicle_table].get(get_by)["deleted"] == 1
-    new_version = db[chronicle_table].get(get_by)["updated_ms"]
-    assert new_version > version
-    # Now update a column that's part of the compound primary key
-    time.sleep(0.1)
-    if pks == ["id", "name"]:
-        db[table_name].update((2, "Pancakes"), {"name": "Pancakes the corgi"})
-        # This should have renamed the row in the chronicle table as well
-        renamed_row = db[chronicle_table].get((2, "Pancakes the corgi"))
-        assert renamed_row["updated_ms"] > version
-    else:
-        # Update single primary key
-        db[table_name].update(2, {"id": 4})
-        # This should have renamed the row in the chronicle table as well
-        renamed_row = db[chronicle_table].get(4)
-        assert renamed_row["updated_ms"] > version