sqlite-chronicle 0.1__tar.gz

sqlite-chronicle-0.1/PKG-INFO

@@ -0,0 +1,65 @@
+Metadata-Version: 2.1
+Name: sqlite-chronicle
+Version: 0.1
+Summary: Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite
+Author: Simon Willison
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/simonw/sqlite-chronicle
+Project-URL: Changelog, https://github.com/simonw/sqlite-chronicle/releases
+Project-URL: Issues, https://github.com/simonw/sqlite-chronicle/issues
+Project-URL: CI, https://github.com/simonw/sqlite-chronicle/actions
+Classifier: License :: OSI Approved :: Apache Software License
+Description-Content-Type: text/markdown
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: sqlite-utils; extra == "test"
+
+# sqlite-chronicle
+
+[![PyPI](https://img.shields.io/pypi/v/sqlite-chronicle.svg)](https://pypi.org/project/sqlite-chronicle/)
+[![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-chronicle?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-chronicle/releases)
+[![Tests](https://github.com/simonw/sqlite-chronicle/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-chronicle/actions?query=workflow%3ATest)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-chronicle/blob/main/LICENSE)
+
+Use triggers to track when rows in a SQLite table were updated or deleted
+
+## Installation
+
+```bash
+pip install sqlite-chronicle
+```
+
+## enable_chronicle(conn, table_name)
+
+This module provides a single function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
+
+1. Checks if a `_chronicle_{table_name}` table exists alreday. If so, it does nothing. Otherwise...
+2. Creates that table, with the same primary key columns as the original table plus integer columns `updated_ms` and `deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `updated_ms` to the current timestamp in milliseconds
+4. Sets up three triggers on the table:
+  - An after insert trigger, which creates a new row in the chronicle table and sets `updated_ms` to the current time
+  - An after update trigger, which updates that timestamp and also updates any primary keys if they have changed (likely extremely rare)
+  - An after delete trigger, which updates that timestamp and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+
+The end result is a chronicle table that looks something like this:
+
+|  id |    updated_ms | deleted |
+|-----|---------------|---------|
+|  47 | 1694408890954 |       0 |
+|  48 | 1694408874863 |       1 |
+|   1 | 1694408825192 |       0 |
+|   2 | 1694408825192 |       0 |
+|   3 | 1694408825192 |       0 |
+
+## Applications
+
+Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked".
+
+This has numerous potential applications, including:
+
+- Synchronization and replication: other databases can "subscribe" to tables, keeping track of when they last refreshed their copy and requesting just rows that changed since the last time - and deleting rows that have been marked as deleted.
+- Indexing: if you need to update an Elasticsearch index or a vector database embeddings index or similar you can run against just the records that changed since your last run - see also [The denormalized query engine design pattern](https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/)
+- Enrichments: [datasette-enrichments](https://github.com/datasette/datasette-enrichments) needs to to persist something that says "every address column should be geocoded" - then have an enrichment that runs every X seconds and looks for newly inserted or updated rows and enriches just those.
+- Showing people what has changed since their last visit - "52 rows have been updated and 16 deleted since yesterday" kind of thing.

sqlite-chronicle-0.1/README.md

@@ -0,0 +1,49 @@
+# sqlite-chronicle
+
+[![PyPI](https://img.shields.io/pypi/v/sqlite-chronicle.svg)](https://pypi.org/project/sqlite-chronicle/)
+[![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-chronicle?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-chronicle/releases)
+[![Tests](https://github.com/simonw/sqlite-chronicle/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-chronicle/actions?query=workflow%3ATest)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-chronicle/blob/main/LICENSE)
+
+Use triggers to track when rows in a SQLite table were updated or deleted
+
+## Installation
+
+```bash
+pip install sqlite-chronicle
+```
+
+## enable_chronicle(conn, table_name)
+
+This module provides a single function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
+
+1. Checks if a `_chronicle_{table_name}` table exists alreday. If so, it does nothing. Otherwise...
+2. Creates that table, with the same primary key columns as the original table plus integer columns `updated_ms` and `deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `updated_ms` to the current timestamp in milliseconds
+4. Sets up three triggers on the table:
+  - An after insert trigger, which creates a new row in the chronicle table and sets `updated_ms` to the current time
+  - An after update trigger, which updates that timestamp and also updates any primary keys if they have changed (likely extremely rare)
+  - An after delete trigger, which updates that timestamp and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+
+The end result is a chronicle table that looks something like this:
+
+|  id |    updated_ms | deleted |
+|-----|---------------|---------|
+|  47 | 1694408890954 |       0 |
+|  48 | 1694408874863 |       1 |
+|   1 | 1694408825192 |       0 |
+|   2 | 1694408825192 |       0 |
+|   3 | 1694408825192 |       0 |
+
+## Applications
+
+Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked".
+
+This has numerous potential applications, including:
+
+- Synchronization and replication: other databases can "subscribe" to tables, keeping track of when they last refreshed their copy and requesting just rows that changed since the last time - and deleting rows that have been marked as deleted.
+- Indexing: if you need to update an Elasticsearch index or a vector database embeddings index or similar you can run against just the records that changed since your last run - see also [The denormalized query engine design pattern](https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/)
+- Enrichments: [datasette-enrichments](https://github.com/datasette/datasette-enrichments) needs to to persist something that says "every address column should be geocoded" - then have an enrichment that runs every X seconds and looks for newly inserted or updated rows and enriches just those.
+- Showing people what has changed since their last visit - "52 rows have been updated and 16 deleted since yesterday" kind of thing.

sqlite-chronicle-0.1/pyproject.toml

@@ -0,0 +1,19 @@
+[project]
+name = "sqlite-chronicle"
+version = "0.1"
+description = "Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite"
+readme = "README.md"
+authors = [{name = "Simon Willison"}]
+license = {text = "Apache-2.0"}
+classifiers = [
+    "License :: OSI Approved :: Apache Software License"
+]
+
+[project.urls]
+Homepage = "https://github.com/simonw/sqlite-chronicle"
+Changelog = "https://github.com/simonw/sqlite-chronicle/releases"
+Issues = "https://github.com/simonw/sqlite-chronicle/issues"
+CI = "https://github.com/simonw/sqlite-chronicle/actions"
+
+[project.optional-dependencies]
+test = ["pytest", "sqlite-utils"]

sqlite-chronicle-0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sqlite-chronicle-0.1/sqlite_chronicle.egg-info/PKG-INFO

@@ -0,0 +1,65 @@
+Metadata-Version: 2.1
+Name: sqlite-chronicle
+Version: 0.1
+Summary: Use triggers to maintain a chronicle table of updated/deleted timestamps in SQLite
+Author: Simon Willison
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/simonw/sqlite-chronicle
+Project-URL: Changelog, https://github.com/simonw/sqlite-chronicle/releases
+Project-URL: Issues, https://github.com/simonw/sqlite-chronicle/issues
+Project-URL: CI, https://github.com/simonw/sqlite-chronicle/actions
+Classifier: License :: OSI Approved :: Apache Software License
+Description-Content-Type: text/markdown
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: sqlite-utils; extra == "test"
+
+# sqlite-chronicle
+
+[![PyPI](https://img.shields.io/pypi/v/sqlite-chronicle.svg)](https://pypi.org/project/sqlite-chronicle/)
+[![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-chronicle?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-chronicle/releases)
+[![Tests](https://github.com/simonw/sqlite-chronicle/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-chronicle/actions?query=workflow%3ATest)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-chronicle/blob/main/LICENSE)
+
+Use triggers to track when rows in a SQLite table were updated or deleted
+
+## Installation
+
+```bash
+pip install sqlite-chronicle
+```
+
+## enable_chronicle(conn, table_name)
+
+This module provides a single function: `sqlite_chronicle.enable_chronicle(conn, table_name)`, which does the following:
+
+1. Checks if a `_chronicle_{table_name}` table exists alreday. If so, it does nothing. Otherwise...
+2. Creates that table, with the same primary key columns as the original table plus integer columns `updated_ms` and `deleted`
+3. Creates a new row in the chronicle table corresponding to every row in the original table, setting `updated_ms` to the current timestamp in milliseconds
+4. Sets up three triggers on the table:
+  - An after insert trigger, which creates a new row in the chronicle table and sets `updated_ms` to the current time
+  - An after update trigger, which updates that timestamp and also updates any primary keys if they have changed (likely extremely rare)
+  - An after delete trigger, which updates that timestamp and places a `1` in the `deleted` column
+
+The function will raise a `sqlite_chronicle.ChronicleError` exception if the table does not have a single or compound primary key.
+
+The end result is a chronicle table that looks something like this:
+
+|  id |    updated_ms | deleted |
+|-----|---------------|---------|
+|  47 | 1694408890954 |       0 |
+|  48 | 1694408874863 |       1 |
+|   1 | 1694408825192 |       0 |
+|   2 | 1694408825192 |       0 |
+|   3 | 1694408825192 |       0 |
+
+## Applications
+
+Chronicle tables can be used to efficiently answer the question "what rows have been inserted, updated or deleted since I last checked".
+
+This has numerous potential applications, including:
+
+- Synchronization and replication: other databases can "subscribe" to tables, keeping track of when they last refreshed their copy and requesting just rows that changed since the last time - and deleting rows that have been marked as deleted.
+- Indexing: if you need to update an Elasticsearch index or a vector database embeddings index or similar you can run against just the records that changed since your last run - see also [The denormalized query engine design pattern](https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/)
+- Enrichments: [datasette-enrichments](https://github.com/datasette/datasette-enrichments) needs to to persist something that says "every address column should be geocoded" - then have an enrichment that runs every X seconds and looks for newly inserted or updated rows and enriches just those.
+- Showing people what has changed since their last visit - "52 rows have been updated and 16 deleted since yesterday" kind of thing.

sqlite-chronicle-0.1/sqlite_chronicle.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+README.md
+pyproject.toml
+sqlite_chronicle.py
+sqlite_chronicle.egg-info/PKG-INFO
+sqlite_chronicle.egg-info/SOURCES.txt
+sqlite_chronicle.egg-info/dependency_links.txt
+sqlite_chronicle.egg-info/requires.txt
+sqlite_chronicle.egg-info/top_level.txt
+tests/test_sqlite_chronicle.py

sqlite-chronicle-0.1/sqlite_chronicle.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sqlite-chronicle-0.1/sqlite_chronicle.egg-info/requires.txt

@@ -0,0 +1,4 @@
+
+[test]
+pytest
+sqlite-utils

sqlite-chronicle-0.1/sqlite_chronicle.egg-info/top_level.txt

@@ -0,0 +1 @@
+sqlite_chronicle

sqlite-chronicle-0.1/sqlite_chronicle.py

@@ -0,0 +1,108 @@
+import sqlite3
+import textwrap
+
+
+class ChronicleError(Exception):
+    pass
+
+
+def enable_chronicle(conn: sqlite3.Connection, table_name: str):
+    c = conn.cursor()
+
+    # Check if the _chronicle_ table exists
+    c.execute(
+        f"SELECT name FROM sqlite_master WHERE type='table' AND name='_chronicle_{table_name}';"
+    )
+    if c.fetchone():
+        return
+
+    # Determine primary key columns and their types
+    c.execute(f'PRAGMA table_info("{table_name}");')
+    primary_key_columns = [
+        # cid, name, type, notnull, dflt_value, pk
+        (row[1], row[2])
+        for row in c.fetchall()
+        if row[5]
+    ]
+    if not primary_key_columns:
+        raise ChronicleError(f"Table {table_name} has no primary keys")
+
+    # Create the _chronicle_ table
+    pk_def = ", ".join(
+        [f'"{col_name}" {col_type}' for col_name, col_type in primary_key_columns]
+    )
+
+    with conn:
+        c.execute(
+            textwrap.dedent(
+                f"""
+            CREATE TABLE "_chronicle_{table_name}" (
+                {pk_def},
+                updated_ms INTEGER,
+                deleted INTEGER DEFAULT 0,
+                PRIMARY KEY ({', '.join([f'"{col[0]}"' for col in primary_key_columns])})
+            );
+        """
+            )
+        )
+        # Add an index on the updated_ms column
+        c.execute(
+            f"""
+            CREATE INDEX "_chronicle_{table_name}_updated_ms" ON "_chronicle_{table_name}" (updated_ms);
+        """.strip()
+        )
+
+        # Populate the _chronicle_ table with existing rows from the original table
+        current_time_expr = (
+            "CAST((julianday('now') - 2440587.5) * 86400 * 1000 AS INTEGER)"
+        )
+        c.execute(
+            f"""
+            INSERT INTO "_chronicle_{table_name}" ({', '.join([col[0] for col in primary_key_columns])}, updated_ms)
+            SELECT {', '.join([f'"{col[0]}"' for col in primary_key_columns])}, {current_time_expr} 
+            FROM "{table_name}";
+        """
+        )
+
+        # Create the after insert trigger
+        c.execute(
+            f"""
+            CREATE TRIGGER "_chronicle_{table_name}_ai"
+            AFTER INSERT ON "{table_name}"
+            FOR EACH ROW
+            BEGIN
+                INSERT INTO "_chronicle_{table_name}" ({', '.join([f'"{col[0]}"' for col in primary_key_columns])}, updated_ms)
+                VALUES ({', '.join(['NEW.' + f'"{col[0]}"' for col in primary_key_columns])}, {current_time_expr});
+            END;
+        """
+        )
+
+        # Create the after update trigger
+        c.execute(
+            f"""
+            CREATE TRIGGER "_chronicle_{table_name}_au"
+            AFTER UPDATE ON "{table_name}"
+            FOR EACH ROW
+            BEGIN
+                UPDATE "_chronicle_{table_name}"
+                SET updated_ms = {current_time_expr},
+                -- Also update primary key columns if they have changed:
+                {', '.join([f'"{col[0]}" = NEW."{col[0]}"' for col in primary_key_columns])}
+                WHERE { ' AND '.join([f'"{col[0]}" = OLD."{col[0]}"' for col in primary_key_columns]) };
+            END;
+        """
+        )
+
+        # Create the after delete trigger
+        c.execute(
+            f"""
+            CREATE TRIGGER "_chronicle_{table_name}_ad"
+            AFTER DELETE ON "{table_name}"
+            FOR EACH ROW
+            BEGIN
+                UPDATE "_chronicle_{table_name}"
+                SET updated_ms = {current_time_expr}, deleted = 1
+                WHERE { ' AND '.join([f'"{col[0]}" = OLD."{col[0]}"' for col in primary_key_columns]) };
+            END;
+        """
+        )

sqlite-chronicle-0.1/tests/test_sqlite_chronicle.py

@@ -0,0 +1,75 @@
+import pytest
+import sqlite_utils
+from sqlite_chronicle import enable_chronicle
+import time
+from unittest.mock import ANY
+
+
+@pytest.mark.parametrize("table_name", ("dogs", "dogs and stuff", "weird.table.name"))
+@pytest.mark.parametrize("pks", (["id"], ["id", "name"]))
+def test_enable_chronicle(table_name, pks):
+    chronicle_table = f"_chronicle_{table_name}"
+    db = sqlite_utils.Database(memory=True)
+    db[table_name].insert_all(
+        [
+            {"id": 1, "name": "Cleo", "color": "black"},
+            {"id": 2, "name": "Pancakes", "color": "corgi"},
+        ],
+        pk=pks[0] if len(pks) == 1 else pks,
+    )
+    enable_chronicle(db.conn, table_name)
+    # It should have the same primary keys
+    assert db[chronicle_table].pks == pks
+    # Should also have updated_ms and deleted columns
+    assert set(db[chronicle_table].columns_dict.keys()) == set(
+        pks + ["updated_ms", "deleted"]
+    )
+    # With an index
+    assert db[chronicle_table].indexes[0].columns == ["updated_ms"]
+    if pks == ["id"]:
+        expected = [
+            {"id": 1, "updated_ms": ANY, "deleted": 0},
+            {"id": 2, "updated_ms": ANY, "deleted": 0},
+        ]
+    else:
+        expected = [
+            {"id": 1, "name": "Cleo", "updated_ms": ANY, "deleted": 0},
+            {"id": 2, "name": "Pancakes", "updated_ms": ANY, "deleted": 0},
+        ]
+    assert list(db[chronicle_table].rows) == expected
+    # Running it again should do nothing because table exists
+    enable_chronicle(db.conn, table_name)
+    # Insert a row
+    db[table_name].insert({"id": 3, "name": "Mango", "color": "orange"})
+    get_by = 3 if pks == ["id"] else (3, "Mango")
+    row = db[chronicle_table].get(get_by)
+    if pks == ["id"]:
+        assert row == {"id": 3, "updated_ms": ANY, "deleted": 0}
+    else:
+        assert row == {"id": 3, "name": "Mango", "updated_ms": ANY, "deleted": 0}
+    record_timestamp = db[chronicle_table].get(get_by)["updated_ms"]
+    time.sleep(0.01)
+    # Update a row
+    db[table_name].update(get_by, {"color": "mango"})
+    assert db[chronicle_table].get(get_by)["updated_ms"] > record_timestamp
+    # Delete a row
+    assert db[table_name].count == 3
+    time.sleep(0.01)
+    db[table_name].delete(get_by)
+    assert db[table_name].count == 2
+    assert db[chronicle_table].get(get_by)["deleted"] == 1
+    new_record_timestamp = db[chronicle_table].get(get_by)["updated_ms"]
+    assert new_record_timestamp > record_timestamp
+    # Now update a column that's part of the compound primary key
+    time.sleep(0.1)
+    if pks == ["id", "name"]:
+        db[table_name].update((2, "Pancakes"), {"name": "Pancakes the corgi"})
+        # This should have renamed the row in the chronicle table as well
+        renamed_row = db[chronicle_table].get((2, "Pancakes the corgi"))
+        assert renamed_row["updated_ms"] > record_timestamp
+    else:
+        # Update single primary key
+        db[table_name].update(2, {"id": 4})
+        # This should have renamed the row in the chronicle table as well
+        renamed_row = db[chronicle_table].get(4)
+        assert renamed_row["updated_ms"] > record_timestamp