reader 3.11__tar.gz → 3.12__tar.gz

{reader-3.11 → reader-3.12}/CHANGES.rst

@@ -6,6 +6,28 @@ Changelog
   :noindex:
 
 
+Version 3.12
+------------
+
+Released 2024-03-05
+
+* Split the :ref:`full-text search <fts>` index into a separate,
+  attached database.
+  (:issue:`323`)
+* Require at least SQLite 3.18.
+  Previously, *reader* core required 3.15,
+  and only :meth:`~Reader.update_search` required 3.18.
+  (:issue:`323`)
+* Enable `write-ahead logging`_ only once, when the database is created,
+  instead of every time it is opened.
+  (:issue:`323`)
+* Vacuum the main database after migrations. (:issue:`323`)
+* Add an internal :ref:`change tracking API <changes>`
+  to formalize how search keeps in sync with storage.
+  (:issue:`323`)
+* Refactor storage internals. (:issue:`323`)
+
+
 Version 3.11
 ------------
 

{reader-3.11/src/reader.egg-info → reader-3.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: reader
-Version: 3.11
+Version: 3.12
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause

{reader-3.11 → reader-3.12}/docs/dev.rst

@@ -57,7 +57,6 @@ but I will prioritize supporting :doc:`contributors <contributing>`
 
   * :ref:`searchable tag values <searchable tags>`, e.g. for comments
   * :ref:`unification with entry.read/important <entry flag unification>`
-  * filter entries by entry tags, :issue:`328`
   * optimistic locking, :issue:`308`
   * filter tags by prefix, :issue:`309`
 
@@ -345,6 +344,8 @@ Storage internal API documented in version 3.10 (November 2023) in :issue:`325`.
 Database optimization
 ~~~~~~~~~~~~~~~~~~~~~
 
+Optimization lessons learned while fixing "database is locked" errors: :issue:`175#issuecomment-657495233`.
+
 Some general guidance on schema/index design: :issue:`327#issuecomment-1859147186`.
 
 Speeding up ``get_entries(sort='recent')``:
@@ -449,7 +450,8 @@ https://death.andgravity.com/query-builder
 Pagination for methods that return iterators
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Why do it for the private implementation: :issue:`167#issuecomment-626753299` (also a comment in storage code).
+Why do it for the private implementation: :issue:`167#issuecomment-626753299`;
+
 
 Detailed requirements and API discussion for public pagination: :issue:`196#issuecomment-706038363`.
 
@@ -464,6 +466,9 @@ From the initial issue:
 
 Enabling search by default, and alternative search APIs: :issue:`252`.
 
+Change tracking API: :issue:`323#issuecomment-1930756417`, model validated in
+`this gist <https://gist.github.com/lemon24/558955ad82ba2e4f50c0184c630c668c>`_.
+
 External resources:
 
 * Comprehensive, although a bit old (2017): `What every software engineer should know about search <https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d>`_ (`full version <http://webcache.googleusercontent.com/search?q=cache:https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d&sca_esv=570067020&prmd=ivn&strip=1&vwsrc=0>`_)

{reader-3.11 → reader-3.12}/docs/guide.rst

@@ -127,6 +127,8 @@ since each connection would be to a *different* database::
     reader.exceptions.StorageError: usage error: cannot use a private database from threads other than the creating thread
 
 
+.. _backups:
+
 Back-ups
 ~~~~~~~~
 
@@ -439,16 +441,27 @@ the entries that changed since the last call,
 so it is OK to call it relatively often.
 
 
-Because search adds  minor overhead to other :class:`Reader` methods
-and can almost double the size of the database,
-it can be turned on/off through the
-:meth:`~Reader.enable_search()` / :meth:`~Reader.disable_search()` methods.
-This is persistent across instances using the same database,
-and only needs to be done once.
-You can also use the ``search_enabled`` :func:`make_reader` argument
-for the same purpose.
-By default, search is disabled,
-and enabled automatically on the first :meth:`~Reader.update_search()` call.
+Search can be turned on/off through the
+:meth:`~Reader.enable_search()` / :meth:`~Reader.disable_search()` methods
+(persistent across instances using the same database),
+or the ``search_enabled`` argument of :func:`make_reader`;
+by default, search is enabled automatically
+on the first :meth:`~Reader.update_search()` call.
+If search is enabled,
+you should call :meth:`~Reader.update_search()` regularly
+to prevent unprocesses changes from accumulating over time.
+
+
+Because the search index can be almost as large as the main database,
+the default implementation splits it into a separate, attached database,
+which allows :ref:`backing up <backups>` the main database separately;
+for a reader created with ``make_reader('db.sqlite')``,
+the search index will be in ``db.sqlite.search``.
+
+
+.. versionchanged:: 3.12
+    Split the full-text search index into a separate database.
+
 
 
 

{reader-3.11 → reader-3.12}/docs/internal.rst

@@ -143,9 +143,11 @@ but at least one other implementation needs to exists before that.
 
     Currently, search is tightly-bound to a storage implementation
     (see :meth:`~.BoundSearchStorageType.make_search`).
-    In the future, it may be possible to mix and match
-    main storage and search implementations,
-    but :class:`StorageType` will likely need
+    While the :ref:`change tracking API <changes>` allows
+    search implementations to keep in sync with text content changes,
+    there is no convenient way for :meth:`SearchType.search_entries`
+    to filter/sort results without storage cooperation;
+    :class:`StorageType` will need
     additional capabilities to support this.
 
 
@@ -167,6 +169,27 @@ but at least one other implementation needs to exists before that.
     :members:
 
 
+.. _changes:
+
+.. autoclass:: ChangeTrackingStorageType()
+    :members:
+    :show-inheritance:
+
+.. autoclass:: ChangeTrackerType()
+    :members:
+
+.. autoclass:: Change
+    :members:
+
+.. autoclass:: Action
+    :members:
+
+.. autoattribute:: reader.Entry._sequence
+
+.. autoexception:: reader.exceptions.ChangeTrackingNotEnabledError
+    :show-inheritance:
+
+
 Data objects
 ~~~~~~~~~~~~
 

{reader-3.11 → reader-3.12}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.11'
+__version__ = '3.12'
 
 from .core import (
     Reader as Reader,

{reader-3.11 → reader-3.12}/src/reader/_app/cli.py

@@ -35,9 +35,6 @@ def serve(config, host, port, plugin, verbose):
     if plugin:
         config['app']['plugins'] = dict.fromkeys(plugin)
 
-    # TODO: remove this once we make debug_storage a storage_arg
-    config['default']['reader'].pop('debug_storage', None)
-
     app = create_app(config)
     app.wsgi_app = make_add_response_headers_middleware(
         app.wsgi_app,

{reader-3.11 → reader-3.12}/src/reader/_cli.py

@@ -1,5 +1,4 @@
 import functools
-import json
 import logging
 import os.path
 import shutil
@@ -17,7 +16,6 @@ from ._config import make_reader_config
 from ._config import make_reader_from_config
 from ._plugins import Loader
 from ._plugins import LoaderError
-from ._sqlite_utils import DebugConnection
 
 
 APP_NAME = reader.__name__
@@ -45,23 +43,7 @@ def abort(message, *args, **kwargs):
     raise click.ClickException(message.format(*args, **kwargs))
 
 
-def make_reader_with_plugins(*, debug_storage=False, **kwargs):
-    if debug_storage:
-        # TODO: the web app should be able to do this too
-
-        log_debug = logging.getLogger('reader._storage').debug
-        pid = os.getpid()
-
-        class Connection(DebugConnection):
-            _io_counters = True
-
-            @staticmethod
-            def _log_method(data):
-                data['pid'] = pid
-                log_debug(json.dumps(data))
-
-        kwargs['_storage_factory'] = Connection
-
+def make_reader_with_plugins(**kwargs):
     try:
         return make_reader_from_config(**kwargs)
     except StorageError as e:
@@ -170,7 +152,7 @@ def pass_reader(fn):
     @functools.wraps(fn)
     def wrapper(*args, **kwargs):
         ctx = click.get_current_context().find_root()
-        # TODO: replace with ctx.obj.make_reader('cli') when we get rid of debug_storage
+        # TODO: replace with ctx.obj.make_reader('cli')
         reader = make_reader_with_plugins(**ctx.obj.merged('cli').get('reader', {}))
         ctx.call_on_close(reader.close)
         return fn(reader, *args, **kwargs)
@@ -215,14 +197,9 @@ def pass_reader(fn):
         "If not provided, don't open local feeds."
     ),
 )
-@click.option(
-    '--debug-storage/--no-debug-storage',
-    hidden=True,
-    help="NOT TESTED. With -vv, log storage database calls.",
-)
 @click.version_option(reader.__version__, message='%(prog)s %(version)s')
 @click.pass_obj
-def cli(config, db, plugin, cli_plugin, feed_root, debug_storage):
+def cli(config, db, plugin, cli_plugin, feed_root):
     # TODO: mention in docs that --db/--plugin/envvars ALWAYS override the config
     # (same for wsgi envvars)
     # NOTE: we can never use click defaults for --db/--plugin, because they would override the config always
@@ -247,10 +224,6 @@ def cli(config, db, plugin, cli_plugin, feed_root, debug_storage):
     if feed_root is not None:
         config['default']['reader']['feed_root'] = feed_root
 
-    # until we make debug_storage a proper make_reader argument,
-    # and we get rid of make_reader_with_plugins
-    config['default']['reader']['debug_storage'] = debug_storage
-
     try:
         loader = Loader()
         loader.init(config, config.merged('cli').get('plugins', {}))

reader-3.12/src/reader/_storage/__init__.py

@@ -0,0 +1,39 @@
+from __future__ import annotations
+
+from typing import Any
+
+from .._types import ChangeTrackerType
+from .._types import SearchType
+from ._base import StorageBase
+from ._changes import Changes
+from ._entries import EntriesMixin
+from ._feeds import FeedsMixin
+from ._tags import TagsMixin
+
+
+# Row value support was added in 3.15.
+# pragma_*() tabled-valued functions were added in 3.16.
+# last_insert_rowid() support for FTS5 was added in 3.18.
+MINIMUM_SQLITE_VERSION = (3, 18)
+
+# Both storage and search use the JSON1 extension.
+REQUIRED_SQLITE_FUNCTIONS = ['json']
+
+
+class Storage(FeedsMixin, EntriesMixin, TagsMixin, StorageBase):
+    """Data access object used for all storage (except search).
+
+    This class is split into per-domain mixins, add new methods accordingly.
+
+    Add a test_storage.py::test_errors_locked test for each new public method.
+
+    """
+
+    def __init__(self, path: str, timeout: float | None = None):
+        super().__init__(path, timeout)
+        self.changes: ChangeTrackerType = Changes(self)
+
+    def make_search(self) -> SearchType:
+        from ._search import Search
+
+        return Search(self)

reader-3.12/src/reader/_storage/_base.py

@@ -0,0 +1,110 @@
+from __future__ import annotations
+
+import json
+import os
+import sqlite3
+import sys
+from collections.abc import Callable
+from collections.abc import Iterable
+from functools import partial
+from typing import Any
+from typing import TypeVar
+
+from . import _sqlite_utils
+from ..exceptions import StorageError
+from ._sql_utils import paginated_query
+from ._sql_utils import Query
+
+
+APPLICATION_ID = b'read'
+
+_T = TypeVar('_T')
+
+
+# also used by tests
+CONNECTION_CLS = sqlite3.Connection
+
+debug = os.environ.get('READER_DEBUG_STORAGE', '')
+assert set(debug) <= {'m', 't', 'i'}, f"invalid READER_DEBUG_STORAGE={debug}"
+
+if debug:  # pragma: no cover
+
+    class CONNECTION_CLS(_sqlite_utils.DebugConnection):  # type: ignore # noqa: F811
+        _set_trace = 't' in debug
+        _io_counters = 'i' in debug
+        _pid = os.getpid()
+
+        def _log_method(self, data):  # type: ignore
+            data['pid'] = self._pid
+            print('STORAGE_DEBUG', json.dumps(data), file=sys.stderr)
+
+
+wrap_exceptions = partial(_sqlite_utils.wrap_exceptions, StorageError)
+
+
+class StorageBase:
+    #: Private storage API.
+    chunk_size = 2**8
+
+    @wrap_exceptions(message="while opening database")
+    def __init__(self, path: str, timeout: float | None = None):
+        kwargs: dict[str, Any] = {'factory': CONNECTION_CLS}
+        if timeout is not None:
+            kwargs['timeout'] = timeout
+
+        self.factory = _sqlite_utils.LocalConnectionFactory(path, **kwargs)
+        db = self.factory()
+        try:
+            self.setup_db(db)
+        except BaseException:
+            db.close()
+            raise
+
+        self.path = path
+        self.timeout = timeout
+
+    def get_db(self) -> sqlite3.Connection:
+        return self.factory()
+
+    @staticmethod
+    def setup_db(db: sqlite3.Connection) -> None:
+        """Private storage API (used by tests)."""
+        from ._schema import MIGRATION
+        from . import MINIMUM_SQLITE_VERSION, REQUIRED_SQLITE_FUNCTIONS
+
+        return _sqlite_utils.setup_db(
+            db,
+            migration=MIGRATION,
+            id=APPLICATION_ID,
+            minimum_sqlite_version=MINIMUM_SQLITE_VERSION,
+            required_sqlite_functions=REQUIRED_SQLITE_FUNCTIONS,
+        )
+
+    @wrap_exceptions()
+    def __enter__(self) -> None:
+        self.factory.__enter__()
+
+    @wrap_exceptions()
+    def __exit__(self, *_: Any) -> None:
+        self.factory.__exit__()
+
+    @wrap_exceptions()
+    def close(self) -> None:
+        self.factory.close()
+
+    def paginated_query(
+        self,
+        make_query: Callable[[], tuple[Query, dict[str, Any]]],
+        limit: int | None = None,
+        last: tuple[Any, ...] | None = None,
+        row_factory: Callable[[tuple[Any, ...]], _T] | None = None,
+    ) -> Iterable[_T]:
+        with wrap_exceptions():
+            yield from paginated_query(
+                self.get_db(),
+                make_query,
+                self.chunk_size,
+                limit or 0,
+                last,
+                row_factory,
+            )

reader-3.12/src/reader/_storage/_changes.py

@@ -0,0 +1,213 @@
+from __future__ import annotations
+
+import sqlite3
+from typing import Any
+from typing import TYPE_CHECKING
+
+from .._types import Action
+from .._types import Change
+from ..exceptions import ChangeTrackingNotEnabledError
+from ._base import wrap_exceptions
+from ._sql_utils import parse_schema
+from ._sql_utils import Query
+from ._sqlite_utils import ddl_transaction
+
+if TYPE_CHECKING:  # pragma: no cover
+    from ._base import StorageBase
+
+
+ENABLED_EXC = {'no such table': lambda _: ChangeTrackingNotEnabledError()}
+
+
+class Changes:
+    def __init__(self, storage: StorageBase):
+        self.storage = storage
+
+    @wrap_exceptions()
+    def enable(self) -> None:
+        with ddl_transaction(self.storage.get_db()) as db:
+            try:
+                self._enable(db)
+            except sqlite3.OperationalError as e:
+                if "table changes already exists" in str(e).lower():
+                    return
+                raise  # pragma: no cover
+
+    @classmethod
+    def _enable(cls, db: sqlite3.Connection) -> None:
+        assert db.in_transaction
+        for objects in SCHEMA.values():
+            for object in objects.values():
+                object.create(db)
+        db.execute("UPDATE entries SET sequence = randomblob(16)")
+        db.execute(
+            """
+            INSERT INTO changes
+            SELECT sequence, feed, id, '', 1 FROM entries
+            """
+        )
+
+    @wrap_exceptions()
+    def disable(self) -> None:
+        with ddl_transaction(self.storage.get_db()) as db:
+            self._disable(db)
+
+    @classmethod
+    def _disable(cls, db: sqlite3.Connection) -> None:
+        assert db.in_transaction
+        for objects in SCHEMA.values():
+            for object in objects.values():
+                db.execute(f"DROP {object.type} IF EXISTS {object.name}")
+        db.execute("UPDATE entries SET sequence = NULL")
+
+    @wrap_exceptions(ENABLED_EXC)
+    def get(
+        self, action: Action | None = None, limit: int | None = None
+    ) -> list[Change]:
+        if not limit or limit > self.storage.chunk_size:
+            limit = self.storage.chunk_size
+        context = {'limit': limit}
+        # the ORDER_BY is only used for testing; should this return a set instead?
+        query = Query().SELECT('*').FROM('changes').ORDER_BY('rowid').LIMIT(':limit')
+        if action:
+            query.WHERE('action = :action')
+            context['action'] = action.value
+        rows = self.storage.get_db().execute(str(query), context)
+        return list(map(change_factory, rows))
+
+    @wrap_exceptions(ENABLED_EXC)
+    def done(self, changes: list[Change]) -> None:
+        if len(changes) > self.storage.chunk_size:
+            raise ValueError(f"too many changes, expected <= {self.storage.chunk_size}")
+        with self.storage.get_db() as db:
+            for change in changes:
+                db.execute(
+                    """
+                    DELETE FROM changes
+                    WHERE (sequence, feed, id, key, action)
+                        = (:sequence, :feed, :id, :key, :action)
+                    """,
+                    change_to_dict(change),
+                )
+
+
+def change_factory(row: tuple[Any, ...]) -> Change:
+    sequence, feed, id, key, action = row
+    resource = tuple(filter(bool, (feed, id)))
+    return Change(Action(action), sequence, resource, key or None)
+
+
+def change_to_dict(change: Change) -> dict[str, Any]:
+    resource = change.resource_id + ('', '')
+    return dict(
+        sequence=change.sequence,
+        feed=resource[0],
+        id=resource[1],
+        key=change.tag_key or '',
+        action=change.action.value,
+    )
+
+
+SCHEMA = parse_schema("""
+
+CREATE TABLE changes (
+    sequence BLOB NOT NULL,
+    feed TEXT NOT NULL,
+    id TEXT NOT NULL,
+    key TEXT NOT NULL,
+    action INTEGER NOT NULL,
+
+    PRIMARY KEY (sequence, feed, id, key)
+);
+
+
+CREATE TRIGGER changes_entry_insert
+AFTER INSERT
+ON entries
+BEGIN
+    -- SELECT print('  entry_insert', new.feed, new.id);
+
+    UPDATE entries
+        SET sequence = randomblob(16)
+        WHERE (new.id, new.feed) = (id, feed);
+
+    INSERT OR REPLACE INTO changes
+        SELECT sequence, feed, id, '', 1
+        FROM entries
+        WHERE (feed, id) = (new.feed, new.id);
+END;
+
+
+-- Can't handle feed URL changes in changes_entry_update because
+-- those entry updates are a consequence of ON UPDATE CASCADE,
+-- which overrides the INSERT OR REPLACE used in the trigger,
+-- because "conflict handling policy of the outer statement"
+-- takes precedence per https://sqlite.org/lang_createtrigger.html.
+-- Instead, we handle feed URL changes in changes_feed_changed.
+
+CREATE TRIGGER changes_entry_update
+AFTER UPDATE
+OF title, summary, content
+ON entries
+WHEN
+    new.id = old.id AND new.feed = old.feed AND (
+        coalesce(new.title, '') != coalesce(old.title, '')
+        OR coalesce(new.summary, '') != coalesce(old.summary, '')
+        OR coalesce(new.content, '') != coalesce(old.content, '')
+    )
+BEGIN
+    -- SELECT print('  entry_update', old.feed, old.id, '->', new.feed, new.id);
+
+    INSERT OR REPLACE INTO changes
+        VALUES (old.sequence, old.feed, old.id, '', 2);
+
+    UPDATE entries
+        SET sequence = randomblob(16)
+        WHERE (new.id, new.feed) = (id, feed);
+
+    INSERT OR REPLACE INTO changes
+        SELECT sequence, feed, id, '', 1
+        FROM entries
+        WHERE (feed, id) = (new.feed, new.id);
+END;
+
+
+CREATE TRIGGER changes_entry_delete
+AFTER DELETE
+ON entries
+BEGIN
+    -- SELECT print('  entry_delete', old.feed, old.id);
+
+    INSERT OR REPLACE INTO changes
+        VALUES (old.sequence, old.feed, old.id, '', 2);
+END;
+
+
+CREATE TRIGGER changes_feed_changed
+AFTER UPDATE
+OF url, title, user_title
+ON feeds
+WHEN
+    new.url != old.url
+    OR coalesce(new.title, '') != coalesce(old.title, '')
+    OR coalesce(new.user_title, '') != coalesce(old.user_title, '')
+BEGIN
+    -- SELECT print('  feed_url_change', old.url, '->', new.url);
+
+    INSERT OR REPLACE INTO changes
+        SELECT sequence, old.url, id, '', 2
+        FROM entries
+        WHERE feed = new.url;
+
+    UPDATE entries
+        SET sequence = randomblob(16)
+        WHERE feed = new.url;
+
+    INSERT OR REPLACE INTO changes
+        SELECT sequence, feed, id, '', 1
+        FROM entries
+        WHERE feed = new.url;
+END;
+
+
+""")  # fmt: skip