reader 3.10__tar.gz → 3.12__tar.gz

{reader-3.10 → reader-3.12}/CHANGES.rst

@@ -6,6 +6,37 @@ Changelog
   :noindex:
 
 
+Version 3.12
+------------
+
+Released 2024-03-05
+
+* Split the :ref:`full-text search <fts>` index into a separate,
+  attached database.
+  (:issue:`323`)
+* Require at least SQLite 3.18.
+  Previously, *reader* core required 3.15,
+  and only :meth:`~Reader.update_search` required 3.18.
+  (:issue:`323`)
+* Enable `write-ahead logging`_ only once, when the database is created,
+  instead of every time it is opened.
+  (:issue:`323`)
+* Vacuum the main database after migrations. (:issue:`323`)
+* Add an internal :ref:`change tracking API <changes>`
+  to formalize how search keeps in sync with storage.
+  (:issue:`323`)
+* Refactor storage internals. (:issue:`323`)
+
+
+Version 3.11
+------------
+
+Released 2023-12-30
+
+* Allow filtering entries by their (entry) tags. (:issue:`328`)
+* Support Python 3.12. (:issue:`326`)
+
+
 Version 3.10
 ------------
 

{reader-3.10/src/reader.egg-info → reader-3.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: reader
-Version: 3.10
+Version: 3.12
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause
@@ -21,6 +21,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Classifier: Topic :: Internet
@@ -68,6 +69,7 @@ Requires-Dist: mechanicalsoup; (implementation_name != "pypy" or python_version
 Requires-Dist: requests-wsgi-adapter; extra == "tests"
 Requires-Dist: lxml; (implementation_name != "pypy" or python_version <= "3.9") and extra == "tests"
 Requires-Dist: html5lib; extra == "tests"
+Requires-Dist: numpy; (implementation_name != "pypy" and os_name == "posix") and extra == "tests"
 Requires-Dist: mypy; implementation_name != "pypy" and extra == "tests"
 Requires-Dist: types-requests; extra == "tests"
 Requires-Dist: types-beautifulsoup4; extra == "tests"

{reader-3.10 → reader-3.12}/docs/api.rst

@@ -173,6 +173,7 @@ The class hierarchy for :mod:`reader` exceptions is:
 Type aliases
 ------------
 
+.. autodata:: reader.types.TagFilterInput
 .. autodata:: reader.types.TristateFilterInput
 
 

{reader-3.10 → reader-3.12}/docs/dev.rst

@@ -34,7 +34,7 @@ but I will prioritize supporting :doc:`contributors <contributing>`
 
 * OPML support, :issue:`165`
 * :ref:`deleting entries <deleting entries>`
-* more feed interaction statistics, :issue:`254`
+* :ref:`feed interaction statistics <counts api>`
 
 * security
 
@@ -57,7 +57,6 @@ but I will prioritize supporting :doc:`contributors <contributing>`
 
   * :ref:`searchable tag values <searchable tags>`, e.g. for comments
   * :ref:`unification with entry.read/important <entry flag unification>`
-  * filter entries by entry tags
   * optimistic locking, :issue:`308`
   * filter tags by prefix, :issue:`309`
 
@@ -339,6 +338,28 @@ Detailed requirements and API discussion: :issue:`168#issuecomment-642002049`.
 
 Minimal work needed to support alternate storages: :issue:`168#issuecomment-1383127564`.
 
+Storage internal API documented in version 3.10 (November 2023) in :issue:`325`.
+
+
+Database optimization
+~~~~~~~~~~~~~~~~~~~~~
+
+Optimization lessons learned while fixing "database is locked" errors: :issue:`175#issuecomment-657495233`.
+
+Some general guidance on schema/index design: :issue:`327#issuecomment-1859147186`.
+
+Speeding up ``get_entries(sort='recent')``:
+
+* first attempt at adding indexes: :issue:`134`
+* using a computed column (``recent_sort``) didn't change things very much: :issue:`279`
+* an index on ``recent_sort`` alone is not enough for pagination,
+  the index needs to match 1:1 the WHERE clause: :issue:`330`.
+
+Speeding up ``get_entry_counts(feed=...)``:
+
+* having an index on entries(feed) yielded a 4x improvement: :issue:`251`
+* even better, we should cache commonly-used counts: :issue:`306#issuecomment-1694655504`
+
 
 Parser
 ~~~~~~
@@ -429,7 +450,8 @@ https://death.andgravity.com/query-builder
 Pagination for methods that return iterators
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Why do it for the private implementation: :issue:`167#issuecomment-626753299` (also a comment in storage code).
+Why do it for the private implementation: :issue:`167#issuecomment-626753299`;
+
 
 Detailed requirements and API discussion for public pagination: :issue:`196#issuecomment-706038363`.
 
@@ -444,6 +466,9 @@ From the initial issue:
 
 Enabling search by default, and alternative search APIs: :issue:`252`.
 
+Change tracking API: :issue:`323#issuecomment-1930756417`, model validated in
+`this gist <https://gist.github.com/lemon24/558955ad82ba2e4f50c0184c630c668c>`_.
+
 External resources:
 
 * Comprehensive, although a bit old (2017): `What every software engineer should know about search <https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d>`_ (`full version <http://webcache.googleusercontent.com/search?q=cache:https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d&sca_esv=570067020&prmd=ivn&strip=1&vwsrc=0>`_)
@@ -522,12 +547,17 @@ and why I want more use-cases before implementing them (basically, YAGNI):
 
 :issue:`253` discusses using entry tags to implement the current entry flags
 (read, important); tl;dr: it's not worth adding entry tags just for this.
+:issue:`327` discusses using entry tags for ``has_enclosures``; tl;dr:
+it wouldn't save a lot of code, it would be only *a bit* slower,
+and it reconfirms that read and important are integral to the data model,
+so we still want them as regular columns.
 
 After closing :issue:`228` with `wontfix` in late 2021,
 in early 2022 (following the :issue:`266` tag/metadata unification)
 I implemented entry and global tags in :issue:`272`;
 there's a list of known use cases in the issue description.
 
+
 Resource tags
 ^^^^^^^^^^^^^
 
@@ -581,6 +611,8 @@ Requirements, open questions, and how it interacts with :mod:`~reader.plugins.en
 A summary of why it isn't easy to do: :issue:`301#issuecomment-1442423151`.
 
 
+.. _counts api:
+
 Counts API
 ~~~~~~~~~~
 
@@ -589,6 +621,11 @@ Detailed requirements and API discussion: :issue:`185#issuecomment-731743327`.
 Tracking additional statistics (e.g. :attr:`~Entry.read_modified`): :issue:`254`;
 how to expose said statistics: :issue:`254#issuecomment-1807064610`.
 
+Notebook with a successful attempt to determine a feed "usefulness" score
+based on how many entries I mark as read / important / don't care;
+highlights a number of gaps in the *reader* API:
+https://gist.github.com/lemon24/93222ef4bc4a775092b56546a6e6cd0f
+
 
 Using None as a special argument value
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -622,6 +659,12 @@ Using a single Reader objects from multiple threads
 
 Some thoughts on why it's difficult to do: :issue:`206#issuecomment-751383418`.
 
+Requirements and use cases: :issue:`206#issuecomment-1179739301`.
+
+When/how to run ``pragma optimize``: :issue:`206#issuecomment-1183660880`.
+
+Full support added in version 2.16 (July 2022).
+
 
 Plugins
 ~~~~~~~

{reader-3.10 → reader-3.12}/docs/guide.rst

@@ -127,6 +127,8 @@ since each connection would be to a *different* database::
     reader.exceptions.StorageError: usage error: cannot use a private database from threads other than the creating thread
 
 
+.. _backups:
+
 Back-ups
 ~~~~~~~~
 
@@ -439,16 +441,27 @@ the entries that changed since the last call,
 so it is OK to call it relatively often.
 
 
-Because search adds  minor overhead to other :class:`Reader` methods
-and can almost double the size of the database,
-it can be turned on/off through the
-:meth:`~Reader.enable_search()` / :meth:`~Reader.disable_search()` methods.
-This is persistent across instances using the same database,
-and only needs to be done once.
-You can also use the ``search_enabled`` :func:`make_reader` argument
-for the same purpose.
-By default, search is disabled,
-and enabled automatically on the first :meth:`~Reader.update_search()` call.
+Search can be turned on/off through the
+:meth:`~Reader.enable_search()` / :meth:`~Reader.disable_search()` methods
+(persistent across instances using the same database),
+or the ``search_enabled`` argument of :func:`make_reader`;
+by default, search is enabled automatically
+on the first :meth:`~Reader.update_search()` call.
+If search is enabled,
+you should call :meth:`~Reader.update_search()` regularly
+to prevent unprocesses changes from accumulating over time.
+
+
+Because the search index can be almost as large as the main database,
+the default implementation splits it into a separate, attached database,
+which allows :ref:`backing up <backups>` the main database separately;
+for a reader created with ``make_reader('db.sqlite')``,
+the search index will be in ``db.sqlite.search``.
+
+
+.. versionchanged:: 3.12
+    Split the full-text search index into a separate database.
+
 
 
 
@@ -493,8 +506,7 @@ in which case the behavior is to ensure the tag exists
 
 Besides storing resource metadata,
 tags can be used for filtering feeds and entries
-(as of version |version|, only by feed tags;
-see the :meth:`~Reader.get_feeds()` documentation for more complex examples)::
+(see :data:`.TagFilterInput` for more complex examples)::
 
     >>> # feeds that have the tag "one"
     >>> [f.title for f in reader.get_feeds(tags=['one'])]

{reader-3.10 → reader-3.12}/docs/internal.rst

@@ -143,9 +143,11 @@ but at least one other implementation needs to exists before that.
 
     Currently, search is tightly-bound to a storage implementation
     (see :meth:`~.BoundSearchStorageType.make_search`).
-    In the future, it may be possible to mix and match
-    main storage and search implementations,
-    but :class:`StorageType` will likely need
+    While the :ref:`change tracking API <changes>` allows
+    search implementations to keep in sync with text content changes,
+    there is no convenient way for :meth:`SearchType.search_entries`
+    to filter/sort results without storage cooperation;
+    :class:`StorageType` will need
     additional capabilities to support this.
 
 
@@ -167,6 +169,27 @@ but at least one other implementation needs to exists before that.
     :members:
 
 
+.. _changes:
+
+.. autoclass:: ChangeTrackingStorageType()
+    :members:
+    :show-inheritance:
+
+.. autoclass:: ChangeTrackerType()
+    :members:
+
+.. autoclass:: Change
+    :members:
+
+.. autoclass:: Action
+    :members:
+
+.. autoattribute:: reader.Entry._sequence
+
+.. autoexception:: reader.exceptions.ChangeTrackingNotEnabledError
+    :show-inheritance:
+
+
 Data objects
 ~~~~~~~~~~~~
 

{reader-3.10 → reader-3.12}/docs/plugins.rst

@@ -67,8 +67,8 @@ However, the plugin used the internal :ref:`parser` API
 and stored old tweets alongside the rendered HTML content
 to avoid retrieving them again when updating the thread/entry.
 
-You can still find the code
-`on GitHub <https://github.com/lemon24/reader/blob/3.6/src/reader/_plugins/twitter.py>`_.
+You can still find the code on GitHub:
+`twitter.py <https://github.com/lemon24/reader/blob/3.6/src/reader/_plugins/twitter.py>`_.
 
 
 .. _tumblr_gdpr:
@@ -84,8 +84,8 @@ including machine-readable RSS feeds).
 This plugin is a good example of how to set cookies
 on the Requests session used to retrieve feeds.
 
-You can still find the code
-`on GitHub <https://github.com/lemon24/reader/blob/3.6/src/reader/_plugins/tumblr_gdpr.py>`_.
+You can still find the code on GitHub:
+`tumblr_gdpr.py <https://github.com/lemon24/reader/blob/3.6/src/reader/_plugins/tumblr_gdpr.py>`_.
 
 
 

{reader-3.10 → reader-3.12}/pyproject.toml

@@ -25,6 +25,7 @@ classifiers = [
     "Programming Language :: Python :: 3 :: Only",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: Implementation :: CPython",
     "Programming Language :: Python :: Implementation :: PyPy",
     "Topic :: Internet",
@@ -112,6 +113,8 @@ tests = [
     # lxml does not have a PyPy 3.10+ wheel.
     'lxml; (implementation_name != "pypy" or python_version <= "3.9")',
     "html5lib",
+    # for bench.py
+    'numpy; (implementation_name != "pypy" and os_name == "posix")',
     # mypy does not work on pypy (yet).
     'mypy; implementation_name != "pypy"',
     "types-requests",

{reader-3.10 → reader-3.12}/run.sh

@@ -99,7 +99,7 @@ function typing {
 
 
 function docs {
-    make -C docs html "$@"
+    make -C docs html SPHINXOPTS="-W" "$@"
 }