reader 3.19.dev0__tar.gz → 3.20__tar.gz

{reader-3.19.dev0 → reader-3.20}/CHANGES.rst

@@ -6,18 +6,65 @@ Changelog
   :no-index:
 
 
+Version 3.20
+------------
+
+Released 2025-11-26
+
+* Add the ``read_only`` :func:`make_reader` argument.
+  Thanks to `Roman Milko`_ for the PR.
+  (:issue:`319`)
+
+* Include the ``alt`` and ``title`` HTML attributes
+  in the :ref:`full-text search <fts>` index.
+  Also, don't add extra whitespace between tags
+  (especially important for inline tags).
+  (:issue:`371`)
+
+  .. note::
+
+    Old entries will **not** be re-indexed automatically;
+    to do so, disable then enable search.
+
+* :mod:`~reader.plugins.entry_dedupe` improvements. (:issue:`371`)
+
+  * Use more heuristics to find potential duplicates
+    (in addition to title matching):
+    match link, match published timestamp,
+    strip common title prefixes for new entries.
+  * When comparing entries,
+    include the ``alt`` and ``title`` HTML attributes,
+    strip accents, and treat dates and versions as single tokens.
+  * Increase required minimum content length from 32 to 48 words.
+  * Add the ``.reader.dedupe.once.link`` and
+    ``.reader.dedupe.once.title.prefix`` escape hatches.
+  * Clean up documentation, refactor code, and add extensive tests.
+
+* Log run times of before/after update hooks.
+
+.. _Roman Milko: https://github.com/rastr-0
+
+
 Version 3.19
 ------------
 
-Unreleased
+Released 2025-10-01
 
 * Support Python 3.14.
+* Remove deprecated empty ``search`` and ``readtime`` extras.
+* Remove undocumented development ``tests``, ``docs``, and ``dev`` extras.
 
 * :mod:`~reader._plugins.enclosure_tags` plugin:
   strip HTML tags from values used as MP3 tags
   (for example, ``<b>`` tags used to highlight search result matches).
 
-* Remove long-deprecated, empty ``search`` and ``readtime`` extras.
+* Project infrastructure improvements.
+
+  * Clean up :doc:`contributing` documentation and :gh:`run.sh <run.sh>` script.
+  * Use GitHub Actions to publish releases to PyPI.
+  * Split type checking and testing jobs in the ``build`` GHA workflow.
+  * Make tox runs slightly faster.
+  * Use dependency groups instead of extras for development dependencies.
 
 
 Version 3.18

{reader-3.19.dev0/src/reader.egg-info → reader-3.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reader
-Version: 3.19.dev0
+Version: 3.20
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause

{reader-3.19.dev0 → reader-3.20}/docs/dev.rst

@@ -226,25 +226,31 @@ Making a release
 
 .. note::
 
-    :gh:`scripts/release.py <scripts/release.py>` already does most of these.
+    The :gh:`release.py <scripts/release.py>` script together with
+    the :gh:`publish <.github/workflows/publish.yaml>`  GitHub Actions workflow
+    already do most of these.
 
 Making a release (from ``x`` to ``y`` == ``x + 1``):
 
 * (release.py) bump version in ``src/reader/__init__.py`` to ``y``
 * (release.py) update changelog with release version and date
 * (release.py) make sure tests pass / docs build
-* (release.py) clean up dist/: ``rm -rf dist/``
-* (release.py) build tarball and wheel: ``python -m build``
 * (release.py) push to GitHub
-* (release.py prompts) wait for GitHub Actions / Codecov / Read the Docs builds to pass
-* upload to test PyPI and check: ``twine upload --repository-url https://test.pypi.org/legacy/ dist/*``
-* (release.py) upload to PyPI: ``twine upload dist/*``
+* (release.py prompts) wait for GitHub Actions / Read the Docs builds to pass
 * (release.py) tag current commit with `<major>.<minor>` and `<major>.x`
-  (e.g. when releasing `1.20`: `1.20` and `1.x`)
-* (release.py prompts) create release in GitHub
-* build docs from latest and enable ``y`` docs version (should happen automatically after the first time)
+  (e.g. when releasing `1.20`: `1.20` and `1.x`) and push tags to GitHub
+* (release.py prompts) approve publish workflow
+* (publish workflow) build and upload to PyPI
+* alternatively, to make a manual release (should not be needed)
+
+  * clean up dist/: ``rm -rf dist/``
+  * build tarball and wheel: ``python -m build``
+  * upload to test PyPI and check: ``twine upload --repository-url https://test.pypi.org/legacy/ dist/*``
+  * upload to PyPI: ``twine upload dist/*``
+
+* (publish workflow) create draft release in GitHub
+* (release.py prompts) publish release in GitHub
 * (release.py) bump versions from ``y`` to ``(y + 1).dev0``, add ``(y + 1)`` changelog section
-* (release.py prompts) trigger Read the Docs build for `<major>.x` (doesn't happen automatically)
 
 
 
@@ -750,11 +756,8 @@ Thoughts on dynamic lists of feeds: :issue:`165#issuecomment-905893420`.
 Duplicate entries
 ~~~~~~~~~~~~~~~~~
 
-Duplicate entries are mainly handled by the :mod:`reader.entry_dedupe` plugin.
-
-* Using MinHash to speed up similarity checks (maybe): https://gist.github.com/lemon24/b9af5ade919713406bda9603847d32e5
-* Discussion of unifying "on-line" dedupe (after an entry is added/updated),
-  and "on-demand" dedupe (backfill): :issue:`340`.
+Duplicate entries are handled by :mod:`~reader.plugins.entry_dedupe`
+(extensive notes in the :gh:`docstring <src/reader/plugins/entry_dedupe.py>`).
 
 However, it is also possible for a feed to have two entries with the same id
 – yes, even though in most (if not all) formats,

{reader-3.19.dev0 → reader-3.20}/pyproject.toml

@@ -99,15 +99,15 @@ tests = [
     "coverage",
     "requests-mock",
     # mechanicalsoup hard-depends on lxml (see below)
-    'mechanicalsoup; (implementation_name != "pypy" and python_version <= "3.12")',
+    'mechanicalsoup; (implementation_name != "pypy" and python_version <= "3.14")',
     "requests-wsgi-adapter",
     # we want to test search with all known bs4 parsers.
     # lxml usually does not have recent PyPy wheels.
     # lxml usually does not have pre-relase CPython wheels.
-    'lxml; (implementation_name != "pypy" and python_version <= "3.12")',
+    'lxml; (implementation_name != "pypy" and python_version <= "3.14")',
     "html5lib",
     # for bench.py
-    'numpy; (implementation_name != "pypy" and os_name == "posix" and python_version <= "3.12")',
+    'numpy; (implementation_name != "pypy" and os_name == "posix" and python_version <= "3.14")',
 ]
 
 typing = [

reader-3.20/scripts/entry_dedupe_backtest.py

@@ -0,0 +1,113 @@
+"""
+Run entry_dedupe .dedupe.once for an entire database,
+and parse the logs into a form that makes backtesting easier.
+
+Run dedupe and pretty-print logs:
+
+    $ python scripts/entry_dedupe_backtest.py dedupe.log run
+
+Pretty-print existing logs:
+
+    $ python scripts/entry_dedupe_backtest.py dedupe.log
+
+"""
+
+import logging
+import os
+import re
+import sys
+import time
+
+from reader import make_reader
+from reader.plugins import entry_dedupe
+
+
+DB_ARCHIVE = "_backups/reader.sqlite.2025-10-04.gz"
+
+file = sys.argv[1]
+run = len(sys.argv) > 2 and sys.argv[2] == 'run'
+only = len(sys.argv) > 3 and sys.argv[3]
+
+if run:
+    os.system(f"rm db.*; gzip -dc {DB_ARCHIVE} > db.sqlite")
+    reader = make_reader('db.sqlite')
+    logging.basicConfig(filename=file, filemode='w', level=logging.DEBUG)
+    start = time.monotonic()
+    for feed in reader.get_feeds():
+        if only and not re.search(only, feed.title or '', re.I):
+            continue
+        reader.set_tag(feed, '.reader.dedupe.once')
+        entry_dedupe.after_feed_update(reader, feed)
+    end = time.monotonic()
+    print(f"TIME: {end-start:.3f} seconds", file=sys.stderr)
+    reader.close()
+    os.system(f"rm db.*; gzip -dc {DB_ARCHIVE} > db.sqlite")
+
+reader = make_reader('db.sqlite')
+
+
+def shorten(s, n=48):
+    if not s:
+        return s
+    s = ' '.join(s.split())
+    if len(s) < n:
+        return s
+    return s[:n] + '...'
+
+
+feed = None
+entries = None
+grouper = None
+prev = ''
+
+for l in open(file):
+    if 'skipping:' in prev and 'skipping:' not in l:
+        print()
+    if 'skipping:' not in prev and 'skipping:' in l:
+        print('    HUGE\n')
+    prev = l
+
+    if m := re.search(r"for feed '([^']+)'", l):
+        feed = reader.get_feed(m[1])
+        entries = {e.id: e for e in reader.get_entries(feed=feed)}
+        grouper = None
+        print(f"{feed.title}  {feed.url}\n")
+        continue
+
+    if m := re.search("grouper ([^:]+): all=.*", l):
+        grouper = m[1]
+        print(f"  grouper {grouper}\n")
+        continue
+
+    # if m := re.search(r"found mass duplication pairs \d+ < \d+, skipping", l):
+    # print("    TOO FEW PAIRS\n")
+    # continue
+
+    if m := re.search(r"found group of size \d+ > \d+, skipping: (.*)", l):
+        ids = eval(m[1])
+        e = entries[ids[0]]
+        fragment = ''
+        if 'link' in grouper:
+            fragment = e.link
+        elif 'published' in grouper:
+            fragment = e.published or e.updated
+        print(f"    {len(ids):>3}  {shorten(e.title)}  {fragment}")
+        continue
+
+    if m := re.search(r"grouper \S+: found (\[.*)", l):
+        ids = eval(m[1])
+        if ids:
+            print('    DUPLICATES\n')
+        for ids in ids:
+            for i, id in enumerate(ids):
+                e = entries[id]
+                print(f"    {shorten(e.title, 64)}  {e.id}")
+                if 'link' in grouper:
+                    print(f"      link = {e.link}")
+                if 'published' in grouper:
+                    print(f"      published = {e.published or e.updated}")
+            print()
+        continue
+
+    if m := re.search(r"grouper \S+: group count by size (.*)", l):
+        print(f"    summary  {m[1]}\n")

{reader-3.19.dev0 → reader-3.20}/scripts/release.py

@@ -34,11 +34,6 @@ def run_tox():
     run_no_venv('tox -p all', shell=True)
 
 
-def build():
-    run('rm -rf dist/', shell=True)
-    run('python -m build', shell=True)
-
-
 def path_sub(pattern, repl, path):
     with open(path) as f:
         text = f.read()
@@ -112,10 +107,6 @@ def check_unpushed():
         abort("unpushed changes\n\n{}\n", p.stdout.strip('\n'))
 
 
-def upload_to_pypi():
-    run('twine upload dist/*', shell=True)
-
-
 def add_and_push_tags(tags):
     for tag in tags:
         run(['git', 'tag', '--force', tag])
@@ -150,10 +141,6 @@ def main(version, new_version, date, tox, changelog):
 
     confirm("Wait for GitHub Actions / Read the Docs builds to pass.")
 
-    confirm("Upload to PyPI?")
-    build()
-    upload_to_pypi()
-
     tags = [version]
     version_x = version.partition('.')[0] + '.x'
     if new_version_is_final:
@@ -162,7 +149,15 @@ def main(version, new_version, date, tox, changelog):
     confirm(f"Add and push tags ({', '.join(tags)})?")
     add_and_push_tags(tags)
 
-    confirm("Create release {} in GitHub (doesn't happen automatically).", version)
+    confirm(
+        "Approve publish workflow to upload to PyPI.\n  {}\nDone?",
+        "https://github.com/lemon24/reader/actions/workflows/publish.yaml",
+    )
+    confirm(
+        "Publish draft release {} in GitHub.\n  {}\nDone?",
+        version,
+        "https://github.com/lemon24/reader/releases",
+    )
 
     new_version_full = f"{new_version}.dev0"
     update_init_version(new_version_full)
@@ -173,11 +168,6 @@ def main(version, new_version, date, tox, changelog):
     confirm("Push version {}?", new_version_full)
     push()
 
-    # TODO: I just enabled branch or tag creation/deletion for the RtD webhook in GitHub, this might not be needed next time.
-    confirm(
-        f"Trigger Read the Docs build for {version_x} (doesn't happen automatically)."
-    )
-
 
 if __name__ == '__main__':
     main()

{reader-3.19.dev0 → reader-3.20}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.19.dev0'
+__version__ = '3.20'
 
 # isort: off
 

{reader-3.19.dev0 → reader-3.20}/src/reader/_storage/__init__.py

@@ -29,8 +29,10 @@ class Storage(FeedsMixin, EntriesMixin, TagsMixin, StorageBase):
 
     """
 
-    def __init__(self, path: str, timeout: float | None = None):
-        super().__init__(path, timeout)
+    def __init__(
+        self, path: str, read_only: bool = False, timeout: float | None = None
+    ):
+        super().__init__(path, read_only, timeout)
         self.changes: ChangeTrackerType = Changes(self)
 
     def make_search(self) -> SearchType:

{reader-3.19.dev0 → reader-3.20}/src/reader/_storage/_base.py

@@ -20,7 +20,6 @@ APPLICATION_ID = b'read'
 
 _T = TypeVar('_T')
 
-
 # also used by tests
 CONNECTION_CLS = sqlite3.Connection
 
@@ -52,7 +51,7 @@ class StorageBase:
     chunk_size = 2**8
 
     @wrap_exceptions(message="while opening database")
-    def __init__(self, path: str, timeout: float | None = None):
+    def __init__(self, path: str, read_only: bool, timeout: float | None = None):
         kwargs: dict[str, Any] = {'factory': CONNECTION_CLS}
         if timeout is not None:
             kwargs['timeout'] = timeout
@@ -61,7 +60,7 @@ class StorageBase:
         # has to run for every connection (in every thread),
         # since it's not persisted across connections
         self.factory = _sqlite_utils.LocalConnectionFactory(
-            path, self.setup_db, **kwargs
+            path, self.setup_db, read_only, **kwargs
         )
 
     def get_db(self) -> sqlite3.Connection:

{reader-3.19.dev0 → reader-3.20}/src/reader/_storage/_html_utils.py

@@ -41,7 +41,8 @@ warnings.filterwarnings(
 def strip_html(html: str, features: str | None = None) -> str:
     soup = get_soup(html)
     remove_nontext_elements(soup)
-    return soup.get_text(separator=' ')
+    add_text_attrs_as_elements(soup)
+    return soup.get_text()
 
 
 def get_soup(html: str, features: str | None = None) -> bs4.BeautifulSoup:
@@ -59,4 +60,17 @@ def remove_nontext_elements(soup: bs4.BeautifulSoup) -> None:
     # shows any free-floating text it contains, so we should keep it around.
     #
     for e in soup.select('script, noscript, style, title'):
-        e.replace_with('\n')
+        e.decompose()
+
+
+def add_text_attrs_as_elements(soup: bs4.BeautifulSoup) -> None:
+    for e in soup.select('img[alt], area[alt], [title]'):
+        strings = []
+        if e.name == 'img' or e.name == 'area':
+            if s := e.attrs.get('alt'):
+                strings.append(s)
+        if s := e.attrs.get('title'):
+            if s not in strings:
+                strings.append(s)
+        if strings:
+            e.insert_after(*[soup.new_tag('span', string=f" {s} ") for s in strings])

{reader-3.19.dev0 → reader-3.20}/src/reader/_storage/_sqlite_utils.py

@@ -22,6 +22,7 @@ from contextlib import nullcontext
 from dataclasses import dataclass
 from datetime import datetime
 from datetime import timezone
+from pathlib import Path
 from typing import Any
 from typing import cast
 from typing import no_type_check
@@ -119,6 +120,7 @@ def wrap_exceptions(
         for fragment, op_exc_type in op_exc_types.items():
             if fragment.lower() in e_msg:
                 raise op_exc_type(str(e)) from None
+
         raise exc_type(message) from e
 
     except sqlite3.ProgrammingError as e:
@@ -210,7 +212,6 @@ class IdError(DBError):
 
 db_errors = [DBError, SchemaVersionError, IntegrityError, RequirementError]
 
-
 _DBFunction = Callable[[sqlite3.Connection], None]
 
 
@@ -411,6 +412,9 @@ class UsageError(DBError):
     display_name = "usage error"
 
 
+TEMPORARY_DB_PATHS = {':memory:', ''}
+
+
 class LocalConnectionFactory:
     """Maintain a set of connections to the same database, one per thread.
 
@@ -437,13 +441,19 @@ class LocalConnectionFactory:
     INLINE_OPTIMIZE_TIMEOUT = 0.1
 
     def __init__(
-        self, path: str, setup_db: _DBFunction = lambda _: None, **kwargs: Any
+        self,
+        path: str,
+        setup_db: _DBFunction = lambda _: None,
+        read_only: bool = False,
+        **kwargs: Any,
     ):
         self.path = path
         self.setup_db = setup_db
+        self.read_only = read_only
         self.kwargs = kwargs
         if kwargs.get('uri'):  # pragma: no cover
             raise NotImplementedError("is_private() does not work for uri=True")
+        self.kwargs['uri'] = True
         self.attached: dict[str, str] = {}
         self._local = _LocalConnectionFactoryState()
         self._local.is_creating_thread = True
@@ -456,7 +466,7 @@ class LocalConnectionFactory:
         if db:
             if not self._local.context_stack:
                 if self._should_optimize(self._local.call_count):
-                    self._optimize(db, self.INLINE_OPTIMIZE_TIMEOUT)
+                    self._optimize(db, self.read_only, self.INLINE_OPTIMIZE_TIMEOUT)
                 self._local.call_count += 1
             return db
 
@@ -473,7 +483,11 @@ class LocalConnectionFactory:
             self._other_threads = True
 
         # TODO: remove needless cast (needed after mypy 1.16.0 update)
-        db = cast(sqlite3.Connection, sqlite3.connect(self.path, **self.kwargs))
+        db = cast(
+            sqlite3.Connection,
+            sqlite3.connect(self._make_uri(self.path), **self.kwargs),
+        )
+
         self._local.db = db
         self._local.call_count = 0
 
@@ -488,8 +502,8 @@ class LocalConnectionFactory:
         # but not on pypy (finalizer runs in main thread);
         # also see https://bugs.python.org/issue14073
         self._local.finalizer = weakref.finalize(
-            threading.current_thread(), self._close, db
-        )
+            threading.current_thread(), self._close, db, self.read_only
+        )  # type: ignore[call-arg]
 
         for name, path in self.attached.items():
             self._attach(db, name, path)
@@ -514,10 +528,10 @@ class LocalConnectionFactory:
             self._local.closed = True
 
     @classmethod
-    def _close(cls, db: sqlite3.Connection) -> None:
+    def _close(cls, db: sqlite3.Connection, read_only: bool = False) -> None:
         try:
             try:
-                cls._optimize(db)
+                cls._optimize(db, read_only)
             finally:
                 db.close()
         except sqlite3.ProgrammingError as e:
@@ -532,7 +546,13 @@ class LocalConnectionFactory:
             raise
 
     @staticmethod
-    def _optimize(db: sqlite3.Connection, timeout: float = 0) -> None:
+    def _optimize(
+        db: sqlite3.Connection, read_only: bool = False, timeout: float = 0
+    ) -> None:
+        # don't optimize the database if it's in a read-only state
+        if read_only:
+            return
+
         # Don't wait too much for a lock, it means the database is busy,
         # and now is likely not a good time to run optimize.
         # Also prevents rare "database is locked" errors in certain conditions
@@ -552,6 +572,7 @@ class LocalConnectionFactory:
                 db.execute("PRAGMA optimize;")
         except sqlite3.OperationalError as e:  # pragma: no cover
             message = str(e).lower()
+
             if "database is locked" not in message:
                 raise
 
@@ -581,11 +602,27 @@ class LocalConnectionFactory:
         self._attach(db, name, path)
 
     def _attach(self, db: sqlite3.Connection, name: str, path: str) -> None:
-        db.execute("ATTACH DATABASE ? AS ?;", (path, name))
+        db.execute("ATTACH DATABASE ? AS ?;", (self._make_uri(path), name))
 
     def is_private(self) -> bool:
         return self._is_private(self.path)
 
+    def _make_uri(self, path: str) -> str:
+        # SQLite allows relative paths by skipping the // after file:,
+        # but pathlib.Path.as_uri() does not support this;
+        # however, temporary databases *must* be relative
+        if path in TEMPORARY_DB_PATHS:
+            # arbitrary paths would need to quote ? to avoid qs injection
+            uri = f"file:{path}"
+        else:
+            # bypasses SQLite's resolution of relative paths, should be fine
+            uri = Path(path).absolute().as_uri()
+
+        if self.read_only:
+            uri += "?mode=ro"
+
+        return uri
+
     @staticmethod
     def _is_private(path: str) -> bool:
         """Does connect(path) from another thread connect to a different database?
@@ -603,7 +640,7 @@ class LocalConnectionFactory:
         https://www.sqlite.org/uri.html
 
         """
-        return path in [':memory:', '']
+        return path in TEMPORARY_DB_PATHS
 
 
 class _LocalConnectionFactoryState(threading.local):

{reader-3.19.dev0 → reader-3.20}/src/reader/_types.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import logging
+import time
 from collections import defaultdict
 from collections.abc import Callable
 from collections.abc import Iterable
@@ -672,6 +673,7 @@ class UpdateHooks:
     ) -> list[SingleUpdateHookError]:
         rv = []
         for hook in self.hooks[when]:
+            start = time.monotonic()
             try:
                 hook(self.target, *args)
             except Exception as e:
@@ -680,6 +682,19 @@ class UpdateHooks:
                 if not return_exceptions:
                     raise wrapper
                 rv.append(wrapper)
+            finally:
+                end = time.monotonic()
+                try:
+                    name = hook.__module__ + ':' + hook.__qualname__
+                except AttributeError:
+                    name = repr(hook)
+                log.debug(
+                    "ran %s hook %s for %r in %f seconds",
+                    when,
+                    name,
+                    resource_id,
+                    end - start,
+                )
         return rv
 
     def group(self, message: str) -> _UpdateHookErrorGrouper:

{reader-3.19.dev0 → reader-3.20}/src/reader/core.py

@@ -103,6 +103,7 @@ def make_reader(
     url: str,
     *,
     feed_root: str | None = None,
+    read_only: bool = False,
     plugins: Iterable[PluginInput] = DEFAULT_PLUGINS,
     session_timeout: TimeoutType = DEFAULT_TIMEOUT,
     reserved_name_scheme: Mapping[str, str] = DEFAULT_RESERVED_NAME_SCHEME,
@@ -166,6 +167,9 @@ def make_reader(
             ``'/path/to/feed/root'``
             (an absolute path that feed paths are relative to).
 
+        read_only (bool):
+            Only allow read-only storage operations.
+
         plugins (iterable(str or callable(Reader)) or None):
             An iterable of built-in plugin names or
             `plugin(reader) --> None` callables.
@@ -236,6 +240,9 @@ def make_reader(
         Wrap exceptions raised during plugin initialization
         in :exc:`PluginInitError` instead of letting them bubble up.
 
+    .. versionadded:: 3.20
+        The ``read_only`` keyword argument.
+
     """
 
     # Do as much work as possible before creating the storage.
@@ -263,12 +270,13 @@ def make_reader(
     # See this comment for details on how it should evolve:
     # https://github.com/lemon24/reader/issues/168#issuecomment-642002049
 
-    storage: StorageType = _storage or Storage(url)
+    storage: StorageType = _storage or Storage(url, read_only=read_only)
 
     try:
         # For now, we're using a storage-bound search provider.
         if not isinstance(storage, BoundSearchStorageType):  # pragma: no cover
             raise TypeError("storage must have a make_search factory")
+
         search = storage.make_search()
 
         if search_enabled is True: