reader 3.19.dev0__tar.gz → 3.21__tar.gz

{reader-3.19.dev0 → reader-3.21}/CHANGES.rst

@@ -6,18 +6,99 @@ Changelog
   :no-index:
 
 
+Version 3.21
+------------
+
+Released 2026-01-12
+
+* :ref:`Scheduled updates <scheduled>` improvements.
+
+  * Only update scheduled feeds by default
+    (:meth:`update_feeds(scheduled=...) <Reader.update_feeds>`
+    defaults to :const:`True`);
+    previously, all feeds would be updated.
+    (:issue:`379`)
+
+    .. warning::
+
+      **This is a minor compatibility break**;
+      the previous behavior is a bug –
+      *reader* should do the right thing
+      even if the user doesn't read the documentation.
+
+    In the CLI, make ``update --scheduled`` the default.
+
+  * Honor the Cache-Control (max-age) and Expires HTTP headers.
+    (:issue:`376`)
+
+  * Limit the :attr:`~Feed.update_after` derived from HTTP headers
+    to a month in the future.
+    (:issue:`384`)
+
+  * If the Retry-After value is a date,
+    make the update interval relative to Date.
+
+
+
+Version 3.20
+------------
+
+Released 2025-11-26
+
+* Add the ``read_only`` :func:`make_reader` argument.
+  Thanks to `Roman Milko`_ for the PR.
+  (:issue:`319`)
+
+* Include the ``alt`` and ``title`` HTML attributes
+  in the :ref:`full-text search <fts>` index.
+  Also, don't add extra whitespace between tags
+  (especially important for inline tags).
+  (:issue:`371`)
+
+  .. note::
+
+    Old entries will **not** be re-indexed automatically;
+    to do so, disable then enable search.
+
+* :mod:`~reader.plugins.entry_dedupe` improvements. (:issue:`371`)
+
+  * Use more heuristics to find potential duplicates
+    (in addition to title matching):
+    match link, match published timestamp,
+    strip common title prefixes for new entries.
+  * When comparing entries,
+    include the ``alt`` and ``title`` HTML attributes,
+    strip accents, and treat dates and versions as single tokens.
+  * Increase required minimum content length from 32 to 48 words.
+  * Add the ``.reader.dedupe.once.link`` and
+    ``.reader.dedupe.once.title.prefix`` escape hatches.
+  * Clean up documentation, refactor code, and add extensive tests.
+
+* Log run times of before/after update hooks.
+
+.. _Roman Milko: https://github.com/rastr-0
+
+
 Version 3.19
 ------------
 
-Unreleased
+Released 2025-10-01
 
 * Support Python 3.14.
+* Remove deprecated empty ``search`` and ``readtime`` extras.
+* Remove undocumented development ``tests``, ``docs``, and ``dev`` extras.
 
 * :mod:`~reader._plugins.enclosure_tags` plugin:
   strip HTML tags from values used as MP3 tags
   (for example, ``<b>`` tags used to highlight search result matches).
 
-* Remove long-deprecated, empty ``search`` and ``readtime`` extras.
+* Project infrastructure improvements.
+
+  * Clean up :doc:`contributing` documentation and :gh:`run.sh <run.sh>` script.
+  * Use GitHub Actions to publish releases to PyPI.
+  * Split type checking and testing jobs in the ``build`` GHA workflow.
+  * Make tox runs slightly faster.
+  * Use dependency groups instead of extras for development dependencies.
 
 
 Version 3.18

{reader-3.19.dev0/src/reader.egg-info → reader-3.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reader
-Version: 3.19.dev0
+Version: 3.21
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause

{reader-3.19.dev0 → reader-3.21}/docs/dev.rst

@@ -226,25 +226,31 @@ Making a release
 
 .. note::
 
-    :gh:`scripts/release.py <scripts/release.py>` already does most of these.
+    The :gh:`release.py <scripts/release.py>` script together with
+    the :gh:`publish <.github/workflows/publish.yaml>`  GitHub Actions workflow
+    already do most of these.
 
 Making a release (from ``x`` to ``y`` == ``x + 1``):
 
 * (release.py) bump version in ``src/reader/__init__.py`` to ``y``
 * (release.py) update changelog with release version and date
 * (release.py) make sure tests pass / docs build
-* (release.py) clean up dist/: ``rm -rf dist/``
-* (release.py) build tarball and wheel: ``python -m build``
 * (release.py) push to GitHub
-* (release.py prompts) wait for GitHub Actions / Codecov / Read the Docs builds to pass
-* upload to test PyPI and check: ``twine upload --repository-url https://test.pypi.org/legacy/ dist/*``
-* (release.py) upload to PyPI: ``twine upload dist/*``
+* (release.py prompts) wait for GitHub Actions / Read the Docs builds to pass
 * (release.py) tag current commit with `<major>.<minor>` and `<major>.x`
-  (e.g. when releasing `1.20`: `1.20` and `1.x`)
-* (release.py prompts) create release in GitHub
-* build docs from latest and enable ``y`` docs version (should happen automatically after the first time)
+  (e.g. when releasing `1.20`: `1.20` and `1.x`) and push tags to GitHub
+* (release.py prompts) approve publish workflow
+* (publish workflow) build and upload to PyPI
+* alternatively, to make a manual release (should not be needed)
+
+  * clean up dist/: ``rm -rf dist/``
+  * build tarball and wheel: ``python -m build``
+  * upload to test PyPI and check: ``twine upload --repository-url https://test.pypi.org/legacy/ dist/*``
+  * upload to PyPI: ``twine upload dist/*``
+
+* (publish workflow) create draft release in GitHub
+* (release.py prompts) publish release in GitHub
 * (release.py) bump versions from ``y`` to ``(y + 1).dev0``, add ``(y + 1)`` changelog section
-* (release.py prompts) trigger Read the Docs build for `<major>.x` (doesn't happen automatically)
 
 
 
@@ -750,11 +756,8 @@ Thoughts on dynamic lists of feeds: :issue:`165#issuecomment-905893420`.
 Duplicate entries
 ~~~~~~~~~~~~~~~~~
 
-Duplicate entries are mainly handled by the :mod:`reader.entry_dedupe` plugin.
-
-* Using MinHash to speed up similarity checks (maybe): https://gist.github.com/lemon24/b9af5ade919713406bda9603847d32e5
-* Discussion of unifying "on-line" dedupe (after an entry is added/updated),
-  and "on-demand" dedupe (backfill): :issue:`340`.
+Duplicate entries are handled by :mod:`~reader.plugins.entry_dedupe`
+(extensive notes in the :gh:`docstring <src/reader/plugins/entry_dedupe.py>`).
 
 However, it is also possible for a feed to have two entries with the same id
 – yes, even though in most (if not all) formats,

{reader-3.19.dev0 → reader-3.21}/docs/guide.rst

@@ -270,11 +270,12 @@ Scheduled updates
 ~~~~~~~~~~~~~~~~~
 
 Because different feeds need to be updated at different rates,
-*reader* also provides a mechanism for scheduling updates.
+*reader* also provides a mechanism for scheduling updates
+that is enabled by default.
 
 On every update, *reader* determines when a feed should be updated next
 based on a configurable update interval;
-:meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
+:meth:`~Reader.update_feeds`
 updates only the feeds that should be updated at or before the current time.
 
 The interval can be **configured by the user**
@@ -289,9 +290,9 @@ such that updates don't happen exactly at the the same time;
 for example, with a one-hour interval, a jitter of 0.25 means
 updates will occur any time in the first 15 minutes of each hour.
 
-If the server sends the Retry-After HTTP header with
-429 Too Many Requests or 503 Service Unavailable responses,
-:meth:`update_feeds(scheduled=True) <Reader.update_feeds>` will honor it.
+If the server responds with any of the
+Cache-Control max-age, Expires, or Retry-After HTTP headers,
+:meth:`update_feeds(scheduled=True) <Reader.update_feeds>` will honor them.
 
 
 .. note::
@@ -308,6 +309,12 @@ If the server sends the Retry-After HTTP header with
 .. versionchanged:: 3.15
     Honor the Retry-After HTTP header.
 
+.. versionchanged:: 3.21
+    Honor the Cache-Control max-age and Expires HTTP headers.
+
+.. versionchanged:: 3.21
+    Only update scheduled feeds by default.
+
 
 Update status
 ~~~~~~~~~~~~~

{reader-3.19.dev0 → reader-3.21}/pyproject.toml

@@ -92,22 +92,21 @@ unstable-plugins = [
 [dependency-groups]
 
 tests = [
-    "pytest>=4",
+    "pytest>=9",
     "pytest-randomly",
-    "pytest-subtests",
     "pytest-rerunfailures",
     "coverage",
     "requests-mock",
     # mechanicalsoup hard-depends on lxml (see below)
-    'mechanicalsoup; (implementation_name != "pypy" and python_version <= "3.12")',
+    'mechanicalsoup; (implementation_name != "pypy" and python_version <= "3.14")',
     "requests-wsgi-adapter",
     # we want to test search with all known bs4 parsers.
     # lxml usually does not have recent PyPy wheels.
     # lxml usually does not have pre-relase CPython wheels.
-    'lxml; (implementation_name != "pypy" and python_version <= "3.12")',
+    'lxml; (implementation_name != "pypy" and python_version <= "3.14")',
     "html5lib",
     # for bench.py
-    'numpy; (implementation_name != "pypy" and os_name == "posix" and python_version <= "3.12")',
+    'numpy; (implementation_name != "pypy" and os_name == "posix" and python_version <= "3.14")',
 ]
 
 typing = [
@@ -118,6 +117,8 @@ typing = [
 
 docs = [
     "sphinx",
+    # TODO: remove when https://github.com/sphinx-doc/sphinx/issues/14186 is solved
+    "roman_numerals<4.1",
     "sphinx-rtd-theme",
     "click>=7",
     "sphinx-click",
@@ -149,6 +150,7 @@ markers = [
     "slow: mark a test as slow.",
     "requires_lxml: mark a test to only run in places where we have lxml.",
     "apptest: mark a test as a web app test (skipped sometimes).",
+    "noscheduled: force update_feeds(scheduled=...) to false.",
     "noautoclose: skip 'autoclose' autouse fixture for a test."
 ]
 filterwarnings = [

{reader-3.19.dev0 → reader-3.21}/run.sh

@@ -63,7 +63,7 @@ function typing-dev {
 }
 
 function docs-dev {
-    rm -r docs/_build/html
+    rm -rf docs/_build/html
     watch sphinx-build -W docs docs/_build/html "$@"
 }
 

reader-3.21/scripts/entry_dedupe_backtest.py

@@ -0,0 +1,113 @@
+"""
+Run entry_dedupe .dedupe.once for an entire database,
+and parse the logs into a form that makes backtesting easier.
+
+Run dedupe and pretty-print logs:
+
+    $ python scripts/entry_dedupe_backtest.py dedupe.log run
+
+Pretty-print existing logs:
+
+    $ python scripts/entry_dedupe_backtest.py dedupe.log
+
+"""
+
+import logging
+import os
+import re
+import sys
+import time
+
+from reader import make_reader
+from reader.plugins import entry_dedupe
+
+
+DB_ARCHIVE = "_backups/reader.sqlite.2025-10-04.gz"
+
+file = sys.argv[1]
+run = len(sys.argv) > 2 and sys.argv[2] == 'run'
+only = len(sys.argv) > 3 and sys.argv[3]
+
+if run:
+    os.system(f"rm db.*; gzip -dc {DB_ARCHIVE} > db.sqlite")
+    reader = make_reader('db.sqlite')
+    logging.basicConfig(filename=file, filemode='w', level=logging.DEBUG)
+    start = time.monotonic()
+    for feed in reader.get_feeds():
+        if only and not re.search(only, feed.title or '', re.I):
+            continue
+        reader.set_tag(feed, '.reader.dedupe.once')
+        entry_dedupe.after_feed_update(reader, feed)
+    end = time.monotonic()
+    print(f"TIME: {end-start:.3f} seconds", file=sys.stderr)
+    reader.close()
+    os.system(f"rm db.*; gzip -dc {DB_ARCHIVE} > db.sqlite")
+
+reader = make_reader('db.sqlite')
+
+
+def shorten(s, n=48):
+    if not s:
+        return s
+    s = ' '.join(s.split())
+    if len(s) < n:
+        return s
+    return s[:n] + '...'
+
+
+feed = None
+entries = None
+grouper = None
+prev = ''
+
+for l in open(file):
+    if 'skipping:' in prev and 'skipping:' not in l:
+        print()
+    if 'skipping:' not in prev and 'skipping:' in l:
+        print('    HUGE\n')
+    prev = l
+
+    if m := re.search(r"for feed '([^']+)'", l):
+        feed = reader.get_feed(m[1])
+        entries = {e.id: e for e in reader.get_entries(feed=feed)}
+        grouper = None
+        print(f"{feed.title}  {feed.url}\n")
+        continue
+
+    if m := re.search("grouper ([^:]+): all=.*", l):
+        grouper = m[1]
+        print(f"  grouper {grouper}\n")
+        continue
+
+    # if m := re.search(r"found mass duplication pairs \d+ < \d+, skipping", l):
+    # print("    TOO FEW PAIRS\n")
+    # continue
+
+    if m := re.search(r"found group of size \d+ > \d+, skipping: (.*)", l):
+        ids = eval(m[1])
+        e = entries[ids[0]]
+        fragment = ''
+        if 'link' in grouper:
+            fragment = e.link
+        elif 'published' in grouper:
+            fragment = e.published or e.updated
+        print(f"    {len(ids):>3}  {shorten(e.title)}  {fragment}")
+        continue
+
+    if m := re.search(r"grouper \S+: found (\[.*)", l):
+        ids = eval(m[1])
+        if ids:
+            print('    DUPLICATES\n')
+        for ids in ids:
+            for i, id in enumerate(ids):
+                e = entries[id]
+                print(f"    {shorten(e.title, 64)}  {e.id}")
+                if 'link' in grouper:
+                    print(f"      link = {e.link}")
+                if 'published' in grouper:
+                    print(f"      published = {e.published or e.updated}")
+            print()
+        continue
+
+    if m := re.search(r"grouper \S+: group count by size (.*)", l):
+        print(f"    summary  {m[1]}\n")

{reader-3.19.dev0 → reader-3.21}/scripts/release.py

@@ -34,11 +34,6 @@ def run_tox():
     run_no_venv('tox -p all', shell=True)
 
 
-def build():
-    run('rm -rf dist/', shell=True)
-    run('python -m build', shell=True)
-
-
 def path_sub(pattern, repl, path):
     with open(path) as f:
         text = f.read()
@@ -112,10 +107,6 @@ def check_unpushed():
         abort("unpushed changes\n\n{}\n", p.stdout.strip('\n'))
 
 
-def upload_to_pypi():
-    run('twine upload dist/*', shell=True)
-
-
 def add_and_push_tags(tags):
     for tag in tags:
         run(['git', 'tag', '--force', tag])
@@ -150,10 +141,6 @@ def main(version, new_version, date, tox, changelog):
 
     confirm("Wait for GitHub Actions / Read the Docs builds to pass.")
 
-    confirm("Upload to PyPI?")
-    build()
-    upload_to_pypi()
-
     tags = [version]
     version_x = version.partition('.')[0] + '.x'
     if new_version_is_final:
@@ -162,7 +149,15 @@ def main(version, new_version, date, tox, changelog):
     confirm(f"Add and push tags ({', '.join(tags)})?")
     add_and_push_tags(tags)
 
-    confirm("Create release {} in GitHub (doesn't happen automatically).", version)
+    confirm(
+        "Approve publish workflow to upload to PyPI.\n  {}\nDone?",
+        "https://github.com/lemon24/reader/actions/workflows/publish.yaml",
+    )
+    confirm(
+        "Publish draft release {} in GitHub.\n  {}\nDone?",
+        version,
+        "https://github.com/lemon24/reader/releases",
+    )
 
     new_version_full = f"{new_version}.dev0"
     update_init_version(new_version_full)
@@ -173,11 +168,6 @@ def main(version, new_version, date, tox, changelog):
     confirm("Push version {}?", new_version_full)
     push()
 
-    # TODO: I just enabled branch or tag creation/deletion for the RtD webhook in GitHub, this might not be needed next time.
-    confirm(
-        f"Trigger Read the Docs build for {version_x} (doesn't happen automatically)."
-    )
-
 
 if __name__ == '__main__':
     main()

{reader-3.19.dev0 → reader-3.21}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.19.dev0'
+__version__ = '3.21'
 
 # isort: off
 

{reader-3.19.dev0 → reader-3.21}/src/reader/_app/v2/forms.py

@@ -116,7 +116,8 @@ ENTRY_SORT_CHOICES = ['recent', 'random']
 class EntryFilter(PresetsMixin, Form):
     feed = HiddenField("feed")
     search = SearchField("search", name='Q')
-    feed_tags = TagFilterField("tags", name='tags')
+    feed_tags = TagFilterField("feed tags", name='tags')
+    tags = TagFilterField("entry tags", name='entry-tags')
     read = radio_field("read", choices=BOOL_CHOICES, default='no')
     important = radio_field("important", choices=TRISTATE_CHOICES, default='maybe')
     has_enclosures = radio_field(

{reader-3.19.dev0 → reader-3.21}/src/reader/_app/v2/templates/v2/entries.html

@@ -52,9 +52,11 @@
   {# TODO: if data or error #}
   {% if form.search.data %}{{ macros.input(form.search, 'search') }}{% endif %}
   {% if form.feed_tags.data %}{{ macros.input(form.feed_tags, 'tag') }}{% endif %}
+  {% if form.feed_tags.data %}{{ macros.input(form.tags, 'tag') }}{% endif %}
   <div class="collapse mb-3" id="nav-more">
     {% if not form.search.data %}{{ macros.input(form.search, 'search') }}{% endif %}
     {% if not form.feed_tags.data %}{{ macros.input(form.feed_tags, 'tag') }}{% endif %}
+    {% if not form.feed_tags.data %}{{ macros.input(form.tags, 'tag') }}{% endif %}
     <div class="mb-3">
     {{ macros.radio(form.read) }}
     {{ macros.radio(form.important) }}

{reader-3.19.dev0 → reader-3.21}/src/reader/_cli.py

@@ -290,7 +290,10 @@ def iter_update_status(it, length):
     help="Only update new (never updated before) feeds.",
 )
 @click.option(
-    '--scheduled/--no-scheduled', help="Only update feeds scheduled to be updated."
+    '--scheduled/--no-scheduled',
+    default=True,
+    show_default=True,
+    help="Only update feeds scheduled to be updated.",
 )
 @click.option(
     '--workers',

{reader-3.19.dev0 → reader-3.21}/src/reader/_parser/__init__.py

@@ -8,6 +8,7 @@ from contextlib import contextmanager
 from dataclasses import dataclass
 from datetime import datetime
 from datetime import timedelta
+from datetime import timezone
 from typing import Any
 from typing import cast
 from typing import ContextManager
@@ -34,6 +35,8 @@ from .requests import TimeoutType
 
 
 if TYPE_CHECKING:  # pragma: no cover
+    from werkzeug.datastructures import RequestCacheControl
+
     from ._lazy import Parser as Parser
 
 
@@ -189,28 +192,64 @@ class HTTPInfo(_namedtuple_compat):
     #: The HTTP response headers.
     headers: Headers
 
-    @property
-    def retry_after(self) -> datetime | timedelta | None:
-        """Parsed Retry-After header.
+    def get_update_after(self, now: datetime) -> datetime | None:
+        """Select the best "update after" date from available headers."""
+        rv = []
+
+        if self.status in (429, 503):
+            try:
+                seconds = int(self.headers.get('retry-after', ''))
+                rv.append(now + timedelta(seconds=seconds))
+            except ValueError:
+                if retry_after := self.parse_date('retry-after', now):
+                    rv.append(retry_after)
+
+        # https://httpwg.org/specs/rfc9111.html#calculating.freshness.lifetime
+        if cache_control := self.cache_control:
+            if not cache_control.no_cache:
+                if max_age := cache_control.max_age:
+                    rv.append(now + timedelta(seconds=max_age))
+        elif expires := self.parse_date('expires', now):
+            rv.append(expires)
+
+        # TODO: RFC 9111 specifies a Last-Modified fallback heuristic,
+        # but it might be better to implement it in the updater
+        # as part of https://github.com/lemon24/reader/issues/382
+
+        return max(rv, default=None)
+
+    def parse_date(self, name: str, now: datetime | None = None) -> datetime | None:
+        """Parse an HTTP date header and return a timezone-aware datetime.
+
+        Return None if missing or if parsing fails.
 
-        None if parsing fails (instead of raising an exception).
-        Always a timezone-aware datetime object;
-        if timezone information is missing, UTC is assumed.
+        If `now` is given and the Date header is set,
+        make the returned value relative to `now`.
 
         """
         # lazy import
         from ._http_utils import parse_date
 
-        value = self.headers.get('retry-after')
+        if value := parse_date(self.headers.get(name, '')):
+            value = value.astimezone(timezone.utc)
+            if now and (date := parse_date(self.headers.get('date', ''))):
+                value = now + (value - date)
+            return value
+
+        return None
+
+    @property
+    def cache_control(self) -> RequestCacheControl | None:
+        """Parsed Cache-Control header, or None if missing."""
+
+        # lazy import
+        from ._http_utils import parse_cache_control_header
+
+        value = self.headers.get('cache-control')
         if not value:
             return None
 
-        try:
-            seconds = int(value)
-        except ValueError:
-            return parse_date(value)
-
-        return timedelta(seconds=seconds)
+        return parse_cache_control_header(value)
 
 
 class RetrieveError(ParseError):

{reader-3.19.dev0 → reader-3.21}/src/reader/_parser/_http_utils.py

@@ -4,8 +4,11 @@ HTTP utilities. Contains no business logic.
 """
 
 from collections.abc import Iterable
+from functools import partial
 
 import werkzeug.http
+from werkzeug.datastructures import MIMEAccept
+from werkzeug.datastructures import ResponseCacheControl
 
 
 parse_options_header = werkzeug.http.parse_options_header
@@ -14,4 +17,9 @@ parse_date = werkzeug.http.parse_date
 
 
 def unparse_accept_header(values: Iterable[tuple[str, float]]) -> str:
-    return werkzeug.datastructures.MIMEAccept(values).to_header()
+    return MIMEAccept(values).to_header()
+
+
+parse_cache_control_header = partial(
+    werkzeug.http.parse_cache_control_header, cls=ResponseCacheControl
+)

{reader-3.19.dev0 → reader-3.21}/src/reader/_storage/__init__.py

@@ -29,8 +29,10 @@ class Storage(FeedsMixin, EntriesMixin, TagsMixin, StorageBase):
 
     """
 
-    def __init__(self, path: str, timeout: float | None = None):
-        super().__init__(path, timeout)
+    def __init__(
+        self, path: str, read_only: bool = False, timeout: float | None = None
+    ):
+        super().__init__(path, read_only, timeout)
         self.changes: ChangeTrackerType = Changes(self)
 
     def make_search(self) -> SearchType:

{reader-3.19.dev0 → reader-3.21}/src/reader/_storage/_base.py

@@ -20,7 +20,6 @@ APPLICATION_ID = b'read'
 
 _T = TypeVar('_T')
 
-
 # also used by tests
 CONNECTION_CLS = sqlite3.Connection
 
@@ -52,7 +51,7 @@ class StorageBase:
     chunk_size = 2**8
 
     @wrap_exceptions(message="while opening database")
-    def __init__(self, path: str, timeout: float | None = None):
+    def __init__(self, path: str, read_only: bool, timeout: float | None = None):
         kwargs: dict[str, Any] = {'factory': CONNECTION_CLS}
         if timeout is not None:
             kwargs['timeout'] = timeout
@@ -61,7 +60,7 @@ class StorageBase:
         # has to run for every connection (in every thread),
         # since it's not persisted across connections
         self.factory = _sqlite_utils.LocalConnectionFactory(
-            path, self.setup_db, **kwargs
+            path, self.setup_db, read_only, **kwargs
         )
 
     def get_db(self) -> sqlite3.Connection:

{reader-3.19.dev0 → reader-3.21}/src/reader/_storage/_html_utils.py

@@ -41,7 +41,8 @@ warnings.filterwarnings(
 def strip_html(html: str, features: str | None = None) -> str:
     soup = get_soup(html)
     remove_nontext_elements(soup)
-    return soup.get_text(separator=' ')
+    add_text_attrs_as_elements(soup)
+    return soup.get_text()
 
 
 def get_soup(html: str, features: str | None = None) -> bs4.BeautifulSoup:
@@ -59,4 +60,17 @@ def remove_nontext_elements(soup: bs4.BeautifulSoup) -> None:
     # shows any free-floating text it contains, so we should keep it around.
     #
     for e in soup.select('script, noscript, style, title'):
-        e.replace_with('\n')
+        e.decompose()
+
+
+def add_text_attrs_as_elements(soup: bs4.BeautifulSoup) -> None:
+    for e in soup.select('img[alt], area[alt], [title]'):
+        strings = []
+        if e.name == 'img' or e.name == 'area':
+            if s := e.attrs.get('alt'):
+                strings.append(s)
+        if s := e.attrs.get('title'):
+            if s not in strings:
+                strings.append(s)
+        if strings:
+            e.insert_after(*[soup.new_tag('span', string=f" {s} ") for s in strings])