reader 3.24__tar.gz → 3.26__tar.gz

{reader-3.24 → reader-3.26}/CHANGES.rst

@@ -6,6 +6,42 @@ Changelog
   :no-index:
 
 
+Version 3.26
+------------
+
+Released 2026-06-24
+
+* Allow exporting the entire database (e.g. for backup or migration).
+
+  * Through the :meth:`~reader._types.StorageType.export` storage method.
+  * In the web app, on the settings page; :ref:`screenshots <screenshots-export>`.
+  * In the CLI, as a new ``export`` command.
+
+
+Version 3.25
+------------
+
+Released 2026-05-25
+
+* Stable feed autodiscovery support. (:issue:`404`)
+
+  * Add :mod:`~reader.plugins.autodiscover` built-in plugin.
+  * Add :mod:`reader.discover` module providing low level discovery.
+  * In the web app, show :mod:`~reader.plugins.autodiscover` suggestions
+    when adding a feed; :ref:`screenshots <screenshots-autodiscovery>`.
+
+* Fix bugs that may cause the 3.24 database migration to fail. (:issue:`405`)
+
+  * Don't fail for ``John Doe (...) (email)`` RSS authors.
+  * Support missing :attr:`EntrySource` ``author`` field during migration.
+
+  Note there should be no integrity issues,
+  as migrations happen inside a transaction.
+
+* Simplify parser lazy initialization (make it explicit and opt-in by plugins).
+* Allow before/after update hook functions to take *fewer* arguments than documented.
+
+
 Version 3.24
 ------------
 

{reader-3.24/src/reader.egg-info → reader-3.26}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reader
-Version: 3.24
+Version: 3.26
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause
@@ -49,6 +49,8 @@ Provides-Extra: app
 Requires-Dist: flask>=0.10; extra == "app"
 Requires-Dist: humanize>=4; extra == "app"
 Requires-Dist: PyYAML; extra == "app"
+Requires-Dist: click; extra == "app"
+Requires-Dist: platformdirs; extra == "app"
 Requires-Dist: WTForms; extra == "app"
 Requires-Dist: Flask-WTF; extra == "app"
 Requires-Dist: jinja2-fragments; extra == "app"
@@ -114,6 +116,7 @@ you'd probably call it a fat model.
 * full-text search articles
 * get statistics on feed and user activity
 * import / export feeds as OPML
+* automatically discover feeds in web pages
 * write plugins to extend its functionality
 * skip all the low level stuff and focus on what makes your feed reader different
 

{reader-3.24 → reader-3.26}/README.rst

@@ -50,6 +50,7 @@ you'd probably call it a fat model.
 * full-text search articles
 * get statistics on feed and user activity
 * import / export feeds as OPML
+* automatically discover feeds in web pages
 * write plugins to extend its functionality
 * skip all the low level stuff and focus on what makes your feed reader different
 

{reader-3.24 → reader-3.26}/docs/api.rst

@@ -243,3 +243,9 @@ Utilities
 ---------------
 
 .. automodule:: reader.opml
+
+
+``reader.discover``
+-------------------
+
+.. automodule:: reader.discover

{reader-3.24 → reader-3.26}/docs/app.rst

@@ -19,6 +19,8 @@ Features, as of *reader* version |version|:
 * article view
 * custom feed titles
 * dark mode
+* feed autodiscovery
+* full data export
 
 More features already implemented in the library
 and the :ref:`legacy web app <legacy web app>`
@@ -86,6 +88,20 @@ Screenshots
 
     contextual help
 
+.. _screenshots-autodiscovery:
+
+.. figure:: screenshots/autodiscovery.png
+    :width: 240px
+
+    feed autodiscovery
+
+.. _screenshots-export:
+
+.. figure:: screenshots/export-database.png
+    :width: 240px
+
+    database exports
+
 
 Serving the web app
 -------------------

{reader-3.24 → reader-3.26}/docs/conf.py

@@ -23,7 +23,7 @@ for name in [
     'structlog.contextvars',
     'structlog.stdlib',
 ]:
-    sys.modules[name] = unittest.mock.Mock()
+    sys.modules[name] = unittest.mock.MagicMock()
 
 import reader
 

{reader-3.24 → reader-3.26}/docs/guide.rst

@@ -139,15 +139,18 @@ since each connection would be to a *different* database::
 
 .. _backups:
 
-Back-ups
-~~~~~~~~
+Backups
+~~~~~~~
 
-Making back-ups depends on the storage used.
+Making backups generally depends on the storage used.
 
-For the SQLite storage, you should use
-the `sqlite3 .backup command <https://sqlite.org/cli.html>`_
-or `VACUUM INTO <https://www.sqlite.org/lang_vacuum.html>`_
-(see :gh:`backup.sh <scripts/backup.sh>` for an example).
+If the storage implements it, you can use
+the :meth:`~reader._types.StorageType.export` method
+or the ``export`` :doc:`CLI <cli>` command.
+
+For the SQLite storage, you can also use
+the `.backup command <https://sqlite.org/cli.html>`_
+or `VACUUM INTO <https://www.sqlite.org/lang_vacuum.html>`_.
 
 
 

{reader-3.24 → reader-3.26}/docs/install.rst

@@ -13,13 +13,10 @@ Dependencies
 
 These packages will be installed automatically when installing *reader*:
 
-* `feedparser`_ parses feeds; *reader* is essentially feedparser + state.
-* `requests`_ retrieves feeds from the internet;
-  it replaces feedparser's default use of :mod:`urllib`
-  to make it easier to write plugins.
+* `feedparser`_ parses Atom and RSSfeeds.
+* `requests`_ retrieves feeds from the internet.
 * `werkzeug`_ provides HTTP utilities.
-* `beautifulsoup4`_ is used to strip HTML tags before adding entries
-  to the search index.
+* `beautifulsoup4`_ is used to strip HTML tags and by various plugins.
 * `structlog`_ is used for structured logging.
 * `typing-extensions`_ is used for :mod:`typing` backports.
 
@@ -58,8 +55,13 @@ As of version |version|, *reader* has the following extras:
 * Specific plugins may require additional dependencies;
   refer to their documentation for details.
 
+Beautiful Soup supports `multiple parsers`_,
+and will use the best available one;
+for speed, install `lxml`_.
 
 .. _beautifulsoup4: https://www.crummy.com/software/BeautifulSoup/
+.. _multiple parsers: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
+.. _lxml: https://lxml.de/
 .. _feedparser: https://feedparser.readthedocs.io/en/latest/
 .. _requests: https://requests.readthedocs.io/
 .. _werkzeug: https://werkzeug.palletsprojects.com/

{reader-3.24 → reader-3.26}/docs/internal.rst

@@ -31,16 +31,6 @@ Parser
     :special-members: __call__
 
 
-.. module:: reader._parser.requests
-
-.. autoclass:: SessionFactory(...)
-    :members:
-    :special-members: __call__
-
-.. autoclass:: SessionWrapper(...)
-    :members:
-
-
 Protocols
 ~~~~~~~~~
 
@@ -79,18 +69,6 @@ Protocols
     :show-inheritance:
 
 
-.. module:: reader._parser.requests
-    :no-index:
-
-.. autoclass:: RequestHook
-    :members:
-    :special-members: __call__
-
-.. autoclass:: ResponseHook
-    :members:
-    :special-members: __call__
-
-
 Data objects
 ~~~~~~~~~~~~
 
@@ -113,6 +91,35 @@ Data objects
     :members:
 
 
+Built-in retrievers and parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. module:: reader._parser.http
+
+.. autoclass:: HTTPRetriever
+    :members: request_hooks, response_hooks
+
+.. autoclass:: RequestHook
+    :members:
+    :special-members: __call__
+
+.. autoclass:: ResponseHook
+    :members:
+    :special-members: __call__
+
+.. module:: reader._parser.file
+
+.. autoclass:: FileRetriever
+
+.. module:: reader._parser.feedparser
+
+.. autoclass:: FeedparserParser
+
+.. module:: reader._parser.jsonfeed
+
+.. autoclass:: JSONFeedParser
+
+
 .. _storage:
 
 Storage

{reader-3.24 → reader-3.26}/docs/plugins.rst

@@ -16,6 +16,7 @@ This is a list of built-in plugins that are considered stable.
 See the :ref:`Plugins <plugins>` section of the user guide
 for details on how built-in plugins are loaded.
 
+.. automodule:: reader.plugins.autodiscover
 .. automodule:: reader.plugins.enclosure_dedupe
 .. automodule:: reader.plugins.entry_dedupe
 .. automodule:: reader.plugins.mark_as_read

{reader-3.24 → reader-3.26}/pyproject.toml

@@ -65,8 +65,11 @@ cli = [
 app = [
     "flask>=0.10",
     "humanize>=4",
-    # for config
+    # for legacy (tags)
     "PyYAML",
+    # for cli
+    "click",
+    "platformdirs",
     # for v2
     "WTForms",
     "Flask-WTF",
@@ -186,6 +189,7 @@ module = [
     "reader.plugins.ua_fallback",
     "reader.plugins.readtime",
     "reader.plugins.mark_as_read",
+    "reader.plugins.autodiscover",
     "reader._vendor.feedparser.*",
 ]
 ignore_errors = true

{reader-3.24 → reader-3.26}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.24'
+__version__ = '3.26'
 
 # isort: off
 

{reader-3.24 → reader-3.26}/src/reader/_app/__init__.py

@@ -19,6 +19,7 @@ from flask import render_template
 from flask import render_template_string
 from flask import request
 from flask import Response
+from flask import send_from_directory
 from flask import stream_with_context
 from flask import url_for
 from flask_wtf.csrf import CSRFError
@@ -35,6 +36,9 @@ from reader import InvalidFeedURLError
 from reader import opml
 from reader import UpdateError
 
+from .exports import ExportNotFoundError
+from .exports import TooManyExportsError
+from .ext import get_exports
 from .ext import get_reader
 from .ext import ReaderExtension
 from .forms import AddFeed
@@ -88,6 +92,15 @@ def entries():
     if feed_url := form.feed.data:
         feed = reader.get_feed(feed_url)
 
+    autodiscover = None
+    if feed and feed.last_exception and not feed.last_updated:
+        autodiscover_tag = reader.make_reader_reserved_name('autodiscover')
+        hostname = urlparse(feed.url).hostname
+        autodiscover = [
+            (link, hostname and urlparse(link['href']).hostname == hostname)
+            for link in reader.get_tag(feed, autodiscover_tag, ())
+        ]
+
     kwargs = dict(form.data)
 
     if not (feed or kwargs.get('starting_after')):
@@ -106,6 +119,7 @@ def entries():
         form=form,
         entries=entries,
         feed=feed,
+        autodiscover=autodiscover,
         limit=limit,
     )
 
@@ -277,6 +291,31 @@ def add_feed():
     return render_template('add_feed.html', form=form)
 
 
+@blueprint.route('/feeds/change', methods=['POST'])
+def change_feed_url():
+    # NOTE: currently only for post-add autodiscover change,
+    # actual "change feed URL" UX needs design (e.g. needs preview)
+
+    reader = get_reader()
+    old = request.form['old']
+    new = request.form['new']
+
+    try:
+        reader.change_feed_url(old, new)
+    except FeedExistsError:
+        flash(f"Feed {new} already exists.", 'secondary')
+        return redirect(url_for('.entries', feed=old), code=303)
+    else:
+        # TODO: updating should be out of band
+        try:
+            reader.update_feed(new)
+        except UpdateError:
+            pass
+        else:
+            flash("Changed and updated feed.", 'success')
+        return redirect(url_for('.entries', feed=new), code=303)
+
+
 @blueprint.route('/feeds/import', methods=['GET', 'POST'])
 def import_feeds():
     reader = get_reader()
@@ -321,6 +360,38 @@ def help():
     return render_template('help.html')
 
 
+@blueprint.route('/settings')
+def settings():
+    return render_template('settings.html', exports=get_exports().list())
+
+
+@blueprint.route('/settings/exports', methods=['POST'])
+def create_export():
+    try:
+        get_exports().create()
+    except TooManyExportsError:
+        flash("Too many exports, delete some of them first.", 'exports.danger')
+    return redirect(url_for('.settings', _anchor='exports'), code=303)
+
+
+@blueprint.route('/settings/exports/<name>')
+def get_export(name):
+    exports = get_exports()
+    if not exports.exists(name):
+        abort(404)
+    return send_from_directory(exports.path, name, as_attachment=True)
+
+
+@blueprint.route('/settings/exports/<name>/delete', methods=['POST'])
+def delete_export(name):
+    # flask should ensure <name> doesn't contain slashes etc.
+    try:
+        get_exports().delete(name)
+    except ExportNotFoundError:
+        abort(404)
+    return redirect(url_for('.settings', _anchor='exports'), code=303)
+
+
 def stream_template(template_name_or_list, **kwargs):
     # Ensure flashed messages get removed from the session,
     # otherwise they keep adding up and never disappear.

{reader-3.24 → reader-3.26}/src/reader/_app/cli.py

@@ -1,5 +1,9 @@
+import pathlib
+
 import click
+import platformdirs
 
+from reader._cli import app_name
 from reader._cli import load_reader_config_from_context
 from reader._cli import setup_logging
 
@@ -26,9 +30,16 @@ def make_add_response_headers_middleware(wsgi_app, headers):
     help="Import path to a web app plug-in. Can be passed multiple times.",
 )
 @click.option('--legacy/--no-legacy', help="Serve the legacy app.")
+@click.option(
+    '--cache-dir',
+    type=click.Path(file_okay=False, path_type=pathlib.Path),
+    default=platformdirs.user_cache_dir(app_name),
+    show_default=True,
+    help="The directory to store cache data in.",
+)
 @click.option('-v', '--verbose', count=True)
 @click.pass_context
-def cli(ctx, host, port, plugins, legacy, verbose):
+def cli(ctx, host, port, plugins, legacy, cache_dir, verbose):
     """Run a local HTTP reader server."""
 
     setup_logging(verbose)

reader-3.26/src/reader/_app/exports.py

@@ -0,0 +1,58 @@
+import pathlib
+from datetime import datetime
+from datetime import timezone
+
+
+class TooManyExportsError(Exception):
+    pass
+
+
+class ExportNotFoundError(Exception):
+    pass
+
+
+class Exports:
+
+    max_files = 2
+    prefix_fmt = "reader.%Y-%m-%d-%H-%M-%S"
+
+    def __init__(self, reader, path):
+        self.reader = reader
+        self.path = pathlib.Path(path)
+        self.path.mkdir(parents=True, exist_ok=True)
+
+    def parse_name(self, name):
+        name_fmt = self.reader._storage.export_name(self.prefix_fmt)
+        try:
+            dt = datetime.strptime(name, name_fmt)
+        except ValueError:
+            return None
+        else:
+            return dt.replace(tzinfo=timezone.utc)
+
+    def create(self):
+        # not looking at file names accounts for exports in progress
+        if len(list(self.path.iterdir())) >= self.max_files:
+            raise TooManyExportsError
+        prefix = self.reader._now().strftime(self.prefix_fmt)
+        self.reader._storage.export(self.path, prefix)
+
+    def exists(self, name):
+        if not self.parse_name(name):
+            return False
+        return (self.path / name).exists()
+
+    def list(self):
+        rv = []
+        for path in self.path.iterdir():
+            if dt := self.parse_name(path.name):
+                rv.append((path, dt))
+        return rv
+
+    def delete(self, name):
+        if not self.parse_name(name):
+            raise ExportNotFoundError
+        try:
+            (self.path / name).unlink()
+        except FileNotFoundError:
+            raise ExportNotFoundError from None

{reader-3.24 → reader-3.26}/src/reader/_app/ext.py

@@ -3,6 +3,7 @@ Flask extension that encapsulates configurable reader-related aspects.
 
 """
 
+import pathlib
 from dataclasses import dataclass
 
 from flask import current_app
@@ -10,13 +11,20 @@ from flask import current_app
 from reader import make_reader
 from reader.plugins._loader import PluginLoader
 
-_plugin_loader = PluginLoader('init_app')
+from .exports import Exports
 
 
 def get_reader():
     return current_app.extensions['reader'].get_reader()
 
 
+def get_exports():
+    return current_app.extensions['reader'].get_exports()
+
+
+_plugin_loader = PluginLoader('init_app')
+
+
 class ReaderExtension:
 
     def __init__(self, app=None, state_cls=None):
@@ -26,26 +34,45 @@ class ReaderExtension:
 
     def init_app(self, app):
         config = app.config['READER']
-        app.extensions['reader'] = self.state_cls(config[''])
+        app.extensions['reader'] = self.state_cls(
+            config[''], config.get('web', {}).get('cache_dir')
+        )
         _plugin_loader.oneshot(app, config.get('web', {}).get('plugins', ()))
 
 
 @dataclass
 class ReaderStateBase:
     args: dict
+    cache_dir: pathlib.Path | None
 
     def get_reader(self):
         raise NotImplementedError
 
+    def get_user_cache_dir(self):
+        raise NotImplementedError
+
     def _make_reader(self, **kwargs):
         return make_reader(**self.args | kwargs)
 
+    def get_exports(self):
+        cache_dir = self.get_user_cache_dir()
+        if not cache_dir:
+            return None
+        return Exports(self.get_reader(), cache_dir / 'exports')
+
 
 class _ReaderState(ReaderStateBase):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self._reader = self._make_reader()
+        self._exports = super().get_exports()
 
     def get_reader(self):
         return self._reader
+
+    def get_user_cache_dir(self):
+        return self.cache_dir
+
+    def get_exports(self):
+        return self._exports

{reader-3.24 → reader-3.26}/src/reader/_app/static/style.css

@@ -69,8 +69,6 @@
   height: auto;
 }
 
-
-:target {
-  outline: 0;
-  box-shadow: var(--bs-focus-ring-x, 0) var(--bs-focus-ring-y, 0) var(--bs-focus-ring-blur, 0) var(--bs-focus-ring-width) var(--bs-focus-ring-color);
+h1:target, h2:target, h3:target, h4:target, h5:target, h6:target {
+  text-decoration: underline;
 }

{reader-3.24 → reader-3.26}/src/reader/_app/templates/add_feed.html

@@ -13,7 +13,15 @@
 
 <h1>Add feed</h1>
 
-<p>Add a single feed:</p>
+<p>
+  Paste a feed or website URL:
+  <a
+    href="{{ url_for('.help', _anchor=help_topic) }}"
+    class="link-secondary d-inline-block px-2"
+  >
+    <i class="bi bi-question-lg" aria-label="help: adding feeds"></i>
+  </a>
+</p>
 
 <form action="{{ url_for(request.endpoint) }}" method="post" class="my-2 mb-4">
 {#- TODO: aria stuff #}