reader 3.24__tar.gz → 3.25__tar.gz

{reader-3.24 → reader-3.25}/CHANGES.rst

@@ -6,6 +6,29 @@ Changelog
   :no-index:
 
 
+Version 3.25
+------------
+
+Released 2026-05-25
+
+* Stable feed autodiscovery support. (:issue:`404`)
+
+  * Add :mod:`~reader.plugins.autodiscover` built-in plugin.
+  * Add :mod:`reader.discover` module providing low level discovery.
+  * In the web app, show :mod:`~reader.plugins.autodiscover` suggestions when adding a feed.
+
+* Fix bugs that may cause the 3.24 database migration to fail. (:issue:`405`)
+
+  * Don't fail for ``John Doe (...) (email)`` RSS authors.
+  * Support missing :attr:`EntrySource` ``author`` field during migration.
+
+  Note there should be no integrity issues,
+  as migrations happen inside a transaction.
+
+* Simplify parser lazy initialization (make it explicit and opt-in by plugins).
+* Allow before/after update hook functions to take *fewer* arguments than documented.
+
+
 Version 3.24
 ------------
 

{reader-3.24/src/reader.egg-info → reader-3.25}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reader
-Version: 3.24
+Version: 3.25
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause
@@ -114,6 +114,7 @@ you'd probably call it a fat model.
 * full-text search articles
 * get statistics on feed and user activity
 * import / export feeds as OPML
+* automatically discover feeds in web pages
 * write plugins to extend its functionality
 * skip all the low level stuff and focus on what makes your feed reader different
 

{reader-3.24 → reader-3.25}/README.rst

@@ -50,6 +50,7 @@ you'd probably call it a fat model.
 * full-text search articles
 * get statistics on feed and user activity
 * import / export feeds as OPML
+* automatically discover feeds in web pages
 * write plugins to extend its functionality
 * skip all the low level stuff and focus on what makes your feed reader different
 

{reader-3.24 → reader-3.25}/docs/api.rst

@@ -243,3 +243,9 @@ Utilities
 ---------------
 
 .. automodule:: reader.opml
+
+
+``reader.discover``
+-------------------
+
+.. automodule:: reader.discover

{reader-3.24 → reader-3.25}/docs/conf.py

@@ -23,7 +23,7 @@ for name in [
     'structlog.contextvars',
     'structlog.stdlib',
 ]:
-    sys.modules[name] = unittest.mock.Mock()
+    sys.modules[name] = unittest.mock.MagicMock()
 
 import reader
 

{reader-3.24 → reader-3.25}/docs/install.rst

@@ -13,13 +13,10 @@ Dependencies
 
 These packages will be installed automatically when installing *reader*:
 
-* `feedparser`_ parses feeds; *reader* is essentially feedparser + state.
-* `requests`_ retrieves feeds from the internet;
-  it replaces feedparser's default use of :mod:`urllib`
-  to make it easier to write plugins.
+* `feedparser`_ parses Atom and RSSfeeds.
+* `requests`_ retrieves feeds from the internet.
 * `werkzeug`_ provides HTTP utilities.
-* `beautifulsoup4`_ is used to strip HTML tags before adding entries
-  to the search index.
+* `beautifulsoup4`_ is used to strip HTML tags and by various plugins.
 * `structlog`_ is used for structured logging.
 * `typing-extensions`_ is used for :mod:`typing` backports.
 
@@ -58,8 +55,13 @@ As of version |version|, *reader* has the following extras:
 * Specific plugins may require additional dependencies;
   refer to their documentation for details.
 
+Beautiful Soup supports `multiple parsers`_,
+and will use the best available one;
+for speed, install `lxml`_.
 
 .. _beautifulsoup4: https://www.crummy.com/software/BeautifulSoup/
+.. _multiple parsers: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
+.. _lxml: https://lxml.de/
 .. _feedparser: https://feedparser.readthedocs.io/en/latest/
 .. _requests: https://requests.readthedocs.io/
 .. _werkzeug: https://werkzeug.palletsprojects.com/

{reader-3.24 → reader-3.25}/docs/internal.rst

@@ -31,16 +31,6 @@ Parser
     :special-members: __call__
 
 
-.. module:: reader._parser.requests
-
-.. autoclass:: SessionFactory(...)
-    :members:
-    :special-members: __call__
-
-.. autoclass:: SessionWrapper(...)
-    :members:
-
-
 Protocols
 ~~~~~~~~~
 
@@ -79,18 +69,6 @@ Protocols
     :show-inheritance:
 
 
-.. module:: reader._parser.requests
-    :no-index:
-
-.. autoclass:: RequestHook
-    :members:
-    :special-members: __call__
-
-.. autoclass:: ResponseHook
-    :members:
-    :special-members: __call__
-
-
 Data objects
 ~~~~~~~~~~~~
 
@@ -113,6 +91,35 @@ Data objects
     :members:
 
 
+Built-in retrievers and parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. module:: reader._parser.http
+
+.. autoclass:: HTTPRetriever
+    :members: request_hooks, response_hooks
+
+.. autoclass:: RequestHook
+    :members:
+    :special-members: __call__
+
+.. autoclass:: ResponseHook
+    :members:
+    :special-members: __call__
+
+.. module:: reader._parser.file
+
+.. autoclass:: FileRetriever
+
+.. module:: reader._parser.feedparser
+
+.. autoclass:: FeedparserParser
+
+.. module:: reader._parser.jsonfeed
+
+.. autoclass:: JSONFeedParser
+
+
 .. _storage:
 
 Storage

{reader-3.24 → reader-3.25}/docs/plugins.rst

@@ -16,6 +16,7 @@ This is a list of built-in plugins that are considered stable.
 See the :ref:`Plugins <plugins>` section of the user guide
 for details on how built-in plugins are loaded.
 
+.. automodule:: reader.plugins.autodiscover
 .. automodule:: reader.plugins.enclosure_dedupe
 .. automodule:: reader.plugins.entry_dedupe
 .. automodule:: reader.plugins.mark_as_read

{reader-3.24 → reader-3.25}/pyproject.toml

@@ -186,6 +186,7 @@ module = [
     "reader.plugins.ua_fallback",
     "reader.plugins.readtime",
     "reader.plugins.mark_as_read",
+    "reader.plugins.autodiscover",
     "reader._vendor.feedparser.*",
 ]
 ignore_errors = true

{reader-3.24 → reader-3.25}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.24'
+__version__ = '3.25'
 
 # isort: off
 

{reader-3.24 → reader-3.25}/src/reader/_app/__init__.py

@@ -88,6 +88,15 @@ def entries():
     if feed_url := form.feed.data:
         feed = reader.get_feed(feed_url)
 
+    autodiscover = None
+    if feed and feed.last_exception and not feed.last_updated:
+        autodiscover_tag = reader.make_reader_reserved_name('autodiscover')
+        hostname = urlparse(feed.url).hostname
+        autodiscover = [
+            (link, hostname and urlparse(link['href']).hostname == hostname)
+            for link in reader.get_tag(feed, autodiscover_tag, None)
+        ]
+
     kwargs = dict(form.data)
 
     if not (feed or kwargs.get('starting_after')):
@@ -106,6 +115,7 @@ def entries():
         form=form,
         entries=entries,
         feed=feed,
+        autodiscover=autodiscover,
         limit=limit,
     )
 
@@ -277,6 +287,31 @@ def add_feed():
     return render_template('add_feed.html', form=form)
 
 
+@blueprint.route('/feeds/change', methods=['POST'])
+def change_feed_url():
+    # NOTE: currently only for post-add autodiscover change,
+    # actual "change feed URL" UX needs design (e.g. needs preview)
+
+    reader = get_reader()
+    old = request.form['old']
+    new = request.form['new']
+
+    try:
+        reader.change_feed_url(old, new)
+    except FeedExistsError:
+        flash(f"Feed {new} already exists.", 'secondary')
+        return redirect(url_for('.entries', feed=old), code=303)
+    else:
+        # TODO: updating should be out of band
+        try:
+            reader.update_feed(new)
+        except UpdateError:
+            pass
+        else:
+            flash("Changed and updated feed.", 'success')
+        return redirect(url_for('.entries', feed=new), code=303)
+
+
 @blueprint.route('/feeds/import', methods=['GET', 'POST'])
 def import_feeds():
     reader = get_reader()

{reader-3.24 → reader-3.25}/src/reader/_app/templates/add_feed.html

@@ -13,7 +13,15 @@
 
 <h1>Add feed</h1>
 
-<p>Add a single feed:</p>
+<p>
+  Paste a feed or website URL:
+  <a
+    href="{{ url_for('.help', _anchor=help_topic) }}"
+    class="link-secondary d-inline-block px-2"
+  >
+    <i class="bi bi-question-lg" aria-label="help: adding feeds"></i>
+  </a>
+</p>
 
 <form action="{{ url_for(request.endpoint) }}" method="post" class="my-2 mb-4">
 {#- TODO: aria stuff #}

{reader-3.24 → reader-3.25}/src/reader/_app/templates/entries.html

@@ -18,6 +18,64 @@
 
 {{ macros.flashed_messages() }}
 
+{%- if feed and not feed.url.startswith('reader:') %}
+  <ul class="list-unstyled mt-3">
+
+    {%- if not feed.last_updated and not feed.last_exception %}
+      <li class="alert alert-secondary">
+        Feed not updated yet. <a href="">Refresh.</a>
+      </li>
+    {%- endif %}
+
+    {%- if feed.last_exception %}
+
+      {%- if autodiscover %}
+      <li class="alert alert-primary">
+
+        <p>
+          <strong>This doesn't look like a feed</strong>,
+          but it links to these feeds:
+        </p>
+
+        <ul class="list-unstyled d-table">
+        {%- for link, official in autodiscover | sort(attribute=1, reverse=True) %}
+          <li class="d-table-row mb-1">
+
+            <span class="d-table-cell pe-4 pb-1">
+              {{ macros.link_object(link) }}
+              {%- if not official %}
+                <i class="bi bi-box-arrow-up-right ms-1" aria-label="external feed" title="external feed"></i>
+              {%- endif %}
+            </span>
+
+            <form
+              action="{{ url_for('.change_feed_url') }}"
+              method="post"
+              class="d-table-cell pb-1"
+            >
+              {{ macros.csrf_token_field() }}
+              <input type="hidden" name="old" value="{{ feed.url }}">
+              <input type="hidden" name="new" value="{{ link.href }}">
+              <input type="submit" class="btn btn-primary btn-sm" value="add">
+            </form>
+
+          </li>
+        {%- endfor %}
+        </ul>
+
+      </li>
+      {%- endif %}
+
+      <li class="alert alert-{{ 'danger' if not autodiscover else 'secondary' }}">
+        <strong>Update error</strong>
+        {%- if not feed.last_updated %} (first update){% endif %}:
+        {{ feed.last_exception.value_str }}
+      </li>
+
+    {%- endif %}
+  </ul>
+{%- endif %}
+
 
 {%- if not feed %}
   <h1>Entries</h1>
@@ -34,23 +92,6 @@
 
   {{ macros.subtitle(feed.subtitle) }}
 
-  {%- if feed and not feed.url.startswith('reader:') %}
-    <ul class="list-unstyled mt-3">
-      {%- if not feed.last_updated and not feed.last_exception %}
-        <li class="alert alert-secondary">
-          Feed not updated yet. <a href="">Refresh.</a>
-        </li>
-      {%- endif %}
-      {%- if feed.last_exception %}
-        <li class="alert alert-danger">
-          <strong>Update error</strong>
-          {%- if not feed.last_updated %} (first update){% endif %}:
-          {{ feed.last_exception.value_str }}
-        </li>
-      {%- endif %}
-    </ul>
-  {%- endif %}
-
   <form
     action="{{ url_for('.feed_actions', from='entries') }}"
     method="post"

{reader-3.24 → reader-3.25}/src/reader/_app/templates/entry.html

@@ -114,8 +114,7 @@
 <ul class="card-text list-unstyled">
 {% for enclosure in entry.enclosures %}
 <li>
-<a href="{{ enclosure.href }}">{{ enclosure.href.split('/')[-1].split('?')[0] }}</a>
-<small>{{ enclosure.type }}</small>
+  {{ macros.link_object(enclosure) }}
 </li>
 {% endfor %}
 </ul>

{reader-3.24 → reader-3.25}/src/reader/_app/templates/help.html

@@ -39,11 +39,17 @@
 <h3 id="add-feed">Adding feeds</h3>
 
 <p>
-  To add a feed, paste its link in the
+  To add a feed, paste the feed or website URL on the
   <a href="{{ url_for('.add_feed') }}">Add feed</a> page.
 
 <p>
-  You can usually find if a website has a feed
+  If you use a website URL,
+  <em>reader</em> will discover any standard feeds it links to
+  and suggest them to you.
+
+<p>
+  If that doesn't work,
+  you can usually find if a website has a feed
   by looking in its header or footer for this icon
   <i class="bi bi-rss" aria-label="RSS logo"></i>,
   or for links called <em>feed</em>, <em>RSS</em>, or <em>Atom</em>.

{reader-3.24 → reader-3.25}/src/reader/_app/templates/layout_base.html

@@ -68,7 +68,7 @@
             class="btn btn-link nav-link d-flex align-items-center"
           >
             <i class="bi bi-question-lg"></i>
-            <span class="d-md-none ms-2" id="theme-text">help</span>
+            <span class="d-md-none ms-2">help</span>
           </a>
         </li>
 

{reader-3.24 → reader-3.25}/src/reader/_app/templates/macros.html

@@ -355,3 +355,11 @@
   or learn more about <a href="{{ url_for('.help') }}">how to use <strong>reader</strong></a>.
 </p>
 {%- endmacro -%}
+
+
+{%- macro link_object(link) %}
+{#- TODO: when links are unified, add a property for this #}
+{%- set title = link.title or link.href.rpartition('?')[0].rstrip('/').rpartition('/')[2] %}
+<a href="{{ link.href }}" target="_blank">{{ title }}</a>
+{%- if link.type %}&ensp;<small>{{ link.type }}</small>{% endif %}
+{%- endmacro %}