reader 3.14__tar.gz → 3.16__tar.gz

{reader-3.14 → reader-3.16}/CHANGES.rst

@@ -3,7 +3,96 @@ Changelog
 =========
 
 .. module:: reader
-  :noindex:
+  :no-index:
+
+
+Version 3.16
+------------
+
+Released 2024-12-08
+
+* Parse and store :attr:`Entry.source` for Atom and RSS feeds. (:issue:`267`)
+
+  * Add :attr:`Entry.feed_resolved_title` and :attr:`Feed.resolved_title` properties.
+  * The ``feed`` search column now indexes :attr:`Entry.feed_resolved_title`,
+    instead of feed :attr:`~Feed.user_title` or :attr:`~Feed.title`.
+  * Allow filtering entries by the entry source.
+
+* Add :func:`reader.utils.archive_entries` for
+  copying entries to an "archived" feed.
+  (:issue:`290`)
+
+  * Add :meth:`~Reader.copy_entry`.
+  * Allow archiving entries from the web app.
+
+* Fix bug causing :class:`Reader` operations
+  from a thread other than the one that created the instance
+  to happen with foreign key constraint enforcement disabled
+  (e.g. deleting a feed from another thread would not delete its entries).
+
+  This bug exists since using :class:`Reader` instances from other threads
+  became allowed in `2.15 <Version 2.15_>`_.
+
+  Serving the web application with ``python -m reader serve``
+  is known to be affected.
+  Serving it with uWSGI without threads (the default)
+  should not be affected.
+
+  .. attention::
+
+    **Your database may be in an inconsistent state because of this bug.**
+
+    It is recommended you run `PRAGMA foreign_key_check`_ on your database.
+
+    If you are upgrading from a version prior to 3.16
+    (i.e. were not using a pre-release version of *reader*),
+    the migration will do so for you.
+    If there are inconsistencies, you will get this error::
+
+      StorageError: integrity error: after migrating to version 43:
+        integrity error: FOREIGN KEY constraint failed
+
+* Fix :meth:`~Reader.enable_search` / :meth:`~Reader.update_search`
+  not working when the search database is missing but change tracking is enabled
+  (e.g. when restoring the main database from backup).
+  (:issue:`362`)
+
+.. _PRAGMA foreign_key_check: https://www.sqlite.org/pragma.html#pragma_foreign_key_check
+
+
+Version 3.15
+------------
+
+Released 2024-11-11
+
+* During :ref:`scheduled updates <scheduled>`,
+  honor the Retry-After HTTP header sent with
+  429 Too Many Requests or 503 Service Unavailable responses.
+  (:issue:`307`)
+* Optimize OR-only :meth:`~Reader.get_feeds()` and :meth:`~Reader.get_entries()`
+  tag filters (e.g. ``[['one', ...]]``). (:issue:`359`)
+* Expose :data:`reader.core.DEFAULT_RESERVED_NAME_SCHEME`. (:issue:`287`)
+
+* Update the (unstable) parser API to expose HTTP information to the updater.
+  (:issue:`307`)
+
+  * The :class:`.RetrieverType` protocol used by retrievers changed
+    (new return type, allow storing arbitrary caching data via
+    :attr:`~.RetrievedFeed.caching_info`).
+  * The ``HTTPAcceptParserType`` was renamed to :class:`.AcceptParserType`.
+
+* Allow :ref:`using the installed feedparser <no-vendored-feedparser>`
+  instead of the one vendored by *reader*.
+  This is useful for working around issues in the vendored feedparser,
+  like the libxml2 <=2.13.3 incompatibility reported in :issue:`350`;
+  thanks to `Maks Verver`_  for reporting,
+  root causing, and following up with both dependencies.
+
+* Fix a number of brittle / broken tests.
+  Thanks to `Maks Verver`_ for the issues and fixes.
+  (:issue:`348`, :issue:`349`, :issue:`355`)
+
+.. _Maks Verver: https://github.com/maksverver
 
 
 Version 3.14
@@ -650,6 +739,8 @@ Released 2022-03-12
   Fix broken error flashing.
 
 
+.. _version 2.9:
+
 Version 2.9
 -----------
 

{reader-3.14/src/reader.egg-info → reader-3.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: reader
-Version: 3.14
+Version: 3.16
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause

{reader-3.14 → reader-3.16}/docs/api.rst

@@ -16,7 +16,7 @@ Most of *reader*'s functionality can be accessed through a :class:`Reader` insta
 
     Split this into sub-sections.
 
-.. autofunction:: make_reader(url, *, feed_root=None, plugins=..., session_timeout=(3.05, 60), reserved_name_scheme=..., search_enabled='auto')
+.. autofunction:: make_reader
 
 .. autoclass:: Reader(...)
     :members:
@@ -40,6 +40,9 @@ Data objects
 .. autoclass:: Enclosure
     :members:
 
+.. autoclass:: EntrySource
+    :members:
+
 .. autoclass:: EntrySearchResult
     :members:
 
@@ -184,4 +187,13 @@ Type aliases
 Constants
 ---------
 
+.. autodata:: reader.core.DEFAULT_RESERVED_NAME_SCHEME
+
 .. autodata:: reader.plugins.DEFAULT_PLUGINS
+
+
+
+Utilities
+---------
+
+.. autofunction:: reader.utils.archive_entries

{reader-3.14 → reader-3.16}/docs/dev.rst

@@ -3,7 +3,7 @@ Development
 ===========
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 Development should follow a problem-solution_ approach.
@@ -34,6 +34,9 @@ but I will prioritize supporting :doc:`contributors <contributing>`
 
 * OPML support, :issue:`165`
 * :ref:`deleting entries <deleting entries>`
+
+  * archiving important entries of deleted feeds, :issue:`290`
+
 * :ref:`feed interaction statistics <counts api>`
 
 * security
@@ -69,7 +72,6 @@ but I will prioritize supporting :doc:`contributors <contributing>`
 
 * add more fields to data objects
 
-  * entry source, :issue:`276`
   * extra data, as an escape hatch, :issue:`277`
 
 * :ref:`multiple storage implementations <multiple storage implementations>`
@@ -378,8 +380,14 @@ Requests session plugins:
 Retriever / parser split:
 
 * :issue:`205#issuecomment-766321855`
-* split exception hierarchy (not implemented as of 3.9): :issue:`218#issuecomment-1687094315`
+* split exception hierarchy (not implemented as of 3.15): :issue:`218#issuecomment-1687094315`
+* API overview as of 3.14, meant to show the dataflow: :issue:`307#issuecomment-2266647310`
+* ~ideal API (mostly implemented in 3.15): :issue:`307#issuecomment-2281797898`
 
+  * exposes HTTP information (so it can be used by the updater)
+  * introduces ~internal RetrieveError and NotModified :exc:`ParseError` subclasses;
+    notably, this doesn't really follow the split exception hierarchy mentioned above,
+    and is only meant to surface HTTP information in error cases
 
 Alternative feed parsers:
 
@@ -398,17 +406,13 @@ Lessons learned from the :ref:`Twitter` plugin:
   :meth:`~reader._parser.ParserType` being generic.
 
 * It is useful for a Retriever to store arbitrary caching data;
-  the plugin (mis)used
-  :attr:`~reader._parser.RetrieveResult.http_etag`
+  the plugin (mis)used ``RetrieveResult.http_etag``
   to store the (integer) id of the newest tweet in the thread.
 
-  It would be nice to formalize this into a single
-  "arbitrary caching data" attribute;
-  also see `this comment <https://github.com/lemon24/reader/blob/0ada24bf4e65c69c3028641bada8eaeabbe02754/src/reader/_parser.py#L569>`_.
+  Update: This was formalized as :attr:`.RetrievedFeed.caching_info` in 3.15.
 
 * It is useful for a Retriever to pass arbitrary data to itself;
-  the plugin (mis)used
-  :attr:`~reader._types.FeedForUpdate.http_etag` to pass from
+  the plugin (mis)used ``FeedForUpdate.http_etag`` to pass from
   :meth:`~reader._parser.FeedForUpdateRetrieverType.process_feed_for_update`
   to :meth:`~reader._parser.RetrieverType.__call__`:
 
@@ -510,6 +514,16 @@ From the initial issue:
 * use cases: :issue:`149#issuecomment-700066794`
 * initial requirements: :issue:`149#issuecomment-700532183`
 
+Splitting a feed into two feeds is discussed in :issue:`221`.
+The main takeaway at the time (2021) was that
+not all entry data can be retrieved through the storage API,
+so it's not fully possible to round-trip an entry.
+Based on a quick 2024 survey,
+the only affected attributes
+are :attr:`~.EntryUpdateIntent.feed_order` (can't get; not critical)
+and :attr:`~Entry.original_feed_url` (can't set; will be required for
+archiving important entries of deleted feeds, :issue:`290`).
+
 
 Resource tags / metadata
 ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -684,6 +698,14 @@ Update hook error handling:
 * expected behavior: :issue:`218#issuecomment-1595691410`, :issue:`218#issuecomment-1666823222`
 * update hook pseudocode + exception hierarchy: :issue:`218#issuecomment-1666869215`
 
+Considerations on using pluggy: :issue:`329`,
+`how adapting reader use cases to the pluggy model may work <https://github.com/pytest-dev/pluggy/issues/151#issuecomment-2421648901>`_.
+Conclusions:
+
+    even if using pluggy would require some work-arounds for calling, I think the collection / discovery functionality is still more than worth it
+
+    long-term and/or if reader becomes really popular, pluggy is definitely the way to go
+
 
 Reserved names
 ~~~~~~~~~~~~~~
@@ -698,16 +720,16 @@ Wrapping underlying storage exceptions
 
 Which exception to wrap, and which not: :issue:`21#issuecomment-365442439`.
 
-In version 3.10 (November 2023), all internal APIs were changed
-to use timezone-aware datetimes, with the timezone set to UTC,
-in preparation for support for any timezone.
-
 
 Timezone handling
 ~~~~~~~~~~~~~~~~~
 
 Aware vs. naive, and what's needed to go fully aware: :issue:`233#issuecomment-881618002`.
 
+In version 3.10 (November 2023), all internal APIs were changed
+to use timezone-aware datetimes, with the timezone set to UTC,
+in preparation for support for any timezone.
+
 
 OPML support
 ~~~~~~~~~~~~

{reader-3.14 → reader-3.16}/docs/guide.rst

@@ -3,7 +3,7 @@ User guide
 ==========
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 This page gives a tour of *reader*'s features,
@@ -272,34 +272,42 @@ Scheduled updates
 Because different feeds need to be updated at different rates,
 *reader* also provides a mechanism for scheduling updates.
 
-Each feed has an update interval that, on every update,
-determines when the feed should be updated next.
-Running :meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
+On every update, *reader* determines when a feed should be updated next
+based on a configurable update interval;
+:meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
 updates only the feeds that should be updated at or before the current time.
 
-The global and per-feed update interval can be **configured by the user**
-via the ``.reader.update`` global/feed tag;
-the default interval is of one hour;
-see :data:`~reader.types.UpdateConfig` for the schema.
-In addition to the interval, the user can specify a jitter;
-for an interval of 24 hours, a jitter of 0.25 means
-the update will occur any time in the first 6 hours of the interval.
+The interval can be **configured by the user**
+through the ``.reader.update`` global tag,
+and overridden for individual feeds through the matching feed tag;
+if no configuration is provided, the default interval is one hour.
+See :data:`~reader.types.UpdateConfig` for the config format.
 
-.. note::
+In addition, the user can specify a jitter,
+a random amount of time added to the interval
+such that updates don't happen exactly at the the same time;
+for example, with a one-hour interval, a jitter of 0.25 means
+updates will occur any time in the first 15 minutes of each hour.
+
+If the server sends the Retry-After HTTP header with
+429 Too Many Requests or 503 Service Unavailable responses,
+:meth:`update_feeds(scheduled=True) <Reader.update_feeds>` will honor it.
 
-    As of |version|, there is no way to specify a minimum update interval.
-    If you want feeds to be updated no more often than e.g. every hour,
-    you have to run :meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
-    no more often than every hour.
-    Please :ref:`open an issue <issues>` if you need a minimum update interval.
 
-In a future version of *reader*,
-the same mechanism will be used to handle
-HTTP 429 Too Many Requests; see :issue:`307` for details.
+.. note::
+
+    As of |version|, there is no way to enforce a minimum update interval;
+    if you want feeds to be updated no more often than e.g. every hour,
+    you have to run :meth:`~Reader.update_feeds` no more often than every hour.
+    Please :ref:`open an issue <issues>` or :ref:`submit a pull request <prs>`
+    if you need a minimum update interval.
 
 
 .. versionadded:: 3.13
 
+.. versionchanged:: 3.15
+    Honor the Retry-After HTTP header.
+
 
 Update status
 ~~~~~~~~~~~~~

{reader-3.14 → reader-3.16}/docs/install.rst

@@ -27,6 +27,20 @@ These packages will be installed automatically when installing *reader*:
 (at least SQLite 3.18 with the `JSON1`_ and `FTS5`_ extensions).
 
 
+.. _no-vendored-feedparser:
+
+.. note::
+
+  Because `feedparser`_ makes PyPI releases at a lower cadence,
+  *reader* uses a vendored version of feedparser's `develop`_ branch
+  by default since :ref:`version 2.9`.
+  To opt out of this behavior, and make *reader* use
+  the installed ``feedparser`` package,
+  set the ``READER_NO_VENDORED_FEEDPARSER`` environment variable to ``1``.
+
+.. _develop: https://github.com/kurtmckee/feedparser
+
+
 .. _optional dependencies:
 
 Optional dependencies

{reader-3.14 → reader-3.16}/docs/internal.rst

@@ -45,7 +45,7 @@ Protocols
 ~~~~~~~~~
 
 .. module:: reader._parser
-    :noindex:
+    :no-index:
 
 .. autoclass:: FeedArgument
     :members:
@@ -54,6 +54,14 @@ Protocols
     :members:
     :special-members: __call__
 
+.. autoexception:: RetrieveError
+    :show-inheritance:
+    :members:
+
+.. autoexception:: NotModified
+    :show-inheritance:
+    :members:
+
 .. autoclass:: FeedForUpdateRetrieverType
     :members:
     :show-inheritance:
@@ -62,7 +70,7 @@ Protocols
     :members:
     :special-members: __call__
 
-.. autoclass:: HTTPAcceptParserType
+.. autoclass:: AcceptParserType
     :members:
     :show-inheritance:
 
@@ -72,7 +80,7 @@ Protocols
 
 
 .. module:: reader._parser.requests
-    :noindex:
+    :no-index:
 
 .. autoclass:: RequestHook
     :members:
@@ -87,30 +95,21 @@ Data objects
 ~~~~~~~~~~~~
 
 .. module:: reader._parser
-    :noindex:
+    :no-index:
 
 .. autoclass:: RetrieveResult
     :members:
 
-.. module:: reader._types
-
-.. autoclass:: ParsedFeed
+.. autoclass:: RetrievedFeed
     :members:
 
-.. autoclass:: FeedData
-    :members:
-    :undoc-members:
-
-.. autoclass:: EntryData
+.. autoclass:: ParseResult
     :members:
-    :undoc-members:
 
-.. todo:: the following should be in the storage section, when we get one
-
-.. autoclass:: FeedForUpdate
+.. autoclass:: ParsedFeed
     :members:
 
-.. autoclass:: EntryForUpdate
+.. autoclass:: HTTPInfo
     :members:
 
 
@@ -155,7 +154,7 @@ but at least one other implementation needs to exists before that.
 .. autoattribute:: reader.Reader._search
 
 .. module:: reader._types
-    :noindex:
+    :no-index:
 
 .. autoclass:: StorageType()
     :members:
@@ -171,6 +170,9 @@ but at least one other implementation needs to exists before that.
 
 .. _changes:
 
+Change tracking
+~~~~~~~~~~~~~~~
+
 .. autoclass:: ChangeTrackingStorageType()
     :members:
     :show-inheritance:
@@ -193,12 +195,26 @@ but at least one other implementation needs to exists before that.
 Data objects
 ~~~~~~~~~~~~
 
+.. autoclass:: FeedData
+    :members:
+    :undoc-members:
+
+.. autoclass:: EntryData
+    :members:
+    :undoc-members:
+
 .. autoclass:: FeedFilter
     :members:
 
 .. autoclass:: EntryFilter
     :members:
 
+.. autoclass:: FeedForUpdate
+    :members:
+
+.. autoclass:: EntryForUpdate
+    :members:
+
 .. autoclass:: FeedUpdateIntent
     :members:
 

{reader-3.14 → reader-3.16}/docs/plugins.rst

@@ -3,7 +3,7 @@ Plugins
 =======
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 .. _built-in plugins:
@@ -116,3 +116,20 @@ For `built-in plugins`_, it is enough to use the plugin name (``reader.XYZ``).
 To load web application plugins, set the ``READER_APP_PLUGIN`` environment variable.
 To load CLI plugins (that customize the CLI),
 set the ``READER_CLI_PLUGIN`` environment variable.
+
+
+
+Recipes
+-------
+
+I currently don't need this functionality,
+but if you'd be interested in maintaining any of these
+as an experimental or even built-in plugin,
+please :doc:`submit a pull request <contributing>`.
+
+.. include:: ../examples/feed_slugs.py
+    :start-after: """
+    :end-before: """
+.. literalinclude:: ../examples/feed_slugs.py
+    :start-at: def init_reader
+    :end-before: if __name__

{reader-3.14 → reader-3.16}/docs/tutorial.rst

@@ -2,7 +2,7 @@ Tutorial
 ========
 
 .. module:: reader
-    :noindex:
+    :no-index:
 
 
 In this tutorial we'll use *reader* to download all the episodes of a podcast,

reader-3.16/examples/feed_slugs.py

@@ -0,0 +1,95 @@
+"""
+Feed slugs
+~~~~~~~~~~
+
+This is a recipe of what a "get feed by slug" plugin may look like
+(e.g. for user-defined short URLs).
+
+Usage::
+
+    >>> from reader import make_reader
+    >>> import feed_slugs
+    >>> reader = make_reader('db.sqlite', plugins=[feed_slugs.init_reader])
+    >>> reader.set_feed_slug('https://death.andgravity.com/_feed/index.xml', 'andgravity')
+    >>> reader.get_feed_by_slug('andgravity')
+    Feed(url='https://death.andgravity.com/_feed/index.xml', ...)
+    >>> reader.get_feed_slug(_.url)
+    'andgravity'
+
+..
+    Originally implemented for https://github.com/lemon24/reader/issues/358.
+
+"""
+
+# fmt: off
+# flake8: noqa
+
+def init_reader(reader):
+    # __get__() allows help(reader.get_feed_by_slug) to work
+    reader.get_feed_by_slug = get_feed_by_slug.__get__(reader)
+    reader.get_feed_slug = get_feed_slug.__get__(reader)
+    reader.set_feed_slug = set_feed_slug.__get__(reader)
+
+def get_feed_by_slug(reader, slug):
+    tag = _make_tag(reader, slug)
+    return next(reader.get_feeds(tags=[tag], limit=1), None)
+
+def get_feed_slug(reader, feed):
+    if tag := next(_get_tags(reader, feed), None):
+        return tag.removeprefix(_make_tag(reader, ''))
+    return None
+
+def set_feed_slug(reader, feed, slug: str | None):
+    feed = reader.get_feed(feed)
+    tag = _make_tag(reader, slug)
+
+    if not slug:
+        reader.delete_tag(feed, tag, missing_ok=True)
+        return
+
+    reader.set_tag(feed, tag)
+
+    # ensure only one feed has the slug; technically a race condition,
+    # when it happens no feed will have the tag
+    for other_feed in reader.get_feeds(tags=[tag]):
+        if feed.url != other_feed.url:
+            reader.delete_tag(other_feed, tag, missing_ok=True)
+
+    # ensure feed has only one slug; technically a race condition,
+    # when it happens the feed will have no slug
+    for other_tag in _get_tags(reader, feed):
+        if tag != other_tag:
+            reader.delete_tag(feed, other_tag, missing_ok=True)
+
+def _make_tag(reader, slug):
+    return reader.make_plugin_reserved_name('slug', slug)
+
+def _get_tags(reader, resource):
+    prefix = _make_tag(reader, '')
+    # filter tags by prefix would make this faster,
+    # https://github.com/lemon24/reader/issues/309
+    return (t for t in reader.get_tag_keys(resource) if t.startswith(prefix))
+
+if __name__ == '__main__':
+    from reader import make_reader
+
+    reader = make_reader('db.sqlite', plugins=[init_reader])
+    url = 'https://death.andgravity.com/_feed/index.xml'
+
+    reader.set_feed_slug(url, 'one')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('one'), 'url', None),
+    )
+
+    reader.set_feed_slug(url, 'two')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('two'), 'url', None),
+    )
+
+    reader.set_feed_slug('https://xkcd.com/atom.xml', 'two')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('two'), 'url', None),
+    )

{reader-3.14 → reader-3.16}/scripts/backup.sh

@@ -1,13 +1,13 @@
 #!/bin/bash
 #
-# back up a SQLite database not owned by the current user
+# back up a SQLite database
 #
 # usage:
-#   backup.sh src dst
-#   backup.sh src
+#   ./backup.sh src dst
+#   ./backup.sh src
 #
 # example:
-#   "backup.sh /src/db.sqlite" -> ./db.sqlite.2023-01-28
+#   "./backup.sh /src/db.sqlite" -> ./db.sqlite.2023-01-28.gz
 #
 
 set -o nounset
@@ -29,5 +29,8 @@ trap 'rm -rf '"$tmpdir" EXIT
 
 tmp=$tmpdir/$( basename "$src" )
 
-time sudo sqlite3 "$src" "VACUUM INTO '$tmp'"
-time cp "$tmp" "$dst"
+du -sh "$src"
+sqlite3 "$src" "VACUUM INTO '$tmp'"
+du -sh "$tmp"
+gzip -c "$tmp" > "$dst.gz"
+du -sh "$dst.gz"

{reader-3.14 → reader-3.16}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.14'
+__version__ = '3.16'
 
 # isort: off
 
@@ -52,6 +52,7 @@ from .types import (
     Entry as Entry,
     Content as Content,
     Enclosure as Enclosure,
+    EntrySource as EntrySource,
     EntrySearchResult as EntrySearchResult,
     HighlightedString as HighlightedString,
     FeedCounts as FeedCounts,