reader 3.14__tar.gz → 3.15__tar.gz

{reader-3.14 → reader-3.15}/CHANGES.rst

@@ -3,7 +3,42 @@ Changelog
 =========
 
 .. module:: reader
-  :noindex:
+  :no-index:
+
+
+Version 3.15
+------------
+
+Released 2024-11-11
+
+* During :ref:`scheduled updates <scheduled>`,
+  honor the Retry-After HTTP header sent with
+  429 Too Many Requests or 503 Service Unavailable responses.
+  (:issue:`307`)
+* Optimize OR-only :meth:`~Reader.get_feeds()` and :meth:`~Reader.get_entries()`
+  tag filters (e.g. ``[['one', ...]]``). (:issue:`359`)
+* Expose :data:`reader.core.DEFAULT_RESERVED_NAME_SCHEME`. (:issue:`287`)
+
+* Update the (unstable) parser API to expose HTTP information to the updater.
+  (:issue:`307`)
+
+  * The :class:`.RetrieverType` protocol used by retrievers changed
+    (new return type, allow storing arbitrary caching data via
+    :attr:`~.RetrievedFeed.caching_info`).
+  * The ``HTTPAcceptParserType`` was renamed to :class:`.AcceptParserType`.
+
+* Allow :ref:`using the installed feedparser <no-vendored-feedparser>`
+  instead of the one vendored by *reader*.
+  This is useful for working around issues in the vendored feedparser,
+  like the libxml2 <=2.13.3 incompatibility reported in :issue:`350`;
+  thanks to `Maks Verver`_  for reporting,
+  root causing, and following up with both dependencies.
+
+* Fix a number of brittle / broken tests.
+  Thanks to `Maks Verver`_ for the issues and fixes.
+  (:issue:`348`, :issue:`349`, :issue:`355`)
+
+.. _Maks Verver: https://github.com/maksverver
 
 
 Version 3.14
@@ -650,6 +685,8 @@ Released 2022-03-12
   Fix broken error flashing.
 
 
+.. _version 2.9:
+
 Version 2.9
 -----------
 

{reader-3.14/src/reader.egg-info → reader-3.15}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: reader
-Version: 3.14
+Version: 3.15
 Summary: A Python feed reader library.
 Author: lemon24
 License: BSD-3-Clause

{reader-3.14 → reader-3.15}/docs/api.rst

@@ -16,7 +16,7 @@ Most of *reader*'s functionality can be accessed through a :class:`Reader` insta
 
     Split this into sub-sections.
 
-.. autofunction:: make_reader(url, *, feed_root=None, plugins=..., session_timeout=(3.05, 60), reserved_name_scheme=..., search_enabled='auto')
+.. autofunction:: make_reader
 
 .. autoclass:: Reader(...)
     :members:
@@ -184,4 +184,6 @@ Type aliases
 Constants
 ---------
 
+.. autodata:: reader.core.DEFAULT_RESERVED_NAME_SCHEME
+
 .. autodata:: reader.plugins.DEFAULT_PLUGINS

{reader-3.14 → reader-3.15}/docs/dev.rst

@@ -3,7 +3,7 @@ Development
 ===========
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 Development should follow a problem-solution_ approach.
@@ -378,8 +378,14 @@ Requests session plugins:
 Retriever / parser split:
 
 * :issue:`205#issuecomment-766321855`
-* split exception hierarchy (not implemented as of 3.9): :issue:`218#issuecomment-1687094315`
+* split exception hierarchy (not implemented as of 3.15): :issue:`218#issuecomment-1687094315`
+* API overview as of 3.14, meant to show the dataflow: :issue:`307#issuecomment-2266647310`
+* ~ideal API (mostly implemented in 3.15): :issue:`307#issuecomment-2281797898`
 
+  * exposes HTTP information (so it can be used by the updater)
+  * introduces ~internal RetrieveError and NotModified :exc:`ParseError` subclasses;
+    notably, this doesn't really follow the split exception hierarchy mentioned above,
+    and is only meant to surface HTTP information in error cases
 
 Alternative feed parsers:
 
@@ -398,17 +404,13 @@ Lessons learned from the :ref:`Twitter` plugin:
   :meth:`~reader._parser.ParserType` being generic.
 
 * It is useful for a Retriever to store arbitrary caching data;
-  the plugin (mis)used
-  :attr:`~reader._parser.RetrieveResult.http_etag`
+  the plugin (mis)used ``RetrieveResult.http_etag``
   to store the (integer) id of the newest tweet in the thread.
 
-  It would be nice to formalize this into a single
-  "arbitrary caching data" attribute;
-  also see `this comment <https://github.com/lemon24/reader/blob/0ada24bf4e65c69c3028641bada8eaeabbe02754/src/reader/_parser.py#L569>`_.
+  Update: This was formalized as :attr:`.RetrievedFeed.caching_info` in 3.15.
 
 * It is useful for a Retriever to pass arbitrary data to itself;
-  the plugin (mis)used
-  :attr:`~reader._types.FeedForUpdate.http_etag` to pass from
+  the plugin (mis)used ``FeedForUpdate.http_etag`` to pass from
   :meth:`~reader._parser.FeedForUpdateRetrieverType.process_feed_for_update`
   to :meth:`~reader._parser.RetrieverType.__call__`:
 
@@ -684,6 +686,14 @@ Update hook error handling:
 * expected behavior: :issue:`218#issuecomment-1595691410`, :issue:`218#issuecomment-1666823222`
 * update hook pseudocode + exception hierarchy: :issue:`218#issuecomment-1666869215`
 
+Considerations on using pluggy: :issue:`329`,
+`how adapting reader use cases to the pluggy model may work <https://github.com/pytest-dev/pluggy/issues/151#issuecomment-2421648901>`_.
+Conclusions:
+
+    even if using pluggy would require some work-arounds for calling, I think the collection / discovery functionality is still more than worth it
+
+    long-term and/or if reader becomes really popular, pluggy is definitely the way to go
+
 
 Reserved names
 ~~~~~~~~~~~~~~

{reader-3.14 → reader-3.15}/docs/guide.rst

@@ -3,7 +3,7 @@ User guide
 ==========
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 This page gives a tour of *reader*'s features,
@@ -272,34 +272,42 @@ Scheduled updates
 Because different feeds need to be updated at different rates,
 *reader* also provides a mechanism for scheduling updates.
 
-Each feed has an update interval that, on every update,
-determines when the feed should be updated next.
-Running :meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
+On every update, *reader* determines when a feed should be updated next
+based on a configurable update interval;
+:meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
 updates only the feeds that should be updated at or before the current time.
 
-The global and per-feed update interval can be **configured by the user**
-via the ``.reader.update`` global/feed tag;
-the default interval is of one hour;
-see :data:`~reader.types.UpdateConfig` for the schema.
-In addition to the interval, the user can specify a jitter;
-for an interval of 24 hours, a jitter of 0.25 means
-the update will occur any time in the first 6 hours of the interval.
+The interval can be **configured by the user**
+through the ``.reader.update`` global tag,
+and overridden for individual feeds through the matching feed tag;
+if no configuration is provided, the default interval is one hour.
+See :data:`~reader.types.UpdateConfig` for the config format.
 
-.. note::
+In addition, the user can specify a jitter,
+a random amount of time added to the interval
+such that updates don't happen exactly at the the same time;
+for example, with a one-hour interval, a jitter of 0.25 means
+updates will occur any time in the first 15 minutes of each hour.
+
+If the server sends the Retry-After HTTP header with
+429 Too Many Requests or 503 Service Unavailable responses,
+:meth:`update_feeds(scheduled=True) <Reader.update_feeds>` will honor it.
 
-    As of |version|, there is no way to specify a minimum update interval.
-    If you want feeds to be updated no more often than e.g. every hour,
-    you have to run :meth:`update_feeds(scheduled=True) <Reader.update_feeds>`
-    no more often than every hour.
-    Please :ref:`open an issue <issues>` if you need a minimum update interval.
 
-In a future version of *reader*,
-the same mechanism will be used to handle
-HTTP 429 Too Many Requests; see :issue:`307` for details.
+.. note::
+
+    As of |version|, there is no way to enforce a minimum update interval;
+    if you want feeds to be updated no more often than e.g. every hour,
+    you have to run :meth:`~Reader.update_feeds` no more often than every hour.
+    Please :ref:`open an issue <issues>` or :ref:`submit a pull request <prs>`
+    if you need a minimum update interval.
 
 
 .. versionadded:: 3.13
 
+.. versionchanged:: 3.15
+    Honor the Retry-After HTTP header.
+
 
 Update status
 ~~~~~~~~~~~~~

{reader-3.14 → reader-3.15}/docs/install.rst

@@ -27,6 +27,20 @@ These packages will be installed automatically when installing *reader*:
 (at least SQLite 3.18 with the `JSON1`_ and `FTS5`_ extensions).
 
 
+.. _no-vendored-feedparser:
+
+.. note::
+
+  Because `feedparser`_ makes PyPI releases at a lower cadence,
+  *reader* uses a vendored version of feedparser's `develop`_ branch
+  by default since :ref:`version 2.9`.
+  To opt out of this behavior, and make *reader* use
+  the installed ``feedparser`` package,
+  set the ``READER_NO_VENDORED_FEEDPARSER`` environment variable to ``1``.
+
+.. _develop: https://github.com/kurtmckee/feedparser
+
+
 .. _optional dependencies:
 
 Optional dependencies

{reader-3.14 → reader-3.15}/docs/internal.rst

@@ -45,7 +45,7 @@ Protocols
 ~~~~~~~~~
 
 .. module:: reader._parser
-    :noindex:
+    :no-index:
 
 .. autoclass:: FeedArgument
     :members:
@@ -54,6 +54,14 @@ Protocols
     :members:
     :special-members: __call__
 
+.. autoexception:: RetrieveError
+    :show-inheritance:
+    :members:
+
+.. autoexception:: NotModified
+    :show-inheritance:
+    :members:
+
 .. autoclass:: FeedForUpdateRetrieverType
     :members:
     :show-inheritance:
@@ -62,7 +70,7 @@ Protocols
     :members:
     :special-members: __call__
 
-.. autoclass:: HTTPAcceptParserType
+.. autoclass:: AcceptParserType
     :members:
     :show-inheritance:
 
@@ -72,7 +80,7 @@ Protocols
 
 
 .. module:: reader._parser.requests
-    :noindex:
+    :no-index:
 
 .. autoclass:: RequestHook
     :members:
@@ -87,30 +95,21 @@ Data objects
 ~~~~~~~~~~~~
 
 .. module:: reader._parser
-    :noindex:
+    :no-index:
 
 .. autoclass:: RetrieveResult
     :members:
 
-.. module:: reader._types
-
-.. autoclass:: ParsedFeed
+.. autoclass:: RetrievedFeed
     :members:
 
-.. autoclass:: FeedData
-    :members:
-    :undoc-members:
-
-.. autoclass:: EntryData
+.. autoclass:: ParseResult
     :members:
-    :undoc-members:
 
-.. todo:: the following should be in the storage section, when we get one
-
-.. autoclass:: FeedForUpdate
+.. autoclass:: ParsedFeed
     :members:
 
-.. autoclass:: EntryForUpdate
+.. autoclass:: HTTPInfo
     :members:
 
 
@@ -155,7 +154,7 @@ but at least one other implementation needs to exists before that.
 .. autoattribute:: reader.Reader._search
 
 .. module:: reader._types
-    :noindex:
+    :no-index:
 
 .. autoclass:: StorageType()
     :members:
@@ -171,6 +170,9 @@ but at least one other implementation needs to exists before that.
 
 .. _changes:
 
+Change tracking
+~~~~~~~~~~~~~~~
+
 .. autoclass:: ChangeTrackingStorageType()
     :members:
     :show-inheritance:
@@ -193,12 +195,26 @@ but at least one other implementation needs to exists before that.
 Data objects
 ~~~~~~~~~~~~
 
+.. autoclass:: FeedData
+    :members:
+    :undoc-members:
+
+.. autoclass:: EntryData
+    :members:
+    :undoc-members:
+
 .. autoclass:: FeedFilter
     :members:
 
 .. autoclass:: EntryFilter
     :members:
 
+.. autoclass:: FeedForUpdate
+    :members:
+
+.. autoclass:: EntryForUpdate
+    :members:
+
 .. autoclass:: FeedUpdateIntent
     :members:
 

{reader-3.14 → reader-3.15}/docs/plugins.rst

@@ -3,7 +3,7 @@ Plugins
 =======
 
 .. module:: reader
-  :noindex:
+  :no-index:
 
 
 .. _built-in plugins:
@@ -116,3 +116,20 @@ For `built-in plugins`_, it is enough to use the plugin name (``reader.XYZ``).
 To load web application plugins, set the ``READER_APP_PLUGIN`` environment variable.
 To load CLI plugins (that customize the CLI),
 set the ``READER_CLI_PLUGIN`` environment variable.
+
+
+
+Recipes
+-------
+
+I currently don't need this functionality,
+but if you'd be interested in maintaining any of these
+as an experimental or even built-in plugin,
+please :doc:`submit a pull request <contributing>`.
+
+.. include:: ../examples/feed_slugs.py
+    :start-after: """
+    :end-before: """
+.. literalinclude:: ../examples/feed_slugs.py
+    :start-at: def init_reader
+    :end-before: if __name__

{reader-3.14 → reader-3.15}/docs/tutorial.rst

@@ -2,7 +2,7 @@ Tutorial
 ========
 
 .. module:: reader
-    :noindex:
+    :no-index:
 
 
 In this tutorial we'll use *reader* to download all the episodes of a podcast,

reader-3.15/examples/feed_slugs.py

@@ -0,0 +1,95 @@
+"""
+Feed slugs
+~~~~~~~~~~
+
+This is a recipe of what a "get feed by slug" plugin may look like
+(e.g. for user-defined short URLs).
+
+Usage::
+
+    >>> from reader import make_reader
+    >>> import feed_slugs
+    >>> reader = make_reader('db.sqlite', plugins=[feed_slugs.init_reader])
+    >>> reader.set_feed_slug('https://death.andgravity.com/_feed/index.xml', 'andgravity')
+    >>> reader.get_feed_by_slug('andgravity')
+    Feed(url='https://death.andgravity.com/_feed/index.xml', ...)
+    >>> reader.get_feed_slug(_.url)
+    'andgravity'
+
+..
+    Originally implemented for https://github.com/lemon24/reader/issues/358.
+
+"""
+
+# fmt: off
+# flake8: noqa
+
+def init_reader(reader):
+    # __get__() allows help(reader.get_feed_by_slug) to work
+    reader.get_feed_by_slug = get_feed_by_slug.__get__(reader)
+    reader.get_feed_slug = get_feed_slug.__get__(reader)
+    reader.set_feed_slug = set_feed_slug.__get__(reader)
+
+def get_feed_by_slug(reader, slug):
+    tag = _make_tag(reader, slug)
+    return next(reader.get_feeds(tags=[tag], limit=1), None)
+
+def get_feed_slug(reader, feed):
+    if tag := next(_get_tags(reader, feed), None):
+        return tag.removeprefix(_make_tag(reader, ''))
+    return None
+
+def set_feed_slug(reader, feed, slug: str | None):
+    feed = reader.get_feed(feed)
+    tag = _make_tag(reader, slug)
+
+    if not slug:
+        reader.delete_tag(feed, tag, missing_ok=True)
+        return
+
+    reader.set_tag(feed, tag)
+
+    # ensure only one feed has the slug; technically a race condition,
+    # when it happens no feed will have the tag
+    for other_feed in reader.get_feeds(tags=[tag]):
+        if feed.url != other_feed.url:
+            reader.delete_tag(other_feed, tag, missing_ok=True)
+
+    # ensure feed has only one slug; technically a race condition,
+    # when it happens the feed will have no slug
+    for other_tag in _get_tags(reader, feed):
+        if tag != other_tag:
+            reader.delete_tag(feed, other_tag, missing_ok=True)
+
+def _make_tag(reader, slug):
+    return reader.make_plugin_reserved_name('slug', slug)
+
+def _get_tags(reader, resource):
+    prefix = _make_tag(reader, '')
+    # filter tags by prefix would make this faster,
+    # https://github.com/lemon24/reader/issues/309
+    return (t for t in reader.get_tag_keys(resource) if t.startswith(prefix))
+
+if __name__ == '__main__':
+    from reader import make_reader
+
+    reader = make_reader('db.sqlite', plugins=[init_reader])
+    url = 'https://death.andgravity.com/_feed/index.xml'
+
+    reader.set_feed_slug(url, 'one')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('one'), 'url', None),
+    )
+
+    reader.set_feed_slug(url, 'two')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('two'), 'url', None),
+    )
+
+    reader.set_feed_slug('https://xkcd.com/atom.xml', 'two')
+    print(
+        reader.get_feed_slug(url),
+        getattr(reader.get_feed_by_slug('two'), 'url', None),
+    )

{reader-3.14 → reader-3.15}/scripts/backup.sh

@@ -1,13 +1,13 @@
 #!/bin/bash
 #
-# back up a SQLite database not owned by the current user
+# back up a SQLite database
 #
 # usage:
-#   backup.sh src dst
-#   backup.sh src
+#   ./backup.sh src dst
+#   ./backup.sh src
 #
 # example:
-#   "backup.sh /src/db.sqlite" -> ./db.sqlite.2023-01-28
+#   "./backup.sh /src/db.sqlite" -> ./db.sqlite.2023-01-28.gz
 #
 
 set -o nounset
@@ -29,5 +29,8 @@ trap 'rm -rf '"$tmpdir" EXIT
 
 tmp=$tmpdir/$( basename "$src" )
 
-time sudo sqlite3 "$src" "VACUUM INTO '$tmp'"
-time cp "$tmp" "$dst"
+du -sh "$src"
+sqlite3 "$src" "VACUUM INTO '$tmp'"
+du -sh "$tmp"
+gzip -c "$tmp" > "$dst.gz"
+du -sh "$dst.gz"

{reader-3.14 → reader-3.15}/src/reader/__init__.py

@@ -37,7 +37,7 @@ Print the titles of the unread entries::
 
 """
 
-__version__ = '3.14'
+__version__ = '3.15'
 
 # isort: off
 

{reader-3.14 → reader-3.15}/src/reader/_cli.py

@@ -122,13 +122,12 @@ def log_command(fn):
 
 def config_option(*args, **kwargs):
     def callback(ctx, param, value):
-        # TODO: the default file is allowed to not exist, a user specified file must exist
+        config_path = value if value is not None else get_default_config_path()
         try:
-            with open(value) as file:
+            with open(config_path) as file:
                 config = make_reader_config(yaml.safe_load(file))
-        except FileNotFoundError as e:
-            if value != param.default:
-                raise click.BadParameter(str(e), ctx=ctx, param=param) from None
+        except FileNotFoundError:
+            assert value is None
             config = make_reader_config({})
 
         ctx.default_map = config['cli'].get('defaults', {})
@@ -136,17 +135,14 @@ def config_option(*args, **kwargs):
         ctx.obj = config
         return config
 
-    def inner(fn):
-        return click.option(
-            *args,
-            type=click.Path(dir_okay=False),
-            callback=callback,
-            is_eager=True,
-            expose_value=False,
-            **kwargs,
-        )(fn)
-
-    return inner
+    return click.option(
+        *args,
+        type=click.Path(exists=True, dir_okay=False),
+        callback=callback,
+        is_eager=True,
+        expose_value=False,
+        **kwargs,
+    )
 
 
 def pass_reader(fn):
@@ -166,7 +162,6 @@ def pass_reader(fn):
     '--db',
     type=click.Path(dir_okay=False),
     envvar=reader._DB_ENVVAR,
-    show_default=True,
     help=f"Path to the reader database. [default: {get_default_db_path()}]",
 )
 @click.option(
@@ -184,9 +179,7 @@ def pass_reader(fn):
 @config_option(
     '--config',
     envvar=reader._CONFIG_ENVVAR,
-    help="Path to the reader config.",
-    default=get_default_config_path(),
-    show_default=True,
+    help=f"Path to the reader config. [default: {get_default_config_path()}]",
 )
 @click.option(
     '--feed-root',