robotframework-parallelrunner 0.4.0__tar.gz → 0.5.0__tar.gz

{robotframework_parallelrunner-0.4.0 → robotframework_parallelrunner-0.5.0}/PKG-INFO

@@ -1,20 +1,19 @@
 Metadata-Version: 2.4
 Name: robotframework-parallelrunner
-Version: 0.4.0
+Version: 0.5.0
 Summary: Run a keyword in parallel inside a single Robot Framework test case, with one clean, ordered log.html.
 License-Expression: MIT
 License-File: LICENSE
 Keywords: robotframework,robot-framework,parallel,threading,testing,automation
 Author: Cristian Garcia
 Author-email: cristian.garcia.vd@gmail.com
-Requires-Python: >=3.9
+Requires-Python: >=3.10
 Classifier: Development Status :: 4 - Beta
 Classifier: Framework :: Robot Framework
 Classifier: Framework :: Robot Framework :: Library
 Classifier: Intended Audience :: Developers
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -53,9 +52,10 @@ correctly ordered `log.html` plus a structured list of per-item results.
 pip install robotframework-parallelrunner
 ```
 
-Requires Python 3.9+ and Robot Framework 5.0+ (installed automatically).
-See [docs/INSTALLATION.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/docs/INSTALLATION.md) for development installs
-and upgrading from the pre-release import path.
+Requires Python 3.10+ and Robot Framework 5.0+ (installed automatically).
+See [docs/INSTALLATION.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/docs/INSTALLATION.md) for development installs,
+upgrading from the pre-release import path, and Python 3.8/3.9 (no longer
+supported, but 0.5.0 was tested and works on them).
 
 ## Quickstart
 
@@ -113,8 +113,8 @@ threading, buffering, and replay logic stay out of your `.robot` files.
 ## Summary of Benefits
 
 - **Speed.** Waiting time is overlapped instead of added up. In this repo's
-  examples, 9 HTTP checks with a 1 s delay each drop from 13 s to 2 s with
-  9 workers, and 4 suites × 8 items × 1 s drop from 32.7 s to 4.6 s with
+  examples, 9 HTTP checks with a 1 s delay each drop from 13.9 s to 2.2 s with
+  9 workers, and 4 suites × 8 items × 1 s drop from 32.9 s to 5.1 s with
   8 workers (see [Using It Together With pabot](#using-it-together-with-pabot)).
 - **Integrity.** Your `log.html` remains the single source of truth — no
   broken XML tags from concurrent writes.
@@ -183,7 +183,7 @@ pabot --processes 4 --pythonpath . atest/
 pabot --processes 4 --testlevelsplit --pythonpath . atest/
 ```
 
-Both commands pass the full acceptance suite of this repository (52 tests;
+Both commands pass the full acceptance suite of this repository (82 tests;
 checked with pabot 5.2.2).
 
 How much time each one saves, from the offline
@@ -192,14 +192,14 @@ How much time each one saves, from the offline
 
 | Run | Wall-clock time | Speedup |
 |---|---|---|
-| robot, sequential loop | 32.7 s | ×1.0 |
-| robot + ParallelRunner (8 workers) | 4.6 s | ×7.1 |
-| pabot (4 processes), sequential loop | 10.6 s | ×3.1 |
-| pabot (4 processes) + ParallelRunner (8 workers) | 3.5 s | ×9.2 |
+| robot, sequential loop | 32.9 s | ×1.0 |
+| robot + ParallelRunner (8 workers) | 5.1 s | ×6.5 |
+| pabot (4 processes), sequential loop | 12.4 s | ×2.7 |
+| pabot (4 processes) + ParallelRunner (8 workers) | 4.6 s | ×7.2 |
 
 ParallelRunner removes the waiting *inside* each test, and pabot removes it
 *between* suites. With suites this short, pabot's process start-up (about
-2.5 s here) is most of what's left, so the combination pays off more as
+3.5 s here) is most of what's left, so the combination pays off more as
 suites get longer.
 
 Things to keep in mind:
@@ -221,6 +221,31 @@ Things to keep in mind:
   could interfere with each other. Keep shared state per process, or split
   by suite.
 
+## How Is This Different From AsyncLibrary?
+
+[AsyncLibrary](https://github.com/rlehfeld/robotframework-async-keyword)
+(`pip install robotframework-async-keyword`) also runs keywords in threads
+inside one test case. It runs *any* keyword, user keywords included: you
+start each call with `Async Run` and collect it later with `Async Get`. It
+buffers each call's output and replays it, so `log.html` keeps the full
+nested keyword structure. To do that, it works with Robot Framework's
+internal execution context and output.
+
+| | **AsyncLibrary** | **ParallelRunner** |
+|---|---|---|
+| Runs | Any keyword, including user keywords | A method of a Python library that accepts `_logger` |
+| Usage | `Async Run` per call, `Async Get` later | One `Run Parallel Scenarios` call over a list, or `repeat=N` |
+| Concurrency | Python's default thread pool size | `ROBOT_THREAD_WORKERS` |
+| Hung calls | `Async Get    timeout=...` stops waiting for them | `call_timeout` fails only that call and frees its slot for the next item |
+| Failures | `Async Get` raises the call's error | One failure listing every failed item, or a `PASS`/`FAIL` per item with `fail_on_error=False` |
+| Robot internals | Works with the internal execution context and output | Not patched; logs only from the main thread, through the public logger API; tested on Robot Framework 5, 6 and 7 |
+
+Use AsyncLibrary when the work lives in user keywords (or in library
+keywords you can't change), or when you want to start something in the
+background, run other steps, and collect it later. Use ParallelRunner to fan
+one Python keyword out over many items as a single batch, with a bounded
+worker pool, per-call timeouts and one failure report for the whole batch.
+
 ## When Not To Use This
 
 - **CPU-bound work.** Threads share Python's GIL — number crunching won't
@@ -230,11 +255,6 @@ Things to keep in mind:
   made thread-safe for you; your own keyword's side effects are not. If it
   writes to a shared variable, file, or object without a lock, running it
   concurrently can race the same way any multi-threaded code can.
-- **You need per-item retries.** Not implemented yet (see
-  [ROADMAP.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/ROADMAP.md)). A per-call limit is: `call_timeout=10s` fails
-  only the call that hangs and lets the others finish. Note that a timed-out
-  call can't be killed - it keeps running in the background unless the
-  keyword declares `_cancelled` and stops when it is set.
 - **You need process-level isolation** (a crash in one call shouldn't be
   able to affect another) or cross-machine parallelism — that's pabot's
   domain, not this library's.

{robotframework_parallelrunner-0.4.0 → robotframework_parallelrunner-0.5.0}/README.md

@@ -19,9 +19,10 @@ correctly ordered `log.html` plus a structured list of per-item results.
 pip install robotframework-parallelrunner
 ```
 
-Requires Python 3.9+ and Robot Framework 5.0+ (installed automatically).
-See [docs/INSTALLATION.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/docs/INSTALLATION.md) for development installs
-and upgrading from the pre-release import path.
+Requires Python 3.10+ and Robot Framework 5.0+ (installed automatically).
+See [docs/INSTALLATION.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/docs/INSTALLATION.md) for development installs,
+upgrading from the pre-release import path, and Python 3.8/3.9 (no longer
+supported, but 0.5.0 was tested and works on them).
 
 ## Quickstart
 
@@ -79,8 +80,8 @@ threading, buffering, and replay logic stay out of your `.robot` files.
 ## Summary of Benefits
 
 - **Speed.** Waiting time is overlapped instead of added up. In this repo's
-  examples, 9 HTTP checks with a 1 s delay each drop from 13 s to 2 s with
-  9 workers, and 4 suites × 8 items × 1 s drop from 32.7 s to 4.6 s with
+  examples, 9 HTTP checks with a 1 s delay each drop from 13.9 s to 2.2 s with
+  9 workers, and 4 suites × 8 items × 1 s drop from 32.9 s to 5.1 s with
   8 workers (see [Using It Together With pabot](#using-it-together-with-pabot)).
 - **Integrity.** Your `log.html` remains the single source of truth — no
   broken XML tags from concurrent writes.
@@ -149,7 +150,7 @@ pabot --processes 4 --pythonpath . atest/
 pabot --processes 4 --testlevelsplit --pythonpath . atest/
 ```
 
-Both commands pass the full acceptance suite of this repository (52 tests;
+Both commands pass the full acceptance suite of this repository (82 tests;
 checked with pabot 5.2.2).
 
 How much time each one saves, from the offline
@@ -158,14 +159,14 @@ How much time each one saves, from the offline
 
 | Run | Wall-clock time | Speedup |
 |---|---|---|
-| robot, sequential loop | 32.7 s | ×1.0 |
-| robot + ParallelRunner (8 workers) | 4.6 s | ×7.1 |
-| pabot (4 processes), sequential loop | 10.6 s | ×3.1 |
-| pabot (4 processes) + ParallelRunner (8 workers) | 3.5 s | ×9.2 |
+| robot, sequential loop | 32.9 s | ×1.0 |
+| robot + ParallelRunner (8 workers) | 5.1 s | ×6.5 |
+| pabot (4 processes), sequential loop | 12.4 s | ×2.7 |
+| pabot (4 processes) + ParallelRunner (8 workers) | 4.6 s | ×7.2 |
 
 ParallelRunner removes the waiting *inside* each test, and pabot removes it
 *between* suites. With suites this short, pabot's process start-up (about
-2.5 s here) is most of what's left, so the combination pays off more as
+3.5 s here) is most of what's left, so the combination pays off more as
 suites get longer.
 
 Things to keep in mind:
@@ -187,6 +188,31 @@ Things to keep in mind:
   could interfere with each other. Keep shared state per process, or split
   by suite.
 
+## How Is This Different From AsyncLibrary?
+
+[AsyncLibrary](https://github.com/rlehfeld/robotframework-async-keyword)
+(`pip install robotframework-async-keyword`) also runs keywords in threads
+inside one test case. It runs *any* keyword, user keywords included: you
+start each call with `Async Run` and collect it later with `Async Get`. It
+buffers each call's output and replays it, so `log.html` keeps the full
+nested keyword structure. To do that, it works with Robot Framework's
+internal execution context and output.
+
+| | **AsyncLibrary** | **ParallelRunner** |
+|---|---|---|
+| Runs | Any keyword, including user keywords | A method of a Python library that accepts `_logger` |
+| Usage | `Async Run` per call, `Async Get` later | One `Run Parallel Scenarios` call over a list, or `repeat=N` |
+| Concurrency | Python's default thread pool size | `ROBOT_THREAD_WORKERS` |
+| Hung calls | `Async Get    timeout=...` stops waiting for them | `call_timeout` fails only that call and frees its slot for the next item |
+| Failures | `Async Get` raises the call's error | One failure listing every failed item, or a `PASS`/`FAIL` per item with `fail_on_error=False` |
+| Robot internals | Works with the internal execution context and output | Not patched; logs only from the main thread, through the public logger API; tested on Robot Framework 5, 6 and 7 |
+
+Use AsyncLibrary when the work lives in user keywords (or in library
+keywords you can't change), or when you want to start something in the
+background, run other steps, and collect it later. Use ParallelRunner to fan
+one Python keyword out over many items as a single batch, with a bounded
+worker pool, per-call timeouts and one failure report for the whole batch.
+
 ## When Not To Use This
 
 - **CPU-bound work.** Threads share Python's GIL — number crunching won't
@@ -196,11 +222,6 @@ Things to keep in mind:
   made thread-safe for you; your own keyword's side effects are not. If it
   writes to a shared variable, file, or object without a lock, running it
   concurrently can race the same way any multi-threaded code can.
-- **You need per-item retries.** Not implemented yet (see
-  [ROADMAP.md](https://github.com/cristiangarciavd/robotframework-parallelrunner/blob/main/ROADMAP.md)). A per-call limit is: `call_timeout=10s` fails
-  only the call that hangs and lets the others finish. Note that a timed-out
-  call can't be killed - it keeps running in the background unless the
-  keyword declares `_cancelled` and stops when it is set.
 - **You need process-level isolation** (a crash in one call shouldn't be
   able to affect another) or cross-machine parallelism — that's pabot's
   domain, not this library's.

{robotframework_parallelrunner-0.4.0 → robotframework_parallelrunner-0.5.0}/pyproject.toml

@@ -6,12 +6,12 @@ build-backend = "poetry.core.masonry.api"
 name = "robotframework-parallelrunner"
 # Single source of truth for the version: bump with `poetry version patch|minor|major`.
 # ParallelRunner.__version__ / ROBOT_LIBRARY_VERSION read it back from the installed metadata.
-version = "0.4.0"
+version = "0.5.0"
 description = "Run a keyword in parallel inside a single Robot Framework test case, with one clean, ordered log.html."
 readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
-requires-python = ">=3.9"
+requires-python = ">=3.10"
 authors = [
     { name = "Cristian Garcia", email = "cristian.garcia.vd@gmail.com" },
 ]
@@ -30,7 +30,6 @@ classifiers = [
     "Intended Audience :: Developers",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",

{robotframework_parallelrunner-0.4.0 → robotframework_parallelrunner-0.5.0}/src/ParallelRunner/parallel_library.py

@@ -15,6 +15,7 @@ from robot.libraries.BuiltIn import BuiltIn
 from robot.running.arguments import PythonArgumentParser
 from robot.utils import secs_to_timestr, timestr_to_secs
 
+from .retry import NO_RETRIES, RetryPolicy
 from .version import __version__
 
 
@@ -54,6 +55,18 @@ def _not_given(value: Any) -> bool:
 _WORKER, _MAIN = "worker", "main"
 _NOT_SET = object()
 
+# Worker threads left running a call that was abandoned - timed out by
+# `call_timeout` or interrupted by a Robot `[Timeout]` - across every keyword
+# call of the run, so the limit (ROBOT_MAX_ABANDONED_THREADS) holds even when
+# the hung calls come from earlier tests. Only the thread running the keywords
+# changes it. Dead threads are dropped by `_abandoned_threads_alive`.
+_abandoned_threads: List[threading.Thread] = []
+
+
+def _abandoned_threads_alive() -> int:
+    _abandoned_threads[:] = [thread for thread in _abandoned_threads if thread.is_alive()]
+    return len(_abandoned_threads)
+
 
 def log_to_robot(msg: Any, level: str = "INFO") -> None:
     """Logs ``msg`` straight to Robot Framework, accepting the same levels as
@@ -179,6 +192,7 @@ class ParallelRunner:
     | logs     | List of ``(level, message)`` tuples captured during the call. |
     | result   | The return value of the call (only when ``status`` is ``PASS``). |
     | error    | The error message (only when ``status`` is ``FAIL``), e.g. ``Timed out after 10 seconds`` with ``call_timeout``. |
+    | attempts | How many times the call ran: ``1``, or more with ``call_retries`` (see `Retries`). ``0`` for a call that never ran. |
 
     = Failures =
 
@@ -255,6 +269,18 @@ class ParallelRunner:
     background, and a new worker thread takes its place so the remaining
     calls still run ``ROBOT_THREAD_WORKERS`` at a time.
 
+    So that calls that hang can't pile up hundreds of threads (and open
+    connections), the abandoned threads still running are limited by
+    ``ROBOT_MAX_ABANDONED_THREADS`` (default: twice ``ROBOT_THREAD_WORKERS``).
+    They are counted across the whole run - calls abandoned by earlier tests,
+    or interrupted by a Robot ``[Timeout]``, count too - and a thread stops
+    counting as soon as its call returns. Above the limit, a warning is logged
+    and timed-out workers are no longer replaced: the remaining calls run with
+    fewer workers, and if none is left they fail without starting, with error
+    ``Not started: abandoned thread limit (8) reached``. Every keyword call
+    still starts with its normal number of workers, whatever earlier tests
+    left running.
+
     A keyword that can stop early declares a ``_cancelled`` parameter.
     `Run Parallel Scenarios` then passes it a ``threading.Event`` that is set
     when that call times out, or when a Robot ``[Timeout]`` interrupts the
@@ -271,6 +297,32 @@ class ParallelRunner:
     existing keywords keep working unchanged. It can't interrupt a call that
     is blocked in a single operation (a network request without a timeout).
 
+    = Retries =
+
+    ``call_retries=N`` runs a failed call up to ``N`` more times before it
+    counts as failed. Only that call is retried; the others are not affected.
+
+    | ${results}= | Run Parallel Scenarios | keyword=Verify Agent | library=MyLibrary | for_loop_iterable=${agents} | call_retries=2 | retry_delay=1s | retry_backoff=2 |
+
+    - Between attempts the call waits ``retry_delay`` (default ``1s``), multiplied by
+      ``retry_backoff`` after every failed retry: above, 1 second before the second
+      attempt and 2 seconds before the third. The wait happens in a worker
+      thread, after the calls that were already waiting.
+    - A call that exceeds ``call_timeout`` is retried too, and ``call_timeout``
+      applies to each attempt. The timed-out attempt keeps running in the
+      background, so it is only retried while the abandoned threads are within
+      ``ROBOT_MAX_ABANDONED_THREADS``; otherwise it fails with its timeout error.
+    - Every attempt's logs are kept, separated by a ``WARN`` such as
+      ``Attempt 1 of 3 failed: <error>; retrying in 1 second``, so ``log.html``
+      shows why a call needed retries even when it passed in the end. Only the
+      last failure is logged as ``ERROR``.
+    - The ``attempts`` key of the result tells how many attempts the call took,
+      to spot calls that are flaky even when the test passes.
+    - A Robot ``[Timeout]`` stops the retries along with the rest of the batch.
+
+    Retry only what is safe to repeat: a call that failed half way may already
+    have changed something (a created record, a sent message).
+
     = Custom logger mappers =
 
     A logger mapper replaces the default buffering logger: the target keyword
@@ -295,6 +347,7 @@ class ParallelRunner:
 
     | =Variable=             | =Description= |
     | ROBOT_THREAD_WORKERS   | Number of worker threads. Read when the library is imported. Default ``4``. An invalid value is ignored with a warning. |
+    | ROBOT_MAX_ABANDONED_THREADS | Maximum number of threads left running by timed-out calls before they are no longer replaced. Read when the library is imported. Default: twice ``ROBOT_THREAD_WORKERS``. ``0`` never replaces them. An invalid value is ignored with a warning. See `Call timeouts and cancellation`. |
     | ROBOT_LOGGER_MAPPER    | Default ``logger_mapper`` as a ``module.function`` path, used when the argument is not given. See `Custom logger mappers`. |
 
     = When not to use it =
@@ -312,6 +365,8 @@ class ParallelRunner:
 
     def __init__(self):
         self.max_workers = self._workers_from_environment()
+        # None: twice the number of workers (see `_abandoned_thread_limit`).
+        self.max_abandoned_threads = self._abandoned_limit_from_environment()
 
     def _workers_from_environment(self) -> int:
         """``ROBOT_THREAD_WORKERS``, or 4. An invalid value is ignored with a
@@ -332,6 +387,30 @@ class ParallelRunner:
             return self.DEFAULT_WORKERS
         return workers
 
+    @staticmethod
+    def _abandoned_limit_from_environment() -> Optional[int]:
+        """``ROBOT_MAX_ABANDONED_THREADS``, or None for the default. An invalid
+        value is ignored with a warning, like ``ROBOT_THREAD_WORKERS``."""
+        value = os.getenv("ROBOT_MAX_ABANDONED_THREADS")
+        if _not_given(value):
+            return None
+        try:
+            limit = int(value)
+        except ValueError:
+            limit = -1
+        if limit < 0:
+            logger.warn(
+                f"Ignoring ROBOT_MAX_ABANDONED_THREADS {value!r}: expected a whole number of at least 0. "
+                "Using twice the number of workers."
+            )
+            return None
+        return limit
+
+    def _abandoned_thread_limit(self) -> int:
+        if self.max_abandoned_threads is None:
+            return 2 * self.max_workers
+        return self.max_abandoned_threads
+
     def run_parallel_scenarios(
         self,
         keyword: str,
@@ -344,6 +423,9 @@ class ParallelRunner:
         return_values_only: bool = False,
         fail_on_error: bool = True,
         call_timeout: Union[str, float, None] = None,
+        call_retries: Union[int, str, None] = 0,
+        retry_delay: Union[str, float, None] = "1s",
+        retry_backoff: Union[float, str, None] = 1,
         **kwargs
     ) -> Union[List[Dict[str, Any]], Tuple[Any, ...]]:
         """Runs ``keyword`` from ``library`` concurrently, once per item or N times.
@@ -388,6 +470,13 @@ class ParallelRunner:
           calls keep running; calls that declare ``_cancelled`` are told to
           stop. Not given (default): no limit. See
           `Call timeouts and cancellation`.
+        - ``call_retries``: How many more times a failed call is tried
+          (``0``, default: none). A call that exceeds ``call_timeout`` is
+          retried too. See `Retries`.
+        - ``retry_delay``: Wait before the first retry, in Robot's time format.
+          Default ``1s``; ``0`` retries right away.
+        - ``retry_backoff``: Multiplies the wait after every failed retry:
+          ``2`` waits 1 s, 2 s, 4 s... Default ``1`` (always the same wait).
         - ``**kwargs``: Any other named argument is passed to every call,
           converted by the method's type hints like in a direct call.
 
@@ -403,6 +492,7 @@ class ParallelRunner:
         | ${id1}    ${id2}    ${id3}= | Run Parallel Scenarios | keyword=Seed Record | library=MyLibrary | repeat=3 | return_values_only=True |
         | ${results}=   | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary       | for_loop_iterable=${agents} | fail_on_error=False |
         | ${results}=   | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary       | for_loop_iterable=${agents} | call_timeout=30s |
+        | ${results}=   | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary       | for_loop_iterable=${agents} | call_retries=2 | retry_delay=500ms |
         """
         # Determine items to process. NOTE: `repeat or 1` would be wrong here -
         # 0 is falsy in Python, so an explicit repeat=0 would silently fall
@@ -422,6 +512,7 @@ class ParallelRunner:
 
         min_level = self._resolve_thread_log_level(thread_log_level)
         timeout_seconds = self._resolve_call_timeout(call_timeout)
+        retry = RetryPolicy.from_arguments(call_retries, retry_delay, retry_backoff)
 
         # Note: We must call the underlying Python method, not BuiltIn().run_keyword
         # because run_keyword is not thread-safe.
@@ -429,7 +520,7 @@ class ParallelRunner:
         method = self._get_keyword_method(lib_instance, keyword)
 
         results = self._run_in_threads(
-            method, list(items), effective_mapper, remove_passing_logs, min_level, kwargs, timeout_seconds
+            method, list(items), effective_mapper, remove_passing_logs, min_level, kwargs, timeout_seconds, retry
         )
 
         # Step: Re-play logs into Robot Framework sequentially
@@ -616,15 +707,25 @@ class ParallelRunner:
         min_level: int,
         kwargs: Dict[str, Any],
         call_timeout: Optional[float] = None,
+        retry: RetryPolicy = NO_RETRIES,
     ) -> List[Dict[str, Any]]:
         """
         Runs ``method`` once per item on up to ``max_workers`` daemon threads
         and returns the result dicts in call order (NOT completion order).
 
+        A failed attempt is tried again while ``retry`` allows it: the worker
+        puts the call back in the queue with the time it may start again, and
+        whichever worker takes it waits until then. ``call_timeout`` applies
+        to each attempt, and a timed-out attempt is retried only while the
+        abandoned threads are within their limit.
+
         With ``call_timeout``, a call still running after that many seconds
         becomes a ``FAIL`` result: its thread is abandoned (it can't be
         killed) and a new worker takes its place, so the other calls keep
-        running ``max_workers`` at a time.
+        running ``max_workers`` at a time - while no more than the limit of
+        abandoned threads (``_abandoned_thread_limit``) are alive. Past it,
+        no replacements are started; if every worker is gone, the calls that
+        did not start fail right away.
 
         If the wait is interrupted (mostly by a Robot ``[Timeout]``), the
         workers stop starting new calls, the logs collected so far are
@@ -647,13 +748,27 @@ class ParallelRunner:
         # Created here, not in the workers, so the logs of calls that are still
         # running can be replayed if the keyword is interrupted or times out.
         log_buffers: List[list] = [[] for _ in items]
-        # Written by the workers. A worker writes `started_at[i]` when call i
-        # starts and `finished[i]` when it has returned - unless the main
-        # thread has timed the call out first. `claims` decides which of the
-        # two wins: dict.setdefault is atomic, so the first claim sticks.
+        # Written by the workers. When an attempt of call i starts, a worker
+        # writes `threads[i]`, `attempts[i]` (its number, 1-based) and then
+        # `attempt_started[i]` - in that order. `attempt_started[i]` is None
+        # while the call waits for a retry (it is cleared before the retry is
+        # queued), and the main thread reads `attempts[i]` first: so it never
+        # pairs an attempt with the start time of an earlier one, which would
+        # time the new attempt out too soon. `started_at[i]` is the start of
+        # the first attempt. `finished[i]`
+        # is written when the last attempt has returned - unless the main
+        # thread has timed the attempt out first. `claims`, keyed by (call,
+        # attempt), decides which of the two wins: dict.setdefault is atomic,
+        # so the first claim sticks.
         started_at: List[Optional[float]] = [None] * count
+        attempt_started: List[Optional[float]] = [None] * count
+        attempts: List[int] = [0] * count
+        threads: List[Optional[threading.Thread]] = [None] * count
         finished: List[Optional[Dict[str, Any]]] = [None] * count
-        claims: Dict[int, str] = {}
+        claims: Dict[Tuple[int, int], str] = {}
+        # One entry per worker that ended because there was nothing left to
+        # run (or the batch was interrupted), not because it was abandoned.
+        exited: List[int] = []
         # Written by the main thread only.
         results: List[Optional[Dict[str, Any]]] = [None] * count
 
@@ -672,36 +787,67 @@ class ParallelRunner:
                     message = str(error)
                     log_buffers[index].append(("ERROR", f"Thread failed for item {item}: {message}"))
                     started_at[index] = time.monotonic()
-                    finished[index] = {"status": "FAIL", "item": item, "logs": log_buffers[index], "error": message}
+                    finished[index] = {
+                        "status": "FAIL", "item": item, "logs": log_buffers[index], "error": message, "attempts": 0,
+                    }
                     continue
-            pending.put(index)
+            # (call, attempt, earliest start): retries go back in the queue too.
+            pending.put((index, 1, 0.0))
         stop = threading.Event()
         workers_started = 0
 
         def worker():
-            while not stop.is_set():
-                try:
-                    index = pending.get_nowait()
-                except queue.Empty:
-                    return
-                call_kwargs = kwargs if cancel_events is None else {**kwargs, "_cancelled": cancel_events[index]}
-                started_at[index] = time.monotonic()
-                result = self._execute_and_capture(
-                    method, items[index], logger_mapper, log_buffers[index], call_kwargs, arguments[index]
-                )
-                if claims.setdefault(index, _WORKER) != _WORKER:
-                    # The call timed out and a new worker took this thread's
-                    # place: stop here, so no more than max_workers calls run.
-                    return
-                finished[index] = result
+            abandoned = False
+            try:
+                while not stop.is_set():
+                    try:
+                        index, attempt, ready_at = pending.get_nowait()
+                    except queue.Empty:
+                        return
+                    wait = ready_at - time.monotonic()
+                    if wait > 0 and stop.wait(wait):
+                        return
+                    call_kwargs = kwargs if cancel_events is None else {**kwargs, "_cancelled": cancel_events[index]}
+                    log_buffer = log_buffers[index]
+                    threads[index] = threading.current_thread()
+                    attempts[index] = attempt
+                    now = time.monotonic()
+                    attempt_started[index] = now
+                    if started_at[index] is None:
+                        started_at[index] = now
+                    last = not retry.can_retry(attempt)
+                    result = self._execute_and_capture(
+                        method, items[index], logger_mapper, log_buffer, call_kwargs, arguments[index],
+                        log_failure=last,
+                    )
+                    if claims.setdefault((index, attempt), _WORKER) != _WORKER:
+                        # The attempt timed out and a new worker took this
+                        # thread's place: stop here, so no more than
+                        # max_workers calls run.
+                        abandoned = True
+                        return
+                    if result["status"] == "FAIL" and not last:
+                        attempt_started[index] = None
+                        log_buffer.append(("WARN", retry.failed_message(attempt, result["error"])))
+                        pending.put((index, attempt + 1, time.monotonic() + retry.delay_after(attempt)))
+                    else:
+                        result["attempts"] = attempt
+                        finished[index] = result
+            finally:
+                if not abandoned:
+                    exited.append(1)
 
         def start_worker():
             nonlocal workers_started
             workers_started += 1
             threading.Thread(target=worker, name=f"ParallelRunner-worker-{workers_started}", daemon=True).start()
 
+        workers_wanted = min(self.max_workers, count)
+        limit = self._abandoned_thread_limit()
+        abandoned_here = 0
+        limit_warned = False
         try:
-            for _ in range(min(self.max_workers, count)):
+            for _ in range(workers_wanted):
                 start_worker()
             # Deliberately no lock, Event or Future wait here: a timeout raised
             # asynchronously while such a wait holds an internal lock could
@@ -721,20 +867,65 @@ class ParallelRunner:
                     if finished[index] is not None:
                         results[index] = finished[index]
                         remaining -= 1
-                    elif (
-                        call_timeout is not None
-                        and now - started_at[index] >= call_timeout
-                        and claims.setdefault(index, _MAIN) == _MAIN
+                        continue
+                    attempt = attempts[index]  # before attempt_started: see above
+                    began = attempt_started[index]
+                    if (
+                        call_timeout is None
+                        or began is None
+                        or now - began < call_timeout
+                        or claims.setdefault((index, attempt), _MAIN) != _MAIN
                     ):
-                        results[index] = self._timed_out_result(items[index], log_buffers[index], call_timeout)
-                        remaining -= 1
+                        still_running.append(index)
+                        continue
+                    if cancel_events is not None:
+                        cancel_events[index].set()
+                    _abandoned_threads.append(threads[index])
+                    abandoned_here += 1
+                    retrying = retry.can_retry(attempt)
+                    if retrying and _abandoned_threads_alive() <= limit:
+                        # The abandoned attempt keeps the old buffer and event:
+                        # its late logs don't mix with the next attempt's.
+                        duration = secs_to_timestr(call_timeout)
+                        log_buffers[index] = log_buffers[index] + [
+                            ("WARN", retry.timed_out_message(attempt, duration))
+                        ]
                         if cancel_events is not None:
-                            cancel_events[index].set()
-                        if not pending.empty():
-                            start_worker()
-                    else:
+                            cancel_events[index] = threading.Event()
+                        attempt_started[index] = None
+                        pending.put((index, attempt + 1, now + retry.delay_after(attempt)))
                         still_running.append(index)
+                    else:
+                        results[index] = self._timed_out_result(
+                            items[index], log_buffers[index], call_timeout, attempt,
+                            not_retried_limit=limit if retrying else None,
+                        )
+                        remaining -= 1
                 running = still_running
+                # Replace the workers abandoned by timed-out calls (or ended
+                # while a timed-out call was put back in the queue) while
+                # calls are still waiting - unless too many abandoned threads
+                # are alive (from this keyword call or earlier ones). Checked
+                # on every round, so a slot freed by a hung call that finally
+                # returns is used again.
+                active = workers_started - abandoned_here - len(exited)
+                missing = workers_wanted - active
+                if missing > 0 and not pending.empty():
+                    alive = _abandoned_threads_alive()
+                    if alive <= limit:
+                        for _ in range(missing):
+                            start_worker()
+                    else:
+                        if not limit_warned:
+                            limit_warned = True
+                            logger.warn(
+                                f"{alive} thread(s) abandoned by timed-out calls are still running, more than the "
+                                f"limit of {limit} (ROBOT_MAX_ABANDONED_THREADS): timed-out workers are no longer "
+                                "replaced, so the remaining calls run with fewer workers."
+                            )
+                        if active == 0:
+                            # No worker left to run them: fail them now instead of waiting forever.
+                            self._fail_not_started(pending, items, started_at, finished, log_buffers, limit)
                 if remaining:
                     time.sleep(_POLL_INTERVAL)
         except BaseException as error:
@@ -742,6 +933,10 @@ class ParallelRunner:
             if cancel_events is not None:
                 for event in cancel_events:
                     event.set()
+            # The calls still running are abandoned too: they count towards the limit.
+            for index in range(count):
+                if attempt_started[index] is not None and results[index] is None and finished[index] is None:
+                    _abandoned_threads.append(threads[index])
             self._replay_interrupted(
                 items, results, finished, started_at, log_buffers, error, remove_passing_logs, min_level
             )
@@ -749,12 +944,42 @@ class ParallelRunner:
         return results
 
     @staticmethod
-    def _timed_out_result(item, log_buffer: list, call_timeout: float) -> Dict[str, Any]:
+    def _timed_out_result(
+        item, log_buffer: list, call_timeout: float, attempt: int = 1, not_retried_limit: Optional[int] = None
+    ) -> Dict[str, Any]:
+        """``not_retried_limit``: the abandoned thread limit, when it is what
+        kept the call from being retried (not given: no retries were left)."""
         duration = secs_to_timestr(call_timeout)
         logs = list(log_buffer)
         logs.append(("ERROR", f"Call for item {item} timed out after {duration} (call_timeout); "
                               "it keeps running in the background"))
-        return {"status": "FAIL", "item": item, "logs": logs, "error": f"Timed out after {duration}"}
+        if not_retried_limit is not None:
+            logs.append(("ERROR", f"Call for item {item} not retried: the abandoned thread limit "
+                                  f"({not_retried_limit}, ROBOT_MAX_ABANDONED_THREADS) was reached"))
+        return {"status": "FAIL", "item": item, "logs": logs, "error": f"Timed out after {duration}", "attempts": attempt}
+
+    @staticmethod
+    def _fail_not_started(pending, items, started_at, finished, log_buffers, limit: int) -> None:
+        """Fails every call still waiting in ``pending``. Only called when no
+        worker is left, so nothing else reads the queue. The results are
+        picked up by the main loop like those of finished calls."""
+        while True:
+            try:
+                index, attempt, _ = pending.get_nowait()
+            except queue.Empty:
+                return
+            item = items[index]
+            log_buffers[index].append((
+                "ERROR",
+                f"Call for item {item} not started: every worker was abandoned by a timed-out call and "
+                f"the abandoned thread limit ({limit}, ROBOT_MAX_ABANDONED_THREADS) was reached",
+            ))
+            finished[index] = {
+                "status": "FAIL", "item": item, "logs": log_buffers[index],
+                "error": f"Not started: abandoned thread limit ({limit}) reached", "attempts": attempt - 1,
+            }
+            if started_at[index] is None:
+                started_at[index] = time.monotonic()
 
     def _replay_interrupted(self, items, results, finished, started_at, log_buffers, error, remove_passing_logs, min_level):
         """Replays what the calls logged before the keyword was interrupted.
@@ -784,10 +1009,13 @@ class ParallelRunner:
 
     def _execute_and_capture(
         self, method, item, logger_mapper: Optional[Any], log_buffer: list, kwargs: Dict[str, Any],
-        argument: Any = _NOT_SET,
+        argument: Any = _NOT_SET, log_failure: bool = True,
     ) -> Dict[str, Any]:
         """Worker wrapper to capture logs and results.
 
+        ``log_failure=False`` (an attempt that will be retried) leaves out the
+        ``ERROR`` line of a failure; the caller logs the attempt as a warning.
+
         ``argument`` is what the call receives as its first argument (``item``
         converted by its type hint), ``item`` what the result reports. ``kwargs``
         is a dict, not ``**kwargs``, so a keyword argument that happens to be
@@ -813,7 +1041,8 @@ class ParallelRunner:
         # SystemExit), the worker must report a result, or the main thread
         # would wait for this call forever.
         except BaseException as e:
-            thread_log(f"Thread failed for item {item}: {str(e)}", "ERROR")
+            if log_failure:
+                thread_log(f"Thread failed for item {item}: {str(e)}", "ERROR")
             return {"status": "FAIL", "item": item, "logs": log_buffer, "error": str(e)}
 
     def _replay_logs(self, results: List[Dict[str, Any]], remove_passing_logs: bool = False, min_level: int = 0):

robotframework_parallelrunner-0.5.0/src/ParallelRunner/retry.py

@@ -0,0 +1,114 @@
+"""Retries of failed calls in `Run Parallel Scenarios`: the ``call_retries``,
+``retry_delay`` and ``retry_backoff`` arguments.
+
+This module only decides *whether* and *when* a call is retried, and what is
+logged between attempts. Running the attempts on the worker threads (and
+retrying calls that exceeded ``call_timeout``) is done by
+``ParallelRunner._run_in_threads``.
+"""
+
+import math
+from dataclasses import dataclass
+from typing import Any
+
+from robot.errors import DataError
+from robot.utils import secs_to_timestr, timestr_to_secs
+
+DEFAULT_RETRY_DELAY = 1.0
+
+
+def _not_given(value: Any) -> bool:
+    """None, an empty string or "None" (any case), like in parallel_library."""
+    return value is None or (isinstance(value, str) and value.strip().upper() in ("", "NONE"))
+
+
+@dataclass(frozen=True)
+class RetryPolicy:
+    """How often and how long apart a failed call is tried again.
+
+    ``retries`` extra attempts after the first one; before attempt ``n + 1``
+    the call waits ``delay * backoff ** (n - 1)`` seconds (``backoff=1``: a
+    fixed delay; ``backoff=2``: 1 s, 2 s, 4 s...).
+    """
+
+    retries: int = 0
+    delay: float = DEFAULT_RETRY_DELAY
+    backoff: float = 1.0
+
+    @property
+    def attempts(self) -> int:
+        """Maximum number of attempts per call."""
+        return self.retries + 1
+
+    def can_retry(self, attempt: int) -> bool:
+        """Whether a call whose attempt number ``attempt`` (1-based) failed may run again."""
+        return attempt < self.attempts
+
+    def delay_after(self, attempt: int) -> float:
+        """Seconds to wait between failed attempt ``attempt`` and the next one."""
+        return self.delay * self.backoff ** (attempt - 1)
+
+    def failed_message(self, attempt: int, error: str) -> str:
+        return f"Attempt {attempt} of {self.attempts} failed: {error}; {self._retrying(attempt)}"
+
+    def timed_out_message(self, attempt: int, duration: str) -> str:
+        return (
+            f"Attempt {attempt} of {self.attempts} timed out after {duration} (call_timeout), "
+            f"it keeps running in the background; {self._retrying(attempt)}"
+        )
+
+    def _retrying(self, attempt: int) -> str:
+        delay = self.delay_after(attempt)
+        return f"retrying in {secs_to_timestr(delay)}" if delay > 0 else "retrying now"
+
+    @classmethod
+    def from_arguments(cls, call_retries: Any = None, retry_delay: Any = None, retry_backoff: Any = None) -> "RetryPolicy":
+        """Validates the keyword arguments, in Robot's formats: ``call_retries``
+        a whole number, ``retry_delay`` a Robot time (``1s``, ``500ms``),
+        ``retry_backoff`` a number. Not given: no retries, a 1 second delay,
+        no backoff. An invalid value raises ``ValueError`` before anything runs."""
+        return cls(
+            retries=cls._resolve_retries(call_retries),
+            delay=cls._resolve_delay(retry_delay),
+            backoff=cls._resolve_backoff(retry_backoff),
+        )
+
+    @staticmethod
+    def _resolve_retries(value: Any) -> int:
+        if _not_given(value):
+            return 0
+        try:
+            # Through str: rejects 1.5 (and True) instead of truncating it.
+            retries = int(str(value).strip())
+        except ValueError:
+            retries = -1
+        if retries < 0:
+            raise ValueError(f"Invalid call_retries {value!r}: expected a whole number of at least 0.")
+        return retries
+
+    @staticmethod
+    def _resolve_delay(value: Any) -> float:
+        if _not_given(value):
+            return DEFAULT_RETRY_DELAY
+        try:
+            seconds = timestr_to_secs(value)
+        except (ValueError, DataError) as error:
+            raise ValueError(f"Invalid retry_delay {value!r}: {error}") from None
+        if seconds < 0:
+            raise ValueError(f"retry_delay can't be negative, got {value!r}.")
+        return seconds
+
+    @staticmethod
+    def _resolve_backoff(value: Any) -> float:
+        if _not_given(value):
+            return 1.0
+        try:
+            backoff = float(str(value).strip())
+        except ValueError:
+            raise ValueError(f"Invalid retry_backoff {value!r}: expected a number of at least 1.") from None
+        if not 1 <= backoff < math.inf:  # also rejects NaN
+            raise ValueError(f"retry_backoff must be at least 1, got {value!r}.")
+        return backoff
+
+
+NO_RETRIES = RetryPolicy()