robotframework-parallelrunner 0.2.0__tar.gz → 0.3.0__tar.gz

{robotframework_parallelrunner-0.2.0 → robotframework_parallelrunner-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: robotframework-parallelrunner
-Version: 0.2.0
+Version: 0.3.0
 Summary: Run a keyword in parallel inside a single Robot Framework test case, with one clean, ordered log.html.
 License-Expression: MIT
 License-File: LICENSE
@@ -112,10 +112,16 @@ threading, buffering, and replay logic stay out of your `.robot` files.
 
 ## Summary of Benefits
 
-- **Speed.** Validating 100 APIs that take 1s each takes roughly 10s (with 10
-  workers) instead of 100s.
+- **Speed.** Waiting time is overlapped instead of added up. In this repo's
+  examples, 9 HTTP checks with a 1 s delay each drop from 13 s to 2 s with
+  9 workers, and 4 suites × 8 items × 1 s drop from 32.7 s to 4.6 s with
+  8 workers (see [Using It Together With pabot](#using-it-together-with-pabot)).
 - **Integrity.** Your `log.html` remains the single source of truth — no
   broken XML tags from concurrent writes.
+- **No hidden failures.** If any call fails, the keyword fails too (after all
+  calls finish and their logs are replayed), with a message listing every
+  failed item. Pass `fail_on_error=False` to get a `PASS`/`FAIL` status per
+  item and check them yourself.
 - **Flexibility.** Pass `repeat=10` to stress-test a single endpoint, or
   `for_loop_iterable=${items}` for batch validation of a whole list.
 - **Ergonomics.** Results come back in call order (not completion order), so
@@ -164,6 +170,57 @@ for pabot. If you need to speed up "this one test case loops over 100 items
 and I want a single readable log instead of 100 sequential HTTP round trips
 (or 100 merged `output.xml` files)," that's what ParallelRunner is for.
 
+## Using It Together With pabot
+
+The two combine without any extra configuration: pabot splits suites (or
+tests) across processes, and inside each process ParallelRunner splits a
+loop across threads. Every pabot process gets its own ParallelRunner
+instance, its own thread pool and its own `output.xml`, so nothing is shared
+between processes, and pabot merges the results as usual.
+
+```bash
+pabot --processes 4 --pythonpath . atest/
+pabot --processes 4 --testlevelsplit --pythonpath . atest/
+```
+
+Both commands pass the full acceptance suite of this repository (52 tests;
+checked with pabot 5.2.2).
+
+How much time each one saves, from the offline
+[pabot timing demo](https://github.com/cristiangarciavd/robotframework-parallelrunner/tree/main/examples/pabot_demo/)
+(4 suites × 8 items × 1 s of simulated I/O wait; `poetry run invoke demo-pabot`):
+
+| Run | Wall-clock time | Speedup |
+|---|---|---|
+| robot, sequential loop | 32.7 s | ×1.0 |
+| robot + ParallelRunner (8 workers) | 4.6 s | ×7.1 |
+| pabot (4 processes), sequential loop | 10.6 s | ×3.1 |
+| pabot (4 processes) + ParallelRunner (8 workers) | 3.5 s | ×9.2 |
+
+ParallelRunner removes the waiting *inside* each test, and pabot removes it
+*between* suites. With suites this short, pabot's process start-up (about
+2.5 s here) is most of what's left, so the combination pays off more as
+suites get longer.
+
+Things to keep in mind:
+
+- **Run pabot from the virtual environment where the library is installed**
+  (or via `poetry run pabot ...`). pabot starts whichever `robot` command is
+  first on the `PATH`. If that belongs to another Python installation, every
+  test that uses the library fails with
+  `No keyword with name 'Run Parallel Scenarios' found`.
+- **Concurrency multiplies.** The number of simultaneous calls is roughly
+  pabot processes × `ROBOT_THREAD_WORKERS`: 4 processes × 4 workers means up
+  to 16 concurrent requests against the same API or database. Lower one of
+  the two if you hit rate limits, connection-pool limits or lock timeouts.
+- **Shared test resources can collide across processes.** Data, files, user
+  accounts, or a table that a `Suite Setup` resets are visible to every
+  process. With `--testlevelsplit`, each test of a suite runs its own
+  `Suite Setup` in its own process. For example, the `examples/db_seed`
+  suite drops and recreates its table in `Suite Setup`, so two of its tests
+  could interfere with each other. Keep shared state per process, or split
+  by suite.
+
 ## When Not To Use This
 
 - **CPU-bound work.** Threads share Python's GIL — number crunching won't
@@ -183,11 +240,13 @@ and I want a single readable log instead of 100 sequential HTTP round trips
 ## Project Structure
 
 ```
-ParallelRunner/
-├── src/ParallelRunner/     # The installable library (core, do not depend on internals prefixed with `_`)
-├── examples/               # Example "business logic" libraries used by the test suites
-├── atest/            # Robot Framework acceptance suites
-├── docs/                   # INSTALLATION, QUICKSTART, API_REFERENCE
+robotframework-parallelrunner/
+├── src/ParallelRunner/     # The installable library (do not depend on internals prefixed with `_`)
+├── examples/               # Example libraries used by the tests, plus the pabot timing demo
+├── atest/                  # Robot Framework acceptance suites
+├── utest/                  # pytest unit tests
+├── docs/                   # INSTALLATION, QUICKSTART, API_REFERENCE, generated keyword docs
+├── tasks.py                # invoke tasks: tests, libdoc, demo-pabot, build, ...
 └── ARCHITECTURE.md         # Technical deep dive
 ```
 

{robotframework_parallelrunner-0.2.0 → robotframework_parallelrunner-0.3.0}/README.md

@@ -78,10 +78,16 @@ threading, buffering, and replay logic stay out of your `.robot` files.
 
 ## Summary of Benefits
 
-- **Speed.** Validating 100 APIs that take 1s each takes roughly 10s (with 10
-  workers) instead of 100s.
+- **Speed.** Waiting time is overlapped instead of added up. In this repo's
+  examples, 9 HTTP checks with a 1 s delay each drop from 13 s to 2 s with
+  9 workers, and 4 suites × 8 items × 1 s drop from 32.7 s to 4.6 s with
+  8 workers (see [Using It Together With pabot](#using-it-together-with-pabot)).
 - **Integrity.** Your `log.html` remains the single source of truth — no
   broken XML tags from concurrent writes.
+- **No hidden failures.** If any call fails, the keyword fails too (after all
+  calls finish and their logs are replayed), with a message listing every
+  failed item. Pass `fail_on_error=False` to get a `PASS`/`FAIL` status per
+  item and check them yourself.
 - **Flexibility.** Pass `repeat=10` to stress-test a single endpoint, or
   `for_loop_iterable=${items}` for batch validation of a whole list.
 - **Ergonomics.** Results come back in call order (not completion order), so
@@ -130,6 +136,57 @@ for pabot. If you need to speed up "this one test case loops over 100 items
 and I want a single readable log instead of 100 sequential HTTP round trips
 (or 100 merged `output.xml` files)," that's what ParallelRunner is for.
 
+## Using It Together With pabot
+
+The two combine without any extra configuration: pabot splits suites (or
+tests) across processes, and inside each process ParallelRunner splits a
+loop across threads. Every pabot process gets its own ParallelRunner
+instance, its own thread pool and its own `output.xml`, so nothing is shared
+between processes, and pabot merges the results as usual.
+
+```bash
+pabot --processes 4 --pythonpath . atest/
+pabot --processes 4 --testlevelsplit --pythonpath . atest/
+```
+
+Both commands pass the full acceptance suite of this repository (52 tests;
+checked with pabot 5.2.2).
+
+How much time each one saves, from the offline
+[pabot timing demo](https://github.com/cristiangarciavd/robotframework-parallelrunner/tree/main/examples/pabot_demo/)
+(4 suites × 8 items × 1 s of simulated I/O wait; `poetry run invoke demo-pabot`):
+
+| Run | Wall-clock time | Speedup |
+|---|---|---|
+| robot, sequential loop | 32.7 s | ×1.0 |
+| robot + ParallelRunner (8 workers) | 4.6 s | ×7.1 |
+| pabot (4 processes), sequential loop | 10.6 s | ×3.1 |
+| pabot (4 processes) + ParallelRunner (8 workers) | 3.5 s | ×9.2 |
+
+ParallelRunner removes the waiting *inside* each test, and pabot removes it
+*between* suites. With suites this short, pabot's process start-up (about
+2.5 s here) is most of what's left, so the combination pays off more as
+suites get longer.
+
+Things to keep in mind:
+
+- **Run pabot from the virtual environment where the library is installed**
+  (or via `poetry run pabot ...`). pabot starts whichever `robot` command is
+  first on the `PATH`. If that belongs to another Python installation, every
+  test that uses the library fails with
+  `No keyword with name 'Run Parallel Scenarios' found`.
+- **Concurrency multiplies.** The number of simultaneous calls is roughly
+  pabot processes × `ROBOT_THREAD_WORKERS`: 4 processes × 4 workers means up
+  to 16 concurrent requests against the same API or database. Lower one of
+  the two if you hit rate limits, connection-pool limits or lock timeouts.
+- **Shared test resources can collide across processes.** Data, files, user
+  accounts, or a table that a `Suite Setup` resets are visible to every
+  process. With `--testlevelsplit`, each test of a suite runs its own
+  `Suite Setup` in its own process. For example, the `examples/db_seed`
+  suite drops and recreates its table in `Suite Setup`, so two of its tests
+  could interfere with each other. Keep shared state per process, or split
+  by suite.
+
 ## When Not To Use This
 
 - **CPU-bound work.** Threads share Python's GIL — number crunching won't
@@ -149,11 +206,13 @@ and I want a single readable log instead of 100 sequential HTTP round trips
 ## Project Structure
 
 ```
-ParallelRunner/
-├── src/ParallelRunner/     # The installable library (core, do not depend on internals prefixed with `_`)
-├── examples/               # Example "business logic" libraries used by the test suites
-├── atest/            # Robot Framework acceptance suites
-├── docs/                   # INSTALLATION, QUICKSTART, API_REFERENCE
+robotframework-parallelrunner/
+├── src/ParallelRunner/     # The installable library (do not depend on internals prefixed with `_`)
+├── examples/               # Example libraries used by the tests, plus the pabot timing demo
+├── atest/                  # Robot Framework acceptance suites
+├── utest/                  # pytest unit tests
+├── docs/                   # INSTALLATION, QUICKSTART, API_REFERENCE, generated keyword docs
+├── tasks.py                # invoke tasks: tests, libdoc, demo-pabot, build, ...
 └── ARCHITECTURE.md         # Technical deep dive
 ```
 

{robotframework_parallelrunner-0.2.0 → robotframework_parallelrunner-0.3.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "poetry.core.masonry.api"
 name = "robotframework-parallelrunner"
 # Single source of truth for the version: bump with `poetry version patch|minor|major`.
 # ParallelRunner.__version__ / ROBOT_LIBRARY_VERSION read it back from the installed metadata.
-version = "0.2.0"
+version = "0.3.0"
 description = "Run a keyword in parallel inside a single Robot Framework test case, with one clean, ordered log.html."
 readme = "README.md"
 license = "MIT"
@@ -72,6 +72,8 @@ coverage = ">=7.0"
 invoke = ">=2.0"
 # Used by the example libraries the acceptance tests run against (examples/).
 requests = ">=2.25"
+# Used by `invoke demo-pabot` (examples/pabot_demo/); not needed by the library itself.
+robotframework-pabot = ">=4.0"
 
 [tool.pytest.ini_options]
 testpaths = ["utest"]

{robotframework_parallelrunner-0.2.0 → robotframework_parallelrunner-0.3.0}/src/ParallelRunner/parallel_library.py

@@ -1,5 +1,6 @@
 import os
 import concurrent.futures
+import importlib
 from typing import Iterable, Any, List, Dict, Optional, Tuple, Union
 from robot.api import logger
 from robot.libraries.BuiltIn import BuiltIn
@@ -9,13 +10,14 @@ from .version import __version__
 
 class ParallelTaskError(RuntimeError):
     """
-    Raised when at least one parallel task failed and there is no meaningful
-    return value to hand back for it - by ``get_result_values``, and by
-    ``run_parallel_scenarios`` when called with ``return_values_only=True``.
+    Raised when at least one parallel task failed: by ``run_parallel_scenarios``
+    (unless called with ``fail_on_error=False``), and by ``get_result_values``,
+    which has no meaningful return value to hand back for a failed task.
 
-    Silently substituting `None` for a failed task's value would let a test
-    keep going with bad data instead of failing for the right reason, so
-    this is raised instead.
+    Returning normally after a failed call would let a test keep going (and
+    pass) with bad or missing data instead of failing for the right reason,
+    so this is raised instead - after every task has finished and all logs
+    have been replayed.
 
     Attributes:
         failures: the subset of result dicts with ``status == "FAIL"``,
@@ -90,15 +92,50 @@ class ParallelRunner:
     | result   | The return value of the call (only when ``status`` is ``PASS``). |
     | error    | The error message (only when ``status`` is ``FAIL``). |
 
-    A failing call does *not* fail the keyword; check ``status`` yourself,
-    or use ``return_values_only=True`` / `Get Result Values`, which fail
-    if any call failed.
+    = Failures =
+
+    If any call raises, `Run Parallel Scenarios` still waits for every other
+    call, replays all logs, and then *fails* with a ``ParallelTaskError``
+    that lists each failed item and its error, e.g.
+    ``2 of 5 parallel task(s) failed: item='4': ...; item='5': ...``.
+    One failed call is never hidden behind a passing keyword.
+
+    To inspect failures yourself instead, pass ``fail_on_error=False``: the
+    keyword then always returns the result list, failed calls get
+    ``status`` ``FAIL`` (their error is also logged at ``ERROR`` level), and
+    it is up to you to assert on them:
+
+    | ${results}= | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary | for_loop_iterable=${agents} | fail_on_error=False |
+    | Should Be Equal | ${results}[1][status] | PASS |
+
+    ``return_values_only=True`` and `Get Result Values` always fail if any
+    call failed, because a failed call has no return value.
+
+    = Custom logger mappers =
+
+    A logger mapper replaces the default buffering logger: the target keyword
+    receives it as ``_logger`` and every message goes straight to your own
+    logging system. Use it when a project already has its own log format
+    (severity codes, JSON...).
+
+    - Give it as a callable or as a ``module.function`` path, either with the
+      ``logger_mapper`` argument or, for a project-wide default, with the
+      ``ROBOT_LOGGER_MAPPER`` environment variable. The argument wins.
+    - Bare names (without a module path) are not supported. A value that
+      can't be resolved is ignored with a warning in the log.
+    - Mapper messages are *not* buffered and replayed into ``log.html``, so
+      ``thread_log_level`` and ``remove_passing_logs`` don't apply to them.
+      The mapper is called from several threads at once, so it must be
+      thread-safe.
+    - ``ROBOT_LOGGER_MAPPER`` must be a real environment variable (e.g. set
+      with ``Set Environment Variable`` or in the shell); a Robot variable
+      given with ``--variable`` is not read.
 
     = Environment variables =
 
     | =Variable=             | =Description= |
     | ROBOT_THREAD_WORKERS   | Number of worker threads. Read when the library is imported. Default ``4``. |
-    | ROBOT_LOGGER_MAPPER    | Default ``logger_mapper`` as a ``module.function`` path, used when the argument is not given. |
+    | ROBOT_LOGGER_MAPPER    | Default ``logger_mapper`` as a ``module.function`` path, used when the argument is not given. See `Custom logger mappers`. |
 
     = When not to use it =
 
@@ -125,6 +162,7 @@ class ParallelRunner:
         thread_log_level: str = "INFO",
         logger_mapper: Optional[Any] = None,
         return_values_only: bool = False,
+        fail_on_error: bool = True,
         **kwargs
     ) -> Union[List[Dict[str, Any]], Tuple[Any, ...]]:
         """Runs ``keyword`` from ``library`` concurrently, once per item or N times.
@@ -147,25 +185,33 @@ class ParallelRunner:
         - ``thread_log_level``: Minimum level replayed from the threads:
           ``INFO`` (default), ``WARN`` or ``ERROR``.
         - ``logger_mapper``: Optional callable ``mapper(msg, level)`` - or a
-          ``module.function`` path to one - injected as ``_logger`` instead of
-          the default buffering logger, to route logs to a custom logging
+          ``module.function`` path to one, such as
+          ``my_package.my_module.my_mapper`` - injected as ``_logger`` instead
+          of the default buffering logger, to route logs to a custom logging
           system. Falls back to the ``ROBOT_LOGGER_MAPPER`` environment
-          variable.
+          variable. A value that can't be resolved (e.g. a bare name without a
+          module) is ignored with a warning. See `Custom logger mappers`.
         - ``return_values_only``: If true, return a plain tuple of each call's
           return value (in call order) instead of the result dictionaries.
           Fails with ``ParallelTaskError`` if any call failed. Same as calling
           `Get Result Values` on the default return value.
+        - ``fail_on_error``: If true (default), fail with ``ParallelTaskError``
+          when any call failed, after all calls have finished and their logs
+          have been replayed. If false, return the results anyway and leave
+          checking ``status`` to the caller. See `Failures`.
         - ``**kwargs``: Any other named argument is passed to every call.
 
         Returns a list of result dictionaries (see `Result format`), in call
         order: entry ``i`` belongs to ``for_loop_iterable[i]`` / repeat
-        index ``i``, regardless of which thread finished first.
+        index ``i``, regardless of which thread finished first. With the
+        default ``fail_on_error=True`` it only returns when every call passed.
 
         Examples:
         | ${agents}=    | Create List            | 1                         | 2                       | 3 |
         | ${results}=   | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary       | for_loop_iterable=${agents} |
         | ${results}=   | Run Parallel Scenarios | keyword=Check Health      | library=MyLibrary       | repeat=8 | agent_id=1 |
         | ${id1}    ${id2}    ${id3}= | Run Parallel Scenarios | keyword=Seed Record | library=MyLibrary | repeat=3 | return_values_only=True |
+        | ${results}=   | Run Parallel Scenarios | keyword=Verify Agent Data | library=MyLibrary       | for_loop_iterable=${agents} | fail_on_error=False |
         """
         # Determine items to process. NOTE: `repeat or 1` would be wrong here -
         # 0 is falsy in Python, so an explicit repeat=0 would silently fall
@@ -205,6 +251,8 @@ class ParallelRunner:
 
         if return_values_only:
             return self.get_result_values(results)
+        if fail_on_error:
+            self._raise_if_any_failed(results)
         return results
 
     def get_result_values(self, results: List[Dict[str, Any]]) -> Tuple[Any, ...]:
@@ -215,7 +263,9 @@ class ParallelRunner:
 
         Passing ``return_values_only=True`` to `Run Parallel Scenarios` does
         the same in one step; use this keyword when you want to inspect the
-        full results first (e.g. assert on ``status`` or ``logs``).
+        full results first (e.g. assert on ``status`` or ``logs``). Such a
+        result list may only contain failed calls if it came from a run with
+        ``fail_on_error=False``.
 
         Fails with ``ParallelTaskError`` if any entry has status ``FAIL`` -
         a failed call has no return value to put in its slot.
@@ -224,57 +274,67 @@ class ParallelRunner:
         | ${results}=                 | Run Parallel Scenarios | keyword=Seed Record | library=MyLibrary | repeat=3 |
         | ${id1}    ${id2}    ${id3}= | Get Result Values      | ${results}          |
         """
+        self._raise_if_any_failed(results)
+        return tuple(entry["result"] for entry in results)
+
+    def _raise_if_any_failed(self, results: List[Dict[str, Any]]) -> None:
         failures = [entry for entry in results if entry.get("status") == "FAIL"]
         if failures:
             raise ParallelTaskError(failures, len(results))
-        return tuple(entry["result"] for entry in results)
 
-    def _resolve_mapper(self, mapper: Optional[Any]) -> Optional[Any]:
+    def _resolve_mapper(self, mapper: Optional[Any], source: str = "logger_mapper") -> Optional[Any]:
         """
-        Resolve a mapper which might be a callable or a string path to a module function.
-        
+        Resolve a mapper given as a callable or as a ``module.function`` path
+        (e.g. "examples.custom_logger.custom_logging_mapper.custom_logger_adapter").
+
+        Returns the callable, or None when no mapper was given. A mapper that
+        was given but can't be resolved (a bare name without a module, a module
+        that can't be imported, a missing or non-callable attribute) is ignored
+        with a warning in the log - it used to be ignored silently, which hid
+        typos and unsupported values such as bare registered names.
+
         Args:
-            mapper: Callable or string path (e.g., "examples.custom_logger.custom_logging_mapper.custom_logger_adapter")
-        
-        Returns:
-            Callable: The mapper function if successfully resolved, None otherwise.
+            mapper: Callable, ``module.function`` string, or "not given": None,
+                an empty string, or the string "None" (any case - Robot
+                Framework's usual spelling of None, e.g. ``logger_mapper=None``).
+            source: Where the value came from, used in the warning message.
         """
-        if mapper is None:
+        if mapper is None or (isinstance(mapper, str) and mapper.strip().upper() in ("", "NONE")):
             return None
-        
-        # If already callable, return it
+
         if callable(mapper):
             return mapper
-        
-        # If it's a string, try to import it
+
         if isinstance(mapper, str):
-            try:
-                parts = mapper.rsplit(".", 1)
-                if len(parts) == 2:
-                    module_name, func_name = parts
-                    module = __import__(module_name, fromlist=[func_name])
+            module_name, _, func_name = mapper.strip().rpartition(".")
+            if not module_name or not func_name:
+                reason = "expected a 'module.function' path such as 'my_package.my_module.my_mapper'"
+            else:
+                try:
+                    module = importlib.import_module(module_name)
+                except ImportError as error:
+                    reason = f"cannot import module '{module_name}' ({error})"
+                else:
                     resolved = getattr(module, func_name, None)
                     if callable(resolved):
                         return resolved
-            except (ImportError, AttributeError):
-                pass
-        
+                    reason = f"module '{module_name}' has no callable '{func_name}'"
+        else:
+            reason = f"expected a callable or a 'module.function' string, got {type(mapper).__name__}"
+
+        logger.warn(f"Ignoring {source} {mapper!r}: {reason}.")
         return None
 
     def _load_mapper_from_environment(self) -> Optional[Any]:
         """
-        Load custom logger mapper from ROBOT_LOGGER_MAPPER environment variable.
-        Supports both registered mapper names and module.function paths.
-        
+        Load the logger mapper named by the ROBOT_LOGGER_MAPPER environment
+        variable, which must be a ``module.function`` path (bare names are not
+        supported and are ignored with a warning).
+
         Returns:
             Callable: The mapper function if found, None otherwise.
         """
-        mapper_name = os.getenv("ROBOT_LOGGER_MAPPER")
-        if not mapper_name:
-            return None
-        
-        # Use _resolve_mapper to handle the string path
-        return self._resolve_mapper(mapper_name)
+        return self._resolve_mapper(os.getenv("ROBOT_LOGGER_MAPPER"), source="ROBOT_LOGGER_MAPPER")
 
     def _get_library_instance_owning_keyword(self, keyword_name: str, library: str) -> Any:
         method_name = keyword_name.replace(" ", "_").lower()