ocrd 3.0.0b1__tar.gz → 3.0.0b3__tar.gz

{ocrd-3.0.0b1/src/ocrd.egg-info → ocrd-3.0.0b3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ocrd
-Version: 3.0.0b1
+Version: 3.0.0b3
 Summary: OCR-D framework
 Author-email: Konstantin Baierer <unixprog@gmail.com>
 License: Apache License 2.0
@@ -94,17 +94,12 @@ complete stack of OCR-D-related software.
 
 The easiest way to install is via `pip`:
 
-```sh
-pip install ocrd
+    pip install ocrd
 
-# or just the functionality you need, e.g.
-
-pip install ocrd_modelfactory
-```
 
 All Python software released by [OCR-D](https://github.com/OCR-D) requires Python 3.8 or higher.
 
-**NOTE** Some OCR-D-Tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
+> **NOTE** Some OCR-D tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
 * using a custom build of [ImageMagick](https://github.com/ImageMagick/ImageMagick), whose format delegates are different from what OCR-D supposes
 * custom Python logging configurations in your personal account
 
@@ -129,7 +124,6 @@ Almost all behaviour of the OCR-D/core software is configured via CLI options an
 
 Some parts of the software are configured via environment variables:
 
-* `OCRD_METS_CACHING`: If set to `true`, access to the METS file is cached, speeding in-memory search and modification.
 * `OCRD_PROFILE`: This variable configures the built-in CPU and memory profiling. If empty, no profiling is done. Otherwise expected to contain any of the following tokens:
   * `CPU`: Enable CPU profiling of processor runs
   * `RSS`: Enable RSS memory profiling
@@ -142,18 +136,46 @@ Some parts of the software are configured via environment variables:
 * `XDG_CONFIG_HOME`: Directory to look for `./ocrd/resources.yml` (i.e. `ocrd resmgr` user database) – defaults to `$HOME/.config`.
 * `XDG_DATA_HOME`: Directory to look for `./ocrd-resources/*` (i.e. `ocrd resmgr` data location) – defaults to `$HOME/.local/share`.
 
-* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of workspace files.
+* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of resources or workspace files.
 * `OCRD_DOWNLOAD_TIMEOUT`: Timeout in seconds for connecting or reading (comma-separated) when downloading.
 
+* `OCRD_MISSING_INPUT`: How to deal with missing input files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed with next page's input
+  * `ABORT`: throw `MissingInputFile` exception
+
+* `OCRD_MISSING_OUTPUT`: How to deal with missing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `COPY`: fall back to copying input PAGE to output fileGrp for page
+  * `ABORT`: re-throw whatever caused processing to fail
+
+* `OCRD_MAX_MISSING_OUTPUTS`: Maximal rate of skipped/fallback pages among all processed pages before aborting (decimal fraction, ignored if negative).
+
+* `OCRD_EXISTING_OUTPUT`: How to deal with already existing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `OVERWRITE`: force writing result to output fileGrp for page
+  * `ABORT`: re-throw `FileExistsError` exception
+
+
 * `OCRD_METS_CACHING`: Whether to enable in-memory storage of OcrdMets data structures for speedup during processing or workspace operations.
 
 * `OCRD_MAX_PROCESSOR_CACHE`: Maximum number of processor instances (for each set of parameters) to be kept in memory (including loaded models) for processing workers or processor servers.
 
+* `OCRD_MAX_PARALLEL_PAGES`: Maximum number of processor threads for page-parallel processing (within each Processor's selected page range, independent of the number of Processing Workers or Processor Servers). If set `>1`, then a METS Server must be used for METS synchronisation.
+
+* `OCRD_PROCESSING_PAGE_TIMEOUT`: Timeout in seconds for processing a single page. If set >0, when exceeded, the same as OCRD_MISSING_OUTPUT applies.
+
 * `OCRD_NETWORK_SERVER_ADDR_PROCESSING`: Default address of Processing Server to connect to (for `ocrd network client processing`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKFLOW`: Default address of Workflow Server to connect to (for `ocrd network client workflow`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKSPACE`: Default address of Workspace Server to connect to (for `ocrd network client workspace`).
 * `OCRD_NETWORK_RABBITMQ_CLIENT_CONNECT_ATTEMPTS`: Number of attempts for a worker to create its queue. Helpful if the rabbitmq-server needs time to be fully started.
 
+* `OCRD_NETWORK_CLIENT_POLLING_SLEEP`: How many seconds to sleep before trying `ocrd network client` again.
+* `OCRD_NETWORK_CLIENT_POLLING_TIMEOUT`: Timeout for a blocking `ocrd network client` (in seconds).
+
+* `OCRD_NETWORK_SOCKETS_ROOT_DIR`: The root directory where all mets server related socket files are created.
+* `OCRD_NETWORK_LOGS_ROOT_DIR`: The root directory where all ocrd_network related file logs are stored.
+
+
 
 ## Packages
 

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/README.md

@@ -47,17 +47,12 @@ complete stack of OCR-D-related software.
 
 The easiest way to install is via `pip`:
 
-```sh
-pip install ocrd
+    pip install ocrd
 
-# or just the functionality you need, e.g.
-
-pip install ocrd_modelfactory
-```
 
 All Python software released by [OCR-D](https://github.com/OCR-D) requires Python 3.8 or higher.
 
-**NOTE** Some OCR-D-Tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
+> **NOTE** Some OCR-D tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
 * using a custom build of [ImageMagick](https://github.com/ImageMagick/ImageMagick), whose format delegates are different from what OCR-D supposes
 * custom Python logging configurations in your personal account
 
@@ -82,7 +77,6 @@ Almost all behaviour of the OCR-D/core software is configured via CLI options an
 
 Some parts of the software are configured via environment variables:
 
-* `OCRD_METS_CACHING`: If set to `true`, access to the METS file is cached, speeding in-memory search and modification.
 * `OCRD_PROFILE`: This variable configures the built-in CPU and memory profiling. If empty, no profiling is done. Otherwise expected to contain any of the following tokens:
   * `CPU`: Enable CPU profiling of processor runs
   * `RSS`: Enable RSS memory profiling
@@ -95,18 +89,46 @@ Some parts of the software are configured via environment variables:
 * `XDG_CONFIG_HOME`: Directory to look for `./ocrd/resources.yml` (i.e. `ocrd resmgr` user database) – defaults to `$HOME/.config`.
 * `XDG_DATA_HOME`: Directory to look for `./ocrd-resources/*` (i.e. `ocrd resmgr` data location) – defaults to `$HOME/.local/share`.
 
-* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of workspace files.
+* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of resources or workspace files.
 * `OCRD_DOWNLOAD_TIMEOUT`: Timeout in seconds for connecting or reading (comma-separated) when downloading.
 
+* `OCRD_MISSING_INPUT`: How to deal with missing input files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed with next page's input
+  * `ABORT`: throw `MissingInputFile` exception
+
+* `OCRD_MISSING_OUTPUT`: How to deal with missing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `COPY`: fall back to copying input PAGE to output fileGrp for page
+  * `ABORT`: re-throw whatever caused processing to fail
+
+* `OCRD_MAX_MISSING_OUTPUTS`: Maximal rate of skipped/fallback pages among all processed pages before aborting (decimal fraction, ignored if negative).
+
+* `OCRD_EXISTING_OUTPUT`: How to deal with already existing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `OVERWRITE`: force writing result to output fileGrp for page
+  * `ABORT`: re-throw `FileExistsError` exception
+
+
 * `OCRD_METS_CACHING`: Whether to enable in-memory storage of OcrdMets data structures for speedup during processing or workspace operations.
 
 * `OCRD_MAX_PROCESSOR_CACHE`: Maximum number of processor instances (for each set of parameters) to be kept in memory (including loaded models) for processing workers or processor servers.
 
+* `OCRD_MAX_PARALLEL_PAGES`: Maximum number of processor threads for page-parallel processing (within each Processor's selected page range, independent of the number of Processing Workers or Processor Servers). If set `>1`, then a METS Server must be used for METS synchronisation.
+
+* `OCRD_PROCESSING_PAGE_TIMEOUT`: Timeout in seconds for processing a single page. If set >0, when exceeded, the same as OCRD_MISSING_OUTPUT applies.
+
 * `OCRD_NETWORK_SERVER_ADDR_PROCESSING`: Default address of Processing Server to connect to (for `ocrd network client processing`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKFLOW`: Default address of Workflow Server to connect to (for `ocrd network client workflow`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKSPACE`: Default address of Workspace Server to connect to (for `ocrd network client workspace`).
 * `OCRD_NETWORK_RABBITMQ_CLIENT_CONNECT_ATTEMPTS`: Number of attempts for a worker to create its queue. Helpful if the rabbitmq-server needs time to be fully started.
 
+* `OCRD_NETWORK_CLIENT_POLLING_SLEEP`: How many seconds to sleep before trying `ocrd network client` again.
+* `OCRD_NETWORK_CLIENT_POLLING_TIMEOUT`: Timeout for a blocking `ocrd network client` (in seconds).
+
+* `OCRD_NETWORK_SOCKETS_ROOT_DIR`: The root directory where all mets server related socket files are created.
+* `OCRD_NETWORK_LOGS_ROOT_DIR`: The root directory where all ocrd_network related file logs are stored.
+
+
 
 ## Packages
 

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/README_bashlib.md

@@ -21,6 +21,9 @@ For example:
 * [`ocrd__log`](#ocrd__log)
 * [`ocrd__minversion`](#ocrd__minversion)
 * [`ocrd__dumpjson`](#ocrd__dumpjson)
+* [`ocrd__resolve_resource`](#ocrd__resolve_resource)
+* [`ocrd__show_resource`](#ocrd__show_resource)
+* [`ocrd__list_resources`](#ocrd__list_resources)
 * [`ocrd__usage`](#ocrd__usage)
 * [`ocrd__parse_argv`](#ocrd__parse_argv)
 <!-- END-MARKDOWN-TOC -->
@@ -56,6 +59,10 @@ export OCRD_TOOL_NAME=ocrd-foo-bar
 
 (Which you automatically get from [`ocrd__wrap`](#ocrd__wrap).)
 
+### `ocrd__resolve_resource`
+
+Output given resource file's path.
+
 ### `ocrd__show_resource`
 
 Output given resource file's content.
@@ -88,6 +95,7 @@ This will be filled by the parser along the following keys:
 - `profile`: whether `--profile` is enabled
 - `profile_file`: the argument of `--profile-file`
 - `log_level`: the argument of `--log-level`
+- `mets_server_url`: the argument of `--mets-server-url` argument
 - `mets_file`: absolute path of the `--mets` argument
 - `working_dir`: absolute path of the `--working-dir` argument or the parent of `mets_file`
 - `page_id`: the argument of `--page-id`
@@ -95,7 +103,7 @@ This will be filled by the parser along the following keys:
 - `output_file_grp`: the argument of `--output-file-grp`
 
 Moreover, there will be an associative array **`params`**
-with the fully expanded runtime values of the ocrd-tool.json parameters.
+with the fully validated and default-expanded runtime values of the `ocrd-tool.json` parameters.
 
 ### `ocrd__wrap`
 

ocrd-3.0.0b3/VERSION

@@ -0,0 +1 @@
+3.0.0b3

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/src/ocrd/cli/__init__.py

@@ -61,11 +61,11 @@ Variables:
 \b
 {config.describe('OCRD_DOWNLOAD_INPUT')}
 \b
-{config.describe('OCRD_MISSING_INPUT')}
+{config.describe('OCRD_MISSING_INPUT', wrap_text=False)}
 \b
-{config.describe('OCRD_MISSING_OUTPUT')}
+{config.describe('OCRD_MISSING_OUTPUT', wrap_text=False)}
 \b
-{config.describe('OCRD_EXISTING_OUTPUT')}
+{config.describe('OCRD_EXISTING_OUTPUT', wrap_text=False)}
 \b
 {config.describe('OCRD_METS_CACHING')}
 \b

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/src/ocrd/lib.bash

@@ -156,6 +156,7 @@ ocrd__parse_argv () {
     while [[ "${1:-}" = -* ]];do
         case "$1" in
             -l|--log-level) ocrd__argv[log_level]=$2 ; shift ;;
+            --log-filename) exec 2> "$2" ; shift ;;
             -h|--help|--usage) ocrd__usage; exit ;;
             -J|--dump-json) ocrd__dumpjson; exit ;;
             -D|--dump-module-dir) echo $(dirname "$OCRD_TOOL_JSON"); exit ;;

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/src/ocrd/mets_server.py

@@ -120,7 +120,7 @@ class ClientSideOcrdMets:
 
     def __init__(self, url, workspace_path: Optional[str] = None):
         self.protocol = "tcp" if url.startswith("http://") else "uds"
-        self.log = getLogger(f"ocrd.mets_client[{url}]")
+        self.log = getLogger(f"ocrd.models.ocrd_mets.client.{url}")
         self.url = url if self.protocol == "tcp" else f'http+unix://{url.replace("/", "%2F")}'
         self.ws_dir_path = workspace_path if workspace_path else None
 

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/src/ocrd/processor/__init__.py

@@ -3,6 +3,7 @@ from .base import (
     ResourceNotFoundError,
     NonUniqueInputFile,
     MissingInputFile,
+    generate_processor_help,
 )
 from .ocrd_page_result import (
     OcrdPageResult,
@@ -11,5 +12,4 @@ from .ocrd_page_result import (
 from .helpers import (
     run_cli,
     run_processor,
-    generate_processor_help
 )

{ocrd-3.0.0b1 → ocrd-3.0.0b3}/src/ocrd/processor/base.py

@@ -23,12 +23,16 @@ import tarfile
 import io
 import weakref
 from frozendict import frozendict
+from concurrent.futures import ThreadPoolExecutor, TimeoutError
+
+from click import wrap_text
 from deprecated import deprecated
 from requests import HTTPError
 
-from ocrd.workspace import Workspace
+from ..workspace import Workspace
+from ..mets_server import ClientSideOcrdMets
 from ocrd_models.ocrd_file import OcrdFileType
-from ocrd.processor.ocrd_page_result import OcrdPageResult
+from .ocrd_page_result import OcrdPageResult
 from ocrd_utils import (
     VERSION as OCRD_VERSION,
     MIMETYPE_PAGE,
@@ -58,7 +62,7 @@ from ocrd_modelfactory import page_from_file
 from ocrd_validators.ocrd_tool_validator import OcrdToolValidator
 
 # XXX imports must remain for backwards-compatibility
-from .helpers import run_cli, run_processor, generate_processor_help # pylint: disable=unused-import
+from .helpers import run_cli, run_processor # pylint: disable=unused-import
 
 
 class ResourceNotFoundError(FileNotFoundError):
@@ -118,7 +122,27 @@ class Processor():
     maximum number of cached instances (ignored if negative), to be applied on top of
     :py:data:`~ocrd_utils.config.OCRD_MAX_PROCESSOR_CACHE` (i.e. whatever is smaller).
 
-    (Override this if you know how many instances fit into memory at once.)
+    (Override this if you know how many instances fit into memory - GPU / CPU RAM - at once.)
+    """
+
+    max_workers : int = -1
+    """
+    maximum number of processor threads for page-parallel processing (ignored if negative),
+    to be applied on top of :py:data:`~ocrd_utils.config.OCRD_MAX_PARALLEL_PAGES` (i.e.
+    whatever is smaller).
+
+    (Override this if you know how many pages fit into processing units - GPU shaders / CPU cores
+    - at once, or if your class is not thread-safe.)
+    """
+
+    max_page_seconds : int = -1
+    """
+    maximum number of seconds may be spent processing a single page (ignored if negative),
+    to be applied on top of :py:data:`~ocrd_utils.config.OCRD_PROCESSING_PAGE_TIMEOUT`
+    (i.e. whatever is smaller).
+
+    (Override this if you know how costly this processor may be, irrespective of image size
+    or complexity of the page.)
     """
 
     @property
@@ -142,7 +166,11 @@ class Processor():
 
         (Override if ``ocrd-tool.json`` is not distributed with the Python package.)
         """
-        return resource_filename(self.__module__.split('.')[0], self.metadata_filename)
+        # XXX HACK
+        module_tokens = self.__module__.split('.')
+        if module_tokens[0] == 'src':
+            module_tokens.pop(0)
+        return resource_filename(module_tokens[0], self.metadata_filename)
 
     @cached_property
     def metadata_rawdict(self) -> dict:
@@ -273,12 +301,12 @@ class Processor():
         if ocrd_tool is not None:
             deprecation_warning("Passing 'ocrd_tool' as keyword argument to Processor is deprecated - "
                                 "use or override metadata/executable/ocrd-tool properties instead")
-            self._ocrd_tool = ocrd_tool
-            self._executable = ocrd_tool['executable']
+            self.ocrd_tool = ocrd_tool
+            self.executable = ocrd_tool['executable']
         if version is not None:
             deprecation_warning("Passing 'version' as keyword argument to Processor is deprecated - "
                                 "use or override metadata/version properties instead")
-            self._version = version
+            self.version = version
         if workspace is not None:
             deprecation_warning("Passing a workspace argument other than 'None' to Processor "
                                 "is deprecated - pass as argument to process_workspace instead")
@@ -422,13 +450,35 @@ class Processor():
             self.workspace = workspace
             self.verify()
             try:
-                # FIXME: add page parallelization by running multiprocessing.Pool (#322)
+                nr_succeeded = 0
+                nr_skipped = 0
+                nr_copied = 0
+
+                # set up multithreading
+                if self.max_workers <= 0:
+                    max_workers = max(0, config.OCRD_MAX_PARALLEL_PAGES)
+                else:
+                    max_workers = max(0, min(config.OCRD_MAX_PARALLEL_PAGES, self.max_workers))
+                if max_workers > 1:
+                    assert isinstance(workspace.mets, ClientSideOcrdMets), \
+                        "OCRD_MAX_PARALLEL_PAGES>1 requires also using --mets-server-url"
+                if self.max_page_seconds <= 0:
+                    max_seconds = max(0, config.OCRD_PROCESSING_PAGE_TIMEOUT)
+                else:
+                    max_seconds = max(0, min(config.OCRD_PROCESSING_PAGE_TIMEOUT, self.max_page_seconds))
+                executor = ThreadPoolExecutor(
+                    max_workers=max_workers or 1,
+                    thread_name_prefix=f"pagetask.{workspace.mets.unique_identifier}"
+                )
+                self._base_logger.debug("started executor %s with %d workers", str(executor), max_workers or 1)
+                tasks = {}
+
                 for input_file_tuple in self.zip_input_files(on_error='abort', require_first=False):
                     input_files : List[Optional[OcrdFileType]] = [None] * len(input_file_tuple)
                     page_id = next(input_file.pageId
                                    for input_file in input_file_tuple
                                    if input_file)
-                    self._base_logger.info(f"processing page {page_id}")
+                    self._base_logger.info(f"preparing page {page_id}")
                     for i, input_file in enumerate(input_file_tuple):
                         if input_file is None:
                             # file/page not found in this file grp
@@ -441,35 +491,56 @@ class Processor():
                         except (ValueError, FileNotFoundError, HTTPError) as e:
                             self._base_logger.error(repr(e))
                             self._base_logger.warning(f"failed downloading file {input_file} for page {page_id}")
+                    # process page
+                    tasks[executor.submit(self.process_page_file, *input_files)] = (page_id, input_files)
+                self._base_logger.debug("submitted %d processing tasks", len(tasks))
+
+                for task in tasks:
+                    # wait for results, handle errors
+                    page_id, input_files = tasks[task]
                     # FIXME: differentiate error cases in various ways:
                     # - ResourceNotFoundError → use ResourceManager to download (once), then retry
                     # - transient (I/O or OOM) error → maybe sleep, retry
                     # - persistent (data) error → skip / dummy / raise
                     try:
-                        self.process_page_file(*input_files)
-                    except Exception as err:
-                        # we have to be broad here, but want to exclude NotImplementedError
-                        if isinstance(err, NotImplementedError):
+                        self._base_logger.debug("waiting for output of task %s (page %s) max_seconds=%d", task, page_id, max_seconds)
+                        task.result(timeout=max_seconds or None)
+                        nr_succeeded += 1
+                    # exclude NotImplementedError, so we can try process() below
+                    except NotImplementedError:
+                        raise
+                    # handle input failures separately
+                    except FileExistsError as err:
+                        if config.OCRD_EXISTING_OUTPUT == 'ABORT':
                             raise err
-                        if isinstance(err, FileExistsError):
-                            if config.OCRD_EXISTING_OUTPUT == 'ABORT':
-                                raise err
-                            if config.OCRD_EXISTING_OUTPUT == 'SKIP':
-                                continue
-                            if config.OCRD_EXISTING_OUTPUT == 'OVERWRITE':
-                                # too late here, must not happen
-                                raise Exception(f"got {err} despite OCRD_EXISTING_OUTPUT==OVERWRITE")
-                        # FIXME: re-usable/actionable logging
-                        self._base_logger.exception(f"Failure on page {page_id}: {err}")
+                        if config.OCRD_EXISTING_OUTPUT == 'SKIP':
+                            continue
+                        if config.OCRD_EXISTING_OUTPUT == 'OVERWRITE':
+                            # too late here, must not happen
+                            raise Exception(f"got {err} despite OCRD_EXISTING_OUTPUT==OVERWRITE")
+                    # broad coverage of output failures (including TimeoutError)
+                    except (Exception, TimeoutError) as err:
+                        # FIXME: add re-usable/actionable logging
                         if config.OCRD_MISSING_OUTPUT == 'ABORT':
+                            self._base_logger.error(f"Failure on page {page_id}: {str(err) or err.__class__.__name__}")
                             raise err
+                        self._base_logger.exception(f"Failure on page {page_id}: {str(err) or err.__class__.__name__}")
                         if config.OCRD_MISSING_OUTPUT == 'SKIP':
+                            nr_skipped += 1
                             continue
                         if config.OCRD_MISSING_OUTPUT == 'COPY':
                             self._copy_page_file(input_files[0])
+                            nr_copied += 1
                         else:
                             desc = config.describe('OCRD_MISSING_OUTPUT', wrap_text=False, indent_text=False)
                             raise ValueError(f"unknown configuration value {config.OCRD_MISSING_OUTPUT} - {desc}")
+
+                if nr_skipped > 0 and nr_succeeded / nr_skipped < config.OCRD_MAX_MISSING_OUTPUTS:
+                    raise Exception(f"too many failures with skipped output ({nr_skipped})")
+                if nr_copied > 0 and nr_succeeded / nr_copied < config.OCRD_MAX_MISSING_OUTPUTS:
+                    raise Exception(f"too many failures with fallback output ({nr_skipped})")
+                executor.shutdown()
+
             except NotImplementedError:
                 # fall back to deprecated method
                 self.process()
@@ -493,13 +564,14 @@ class Processor():
         output_file_id = make_file_id(input_file, self.output_file_grp)
         input_pcgts.set_pcGtsId(output_file_id)
         self.add_metadata(input_pcgts)
-        self.workspace.add_file(file_id=output_file_id,
-                                file_grp=self.output_file_grp,
-                                page_id=input_file.pageId,
-                                local_filename=os.path.join(self.output_file_grp, output_file_id + '.xml'),
-                                mimetype=MIMETYPE_PAGE,
-                                content=to_xml(input_pcgts),
-                                force=config.OCRD_EXISTING_OUTPUT == 'OVERWRITE',
+        self.workspace.add_file(
+            file_id=output_file_id,
+            file_grp=self.output_file_grp,
+            page_id=input_file.pageId,
+            local_filename=os.path.join(self.output_file_grp, output_file_id + '.xml'),
+            mimetype=MIMETYPE_PAGE,
+            content=to_xml(input_pcgts),
+            force=config.OCRD_EXISTING_OUTPUT == 'OVERWRITE',
         )
 
     def process_page_file(self, *input_files : Optional[OcrdFileType]) -> None:
@@ -516,6 +588,7 @@ class Processor():
         input_pcgts : List[Optional[OcrdPage]] = [None] * len(input_files)
         assert isinstance(input_files[0], get_args(OcrdFileType))
         page_id = input_files[0].pageId
+        self._base_logger.info("processing page %s", page_id)
         for i, input_file in enumerate(input_files):
             assert isinstance(input_file, get_args(OcrdFileType))
             self._base_logger.debug(f"parsing file {input_file.ID} for page {page_id}")
@@ -532,6 +605,9 @@ class Processor():
             image_file_id = f'{output_file_id}_{image_result.file_id_suffix}'
             image_file_path = join(self.output_file_grp, f'{image_file_id}.png')
             if isinstance(image_result.alternative_image, PageType):
+                # special case: not an alternative image, but replacing the original image
+                # (this is needed by certain processors when the original's coordinate system
+                #  cannot or must not be kept)
                 image_result.alternative_image.set_imageFilename(image_file_path)
                 image_result.alternative_image.set_imageWidth(image_result.pil.width)
                 image_result.alternative_image.set_imageHeight(image_result.pil.height)
@@ -550,13 +626,14 @@ class Processor():
             )
         result.pcgts.set_pcGtsId(output_file_id)
         self.add_metadata(result.pcgts)
-        self.workspace.add_file(file_id=output_file_id,
-                                file_grp=self.output_file_grp,
-                                page_id=page_id,
-                                local_filename=os.path.join(self.output_file_grp, output_file_id + '.xml'),
-                                mimetype=MIMETYPE_PAGE,
-                                content=to_xml(result.pcgts),
-                                force=config.OCRD_EXISTING_OUTPUT == 'OVERWRITE',
+        self.workspace.add_file(
+            file_id=output_file_id,
+            file_grp=self.output_file_grp,
+            page_id=page_id,
+            local_filename=os.path.join(self.output_file_grp, output_file_id + '.xml'),
+            mimetype=MIMETYPE_PAGE,
+            content=to_xml(result.pcgts),
+            force=config.OCRD_EXISTING_OUTPUT == 'OVERWRITE',
         )
 
     def process_page_pcgts(self, *input_pcgts : Optional[OcrdPage], page_id : Optional[str] = None) -> OcrdPageResult:
@@ -838,3 +915,151 @@ class Processor():
             if ifiles[0] or not require_first:
                 ifts.append(tuple(ifiles))
         return ifts
+
+def generate_processor_help(ocrd_tool, processor_instance=None, subcommand=None):
+    """Generate a string describing the full CLI of this processor including params.
+
+    Args:
+         ocrd_tool (dict): this processor's ``tools`` section of the module's ``ocrd-tool.json``
+         processor_instance (object, optional): the processor implementation
+             (for adding any module/class/function docstrings)
+        subcommand (string): 'worker' or 'server'
+    """
+    doc_help = ''
+    if processor_instance:
+        module = inspect.getmodule(processor_instance)
+        if module and module.__doc__:
+            doc_help += '\n' + inspect.cleandoc(module.__doc__) + '\n'
+        if processor_instance.__doc__:
+            doc_help += '\n' + inspect.cleandoc(processor_instance.__doc__) + '\n'
+        # Try to find the most concrete docstring among the various methods that an implementation
+        # could overload, first serving.
+        # In doing so, compare with Processor to avoid a glitch in the way py>=3.5 inherits docstrings.
+        # (They are supposed to only repeat information inspect.getdoc, rather than inherit __doc__ itself.)
+        for method in ['process_page_pcgts', 'process_page_file', 'process_workspace', 'process']:
+            instance_method = getattr(processor_instance, method)
+            superclass_method = getattr(Processor, method)
+            if instance_method.__doc__ and instance_method.__doc__ != superclass_method.__doc__:
+                doc_help += '\n' + inspect.cleandoc(instance_method.__doc__) + '\n'
+                break
+        if doc_help:
+            doc_help = '\n\n' + wrap_text(doc_help, width=72,
+                                          initial_indent='  > ',
+                                          subsequent_indent='  > ',
+                                          preserve_paragraphs=True)
+    subcommands = '''\
+    worker      Start a processing worker rather than do local processing
+    server      Start a processor server rather than do local processing
+'''
+
+    processing_worker_options = '''\
+  --queue                         The RabbitMQ server address in format
+                                  "amqp://{user}:{pass}@{host}:{port}/{vhost}"
+                                  [amqp://admin:admin@localhost:5672]
+  --database                      The MongoDB server address in format
+                                  "mongodb://{host}:{port}"
+                                  [mongodb://localhost:27018]
+  --log-filename                  Filename to redirect STDOUT/STDERR to,
+                                  if specified.
+'''
+
+    processing_server_options = '''\
+  --address                       The Processor server address in format
+                                  "{host}:{port}"
+  --database                      The MongoDB server address in format
+                                  "mongodb://{host}:{port}"
+                                  [mongodb://localhost:27018]
+'''
+
+    processing_options = '''\
+  -m, --mets URL-PATH             URL or file path of METS to process [./mets.xml]
+  -w, --working-dir PATH          Working directory of local workspace [dirname(URL-PATH)]
+  -I, --input-file-grp USE        File group(s) used as input
+  -O, --output-file-grp USE       File group(s) used as output
+  -g, --page-id ID                Physical page ID(s) to process instead of full document []
+  --overwrite                     Remove existing output pages/images
+                                  (with "--page-id", remove only those).
+                                  Short-hand for OCRD_EXISTING_OUTPUT=OVERWRITE
+  --debug                         Abort on any errors with full stack trace.
+                                  Short-hand for OCRD_MISSING_OUTPUT=ABORT
+  --profile                       Enable profiling
+  --profile-file PROF-PATH        Write cProfile stats to PROF-PATH. Implies "--profile"
+  -p, --parameter JSON-PATH       Parameters, either verbatim JSON string
+                                  or JSON file path
+  -P, --param-override KEY VAL    Override a single JSON object key-value pair,
+                                  taking precedence over --parameter
+  -U, --mets-server-url URL       URL of a METS Server for parallel incremental access to METS
+                                  If URL starts with http:// start an HTTP server there,
+                                  otherwise URL is a path to an on-demand-created unix socket
+  -l, --log-level [OFF|ERROR|WARN|INFO|DEBUG|TRACE]
+                                  Override log level globally [INFO]
+  --log-filename LOG-PATH         File to redirect stderr logging to (overriding ocrd_logging.conf).
+'''
+
+    information_options = '''\
+  -C, --show-resource RESNAME     Dump the content of processor resource RESNAME
+  -L, --list-resources            List names of processor resources
+  -J, --dump-json                 Dump tool description as JSON
+  -D, --dump-module-dir           Show the 'module' resource location path for this processor
+  -h, --help                      Show this message
+  -V, --version                   Show version
+'''
+
+    parameter_help = ''
+    if 'parameters' not in ocrd_tool or not ocrd_tool['parameters']:
+        parameter_help = '  NONE\n'
+    else:
+        def wrap(s):
+            return wrap_text(s, initial_indent=' '*3,
+                             subsequent_indent=' '*4,
+                             width=72, preserve_paragraphs=True)
+        for param_name, param in ocrd_tool['parameters'].items():
+            parameter_help += wrap('"%s" [%s%s]' % (
+                param_name,
+                param['type'],
+                ' - REQUIRED' if 'required' in param and param['required'] else
+                ' - %s' % json.dumps(param['default']) if 'default' in param else ''))
+            parameter_help += '\n ' + wrap(param['description'])
+            if 'enum' in param:
+                parameter_help += '\n ' + wrap('Possible values: %s' % json.dumps(param['enum']))
+            parameter_help += "\n"
+
+    if not subcommand:
+        return f'''\
+Usage: {ocrd_tool['executable']} [worker|server] [OPTIONS]
+
+  {ocrd_tool['description']}{doc_help}
+
+Subcommands:
+{subcommands}
+Options for processing:
+{processing_options}
+Options for information:
+{information_options}
+Parameters:
+{parameter_help}
+'''
+    elif subcommand == 'worker':
+        return f'''\
+Usage: {ocrd_tool['executable']} worker [OPTIONS]
+
+  Run {ocrd_tool['executable']} as a processing worker.
+
+  {ocrd_tool['description']}{doc_help}
+
+Options:
+{processing_worker_options}
+'''
+    elif subcommand == 'server':
+        return f'''\
+Usage: {ocrd_tool['executable']} server [OPTIONS]
+
+  Run {ocrd_tool['executable']} as a processor sever.
+
+  {ocrd_tool['description']}{doc_help}
+
+Options:
+{processing_server_options}
+'''
+    else:
+        pass