dir-content-diff 1.13.0__tar.gz → 1.14.0__tar.gz

{dir_content_diff-1.13.0 → dir_content_diff-1.14.0}/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## [1.14.0](https://github.com/BlueBrain/dir-content-diff/compare/1.13.0..1.14.0)
+
+> 13 September 2025
+
+### New Features
+
+- Add parallel iteration over compared files (Adrien Berchet - [#89](https://github.com/BlueBrain/dir-content-diff/pull/89))
+
+### Documentation Changes
+
+- Improve README to describe parallel execution (Adrien Berchet - [#91](https://github.com/BlueBrain/dir-content-diff/pull/91))
+
+### Refactoring and Updates
+
+- Reorganize the code (Adrien Berchet - [#90](https://github.com/BlueBrain/dir-content-diff/pull/90))
+
 ## [1.13.0](https://github.com/BlueBrain/dir-content-diff/compare/1.12.1..1.13.0)
 
 > 12 September 2025

{dir_content_diff-1.13.0/dir_content_diff.egg-info → dir_content_diff-1.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dir-content-diff
-Version: 1.13.0
+Version: 1.14.0
 Summary: Simple tool to compare directory contents.
 Author: Blue Brain Project, EPFL
 License: Apache License 2.0
@@ -76,7 +76,7 @@ Dynamic: provides-extra
 This project provides simple tools to compare the content of a directory against a reference
 directory.
 
-This is useful to check the results of a process that generate several files, like a luigi
+This is useful to check the results of a process that generates several files, like a luigi
 workflow for example.
 
 
@@ -102,6 +102,8 @@ the files, so it is not able to report which values are different.
 
 ### Compare two directories
 
+#### Basic Directory Comparison
+
 Let's compare two directories with the following structures:
 
 ```bash
@@ -142,6 +144,8 @@ was detected. As mentioned previously, this is because `dir-content-diff` is onl
 in the compared directory that are also present in the reference directory, so the file
 `sub_file_3.b` is just ignored in this case.
 
+#### Using Custom Comparators
+
 If ``reference_dir/file_1.c`` is the following JSON-like file:
 
 ```json
@@ -183,6 +187,8 @@ The previous code will output the following dictionary:
 }
 ```
 
+#### Assertion-based Comparison
+
 It is also possible to check whether the two directories are equal or not with the following code:
 
 ```python
@@ -201,7 +207,9 @@ Changed the value of '[a]' from 1 to 2.
 Changed the value of '[b][0]' from 1 to 10.
 ```
 
-Finally, the comparators have parameters that can be passed either to be used for all files of a
+#### Advanced Configuration Options
+
+The comparators have parameters that can be passed either to be used for all files of a
 given extension or only for a specific file:
 
 ```python
@@ -229,6 +237,8 @@ dir_content_diff.assert_equal_trees(
 
 Each comparator has different arguments that are detailed in the documentation.
 
+##### File-specific Comparators
+
 It's also possible to specify a arbitrary comparator for a specific file:
 
 ```python
@@ -240,6 +250,8 @@ specific_args = {
 }
 ```
 
+##### Pattern-based Configuration
+
 Another possibility is to use regular expressions to associate specific arguments to
 a set of files:
 
@@ -252,7 +264,9 @@ specific_args = {
 }
 ```
 
-And last but not least, it's possible to filter files from the reference directory (for example
+##### File Filtering
+
+Last but not least, it's possible to filter files from the reference directory (for example
 because the reference directory contains temporary files that should not be compared). For
 example, the following code will ignore all files whose name does not start with `file_` and does
 not ends with `_tmp.yaml`:
@@ -269,6 +283,73 @@ dir_content_diff.compare_trees(
 ```
 
 
+### Parallel Execution
+
+By default, `dir-content-diff` runs file comparisons sequentially. However, for improved performance when comparing large numbers of files, parallel execution is available using either thread-based or process-based concurrency.
+
+#### Configuration Options
+
+Parallel execution can be configured using the following parameters:
+
+- **`executor_type`**: Controls the type of parallel execution:
+  - `"sequential"` (default): No parallel execution, files are compared one by one
+  - `"thread"`: Uses `ThreadPoolExecutor` (recommended for I/O-bound tasks)
+  - `"process"`: Uses `ProcessPoolExecutor` (recommended for CPU-intensive comparisons)
+
+- **`max_workers`**: Maximum number of worker threads/processes. If `None` (default), it defaults to `min(32, (os.cpu_count() or 1) + 4)`.
+
+#### Usage Examples
+
+Enable thread-based parallel execution:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="thread",
+    max_workers=8
+)
+```
+
+Enable process-based parallel execution with automatic worker count:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="process"
+)
+```
+
+Using a configuration object:
+
+```python
+import dir_content_diff
+
+config = dir_content_diff.ComparisonConfig(
+    executor_type="thread",
+    max_workers=4
+)
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    config=config
+)
+```
+
+#### Performance Considerations
+
+- **Thread-based execution** (`executor_type="thread"`) is generally recommended for most use cases as file comparisons are typically I/O-bound operations
+- **Process-based execution** (`executor_type="process"`) may be beneficial when using computationally intensive comparators or when dealing with very large files
+- Parallel execution is automatically disabled for single file comparisons and falls back to sequential execution when only one file needs to be compared
+- The optimal number of workers depends on your system's capabilities and the nature of your files; too many workers may actually decrease performance due to overhead
+
+
 ### Export formatted data
 
 Some comparators have to format the data before comparing them. For example, if one wants to

{dir_content_diff-1.13.0 → dir_content_diff-1.14.0}/README.md

@@ -10,7 +10,7 @@
 This project provides simple tools to compare the content of a directory against a reference
 directory.
 
-This is useful to check the results of a process that generate several files, like a luigi
+This is useful to check the results of a process that generates several files, like a luigi
 workflow for example.
 
 
@@ -36,6 +36,8 @@ the files, so it is not able to report which values are different.
 
 ### Compare two directories
 
+#### Basic Directory Comparison
+
 Let's compare two directories with the following structures:
 
 ```bash
@@ -76,6 +78,8 @@ was detected. As mentioned previously, this is because `dir-content-diff` is onl
 in the compared directory that are also present in the reference directory, so the file
 `sub_file_3.b` is just ignored in this case.
 
+#### Using Custom Comparators
+
 If ``reference_dir/file_1.c`` is the following JSON-like file:
 
 ```json
@@ -117,6 +121,8 @@ The previous code will output the following dictionary:
 }
 ```
 
+#### Assertion-based Comparison
+
 It is also possible to check whether the two directories are equal or not with the following code:
 
 ```python
@@ -135,7 +141,9 @@ Changed the value of '[a]' from 1 to 2.
 Changed the value of '[b][0]' from 1 to 10.
 ```
 
-Finally, the comparators have parameters that can be passed either to be used for all files of a
+#### Advanced Configuration Options
+
+The comparators have parameters that can be passed either to be used for all files of a
 given extension or only for a specific file:
 
 ```python
@@ -163,6 +171,8 @@ dir_content_diff.assert_equal_trees(
 
 Each comparator has different arguments that are detailed in the documentation.
 
+##### File-specific Comparators
+
 It's also possible to specify a arbitrary comparator for a specific file:
 
 ```python
@@ -174,6 +184,8 @@ specific_args = {
 }
 ```
 
+##### Pattern-based Configuration
+
 Another possibility is to use regular expressions to associate specific arguments to
 a set of files:
 
@@ -186,7 +198,9 @@ specific_args = {
 }
 ```
 
-And last but not least, it's possible to filter files from the reference directory (for example
+##### File Filtering
+
+Last but not least, it's possible to filter files from the reference directory (for example
 because the reference directory contains temporary files that should not be compared). For
 example, the following code will ignore all files whose name does not start with `file_` and does
 not ends with `_tmp.yaml`:
@@ -203,6 +217,73 @@ dir_content_diff.compare_trees(
 ```
 
 
+### Parallel Execution
+
+By default, `dir-content-diff` runs file comparisons sequentially. However, for improved performance when comparing large numbers of files, parallel execution is available using either thread-based or process-based concurrency.
+
+#### Configuration Options
+
+Parallel execution can be configured using the following parameters:
+
+- **`executor_type`**: Controls the type of parallel execution:
+  - `"sequential"` (default): No parallel execution, files are compared one by one
+  - `"thread"`: Uses `ThreadPoolExecutor` (recommended for I/O-bound tasks)
+  - `"process"`: Uses `ProcessPoolExecutor` (recommended for CPU-intensive comparisons)
+
+- **`max_workers`**: Maximum number of worker threads/processes. If `None` (default), it defaults to `min(32, (os.cpu_count() or 1) + 4)`.
+
+#### Usage Examples
+
+Enable thread-based parallel execution:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="thread",
+    max_workers=8
+)
+```
+
+Enable process-based parallel execution with automatic worker count:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="process"
+)
+```
+
+Using a configuration object:
+
+```python
+import dir_content_diff
+
+config = dir_content_diff.ComparisonConfig(
+    executor_type="thread",
+    max_workers=4
+)
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    config=config
+)
+```
+
+#### Performance Considerations
+
+- **Thread-based execution** (`executor_type="thread"`) is generally recommended for most use cases as file comparisons are typically I/O-bound operations
+- **Process-based execution** (`executor_type="process"`) may be beneficial when using computationally intensive comparators or when dealing with very large files
+- Parallel execution is automatically disabled for single file comparisons and falls back to sequential execution when only one file needs to be compared
+- The optimal number of workers depends on your system's capabilities and the nature of your files; too many workers may actually decrease performance due to overhead
+
+
 ### Export formatted data
 
 Some comparators have to format the data before comparing them. For example, if one wants to

dir_content_diff-1.14.0/dir_content_diff/__init__.py

@@ -0,0 +1,53 @@
+"""dir-content-diff package.
+
+Simple tool to compare directory contents.
+"""
+
+# LICENSE HEADER MANAGED BY add-license-header
+# Copyright (c) 2023-2025 Blue Brain Project, EPFL.
+#
+# This file is part of dir-content-diff.
+# See https://github.com/BlueBrain/dir-content-diff for further info.
+#
+# SPDX-License-Identifier: Apache-2.0
+# LICENSE HEADER MANAGED BY add-license-header
+
+import importlib.metadata
+
+# Import base comparators
+from dir_content_diff.base_comparators import BaseComparator
+from dir_content_diff.base_comparators import DefaultComparator
+
+# Import core comparison functionality
+from dir_content_diff.config import ComparisonConfig
+from dir_content_diff.core import ComparatorType
+from dir_content_diff.core import assert_equal_trees
+from dir_content_diff.core import compare_files
+from dir_content_diff.core import compare_trees
+from dir_content_diff.core import export_formatted_file
+from dir_content_diff.core import pick_comparator
+
+# Import comparator registry functionality
+from dir_content_diff.registry import get_comparators
+from dir_content_diff.registry import register_comparator
+from dir_content_diff.registry import reset_comparators
+from dir_content_diff.registry import unregister_comparator
+
+__version__ = importlib.metadata.version("dir-content-diff")
+
+# For backward compatibility, ensure all public functions are available
+__all__ = [
+    "ComparisonConfig",
+    "ComparatorType",
+    "assert_equal_trees",
+    "compare_files",
+    "compare_trees",
+    "export_formatted_file",
+    "pick_comparator",
+    "get_comparators",
+    "register_comparator",
+    "reset_comparators",
+    "unregister_comparator",
+    "BaseComparator",
+    "DefaultComparator",
+]

{dir_content_diff-1.13.0 → dir_content_diff-1.14.0}/dir_content_diff/base_comparators.py

@@ -513,7 +513,7 @@ class XmlComparator(DictComparator):
             elif str(text).lower() == "false":
                 res = False
             else:
-                raise ValueError("Bool attributes expect 'true' or 'false'.")
+                raise ValueError("Boolean attributes expect 'true' or 'false'.")
         elif value_type == "list":
             res = []
         elif value_type == "dict":

{dir_content_diff-1.13.0 → dir_content_diff-1.14.0}/dir_content_diff/cli/__init__.py

@@ -13,15 +13,17 @@ import json
 import logging
 import sys
 from pathlib import Path
+from typing import Optional
+from typing import Union
 
 import click
 from yaml import safe_load
 
-from dir_content_diff import _DEFAULT_EXPORT_SUFFIX
 from dir_content_diff import compare_files
 from dir_content_diff import compare_trees
 from dir_content_diff import export_formatted_file
 from dir_content_diff import pick_comparator
+from dir_content_diff.core import _DEFAULT_EXPORT_SUFFIX
 from dir_content_diff.util import LOGGER
 
 
@@ -102,6 +104,19 @@ def load_config(ctx, param, value):  # pylint: disable=unused-argument
     default=False,
     help="Sort the differences by file name.",
 )
+@click.option(
+    "--executor-type",
+    type=click.Choice(["sequential", "thread", "process"]),
+    default="sequential",
+    help="Type of executor for execution. 'sequential' for single-threaded, "
+    "'thread' for I/O-bound tasks, 'process' for CPU-bound tasks.",
+)
+@click.option(
+    "--max-workers",
+    type=int,
+    help="Maximum number of worker threads/processes for parallel execution. "
+    "Only used with 'thread' or 'process' executor types.",
+)
 @click.version_option()
 @click.pass_context
 def main(ctx, *args, **kwargs):
@@ -120,16 +135,31 @@ def main(ctx, *args, **kwargs):
 
     ref = Path(kwargs.pop("reference_input"))
     comp = Path(kwargs.pop("compared_input"))
+
+    # Extract execution options
+    executor_type = kwargs.pop("executor_type", "sequential")
+    max_workers = kwargs.pop("max_workers", None)
+
     input_diff(
         ref,
         comp,
         ctx.config,
         kwargs.pop("export_formatted_files", False),
         kwargs.pop("sort_diffs", False),
+        executor_type,
+        max_workers,
     )
 
 
-def input_diff(ref, comp, config, export_formatted_files=False, sort_diffs=False):
+def input_diff(
+    ref: Union[str, Path],
+    comp: Union[str, Path],
+    config,
+    export_formatted_files: Union[bool, str] = False,
+    sort_diffs: bool = False,
+    executor_type: str = "sequential",
+    max_workers: Optional[int] = None,
+):
     """Compute and display differences from given inputs."""
     ref = Path(ref)
     comp = Path(comp)
@@ -145,6 +175,8 @@ def input_diff(ref, comp, config, export_formatted_files=False, sort_diffs=False
             comp,
             specific_args=config,
             export_formatted_files=export_formatted_files,
+            executor_type=executor_type,
+            max_workers=max_workers,
         )
     else:
         comparator_name = config.pop("comparator", None)

dir_content_diff-1.14.0/dir_content_diff/config.py

@@ -0,0 +1,225 @@
+"""Configuration classes for directory content comparison.
+
+This module contains the configuration classes and validation functions.
+"""
+
+# LICENSE HEADER MANAGED BY add-license-header
+# Copyright (c) 2023-2025 Blue Brain Project, EPFL.
+#
+# This file is part of dir-content-diff.
+# See https://github.com/BlueBrain/dir-content-diff for further info.
+#
+# SPDX-License-Identifier: Apache-2.0
+# LICENSE HEADER MANAGED BY add-license-header
+
+import re
+from collections.abc import Callable
+from typing import Any
+from typing import Dict
+from typing import Iterable
+from typing import Literal
+from typing import Optional
+from typing import Pattern
+from typing import Tuple
+from typing import Union
+
+import attrs
+
+from dir_content_diff.base_comparators import BaseComparator
+from dir_content_diff.registry import get_comparators
+
+# Type alias for comparators
+ComparatorType = Union[BaseComparator, Callable]
+
+
+def _convert_iterable_to_tuple(
+    x: Optional[Iterable[str]],
+) -> Optional[Tuple[str, ...]]:
+    """Convert an iterable to a tuple, or return None."""
+    if x is None:
+        return None
+    return tuple(x)
+
+
+def _validate_specific_args(instance, attribute, value):  # pylint: disable=unused-argument
+    """Validate specific_args structure."""
+    for file_path, args in value.items():
+        if not isinstance(args, dict):
+            raise ValueError(f"specific_args['{file_path}'] must be a dictionary")
+        # Note: regex patterns in specific_args will be validated during compilation
+        # in __attrs_post_init__, so no need to validate them here
+
+
+def _validate_export_formatted_files(instance, attribute, value):  # pylint: disable=unused-argument
+    """Validate export_formatted_files is either bool or non-empty string."""
+    if isinstance(value, str) and len(value.strip()) == 0:
+        raise ValueError(
+            "export_formatted_files must be a non-empty string when provided as string"
+        )
+
+
+def _validate_comparators(instance, attribute, value):  # pylint: disable=unused-argument
+    """Validate comparators are either BaseComparator instances or callable."""
+    for ext, comparator in value.items():
+        if not (isinstance(comparator, BaseComparator) or callable(comparator)):
+            raise ValueError(
+                f"Comparator for extension '{ext}' must be a BaseComparator instance "
+                "or callable"
+            )
+
+
+@attrs.frozen
+class ComparisonConfig:
+    """Configuration class to store comparison settings.
+
+    Attributes:
+        include_patterns: A list of regular expression patterns. If the relative path of a
+            file does not match any of these patterns, it is ignored during the comparison. Note
+            that this means that any specific arguments for that file will also be ignored.
+        exclude_patterns: A list of regular expression patterns. If the relative path of a
+            file matches any of these patterns, it is ignored during the comparison. Note that
+            this means that any specific arguments for that file will also be ignored.
+        comparators: A ``dict`` to override the registered comparators.
+        specific_args: A ``dict`` with the args/kwargs that should be given to the
+            comparator for a given file. This ``dict`` should be like the following:
+
+            .. code-block:: Python
+
+                {
+                    <relative_file_path>: {
+                        comparator: ComparatorInstance,
+                        args: [arg1, arg2, ...],
+                        kwargs: {
+                            kwarg_name_1: kwarg_value_1,
+                            kwarg_name_2: kwarg_value_2,
+                        }
+                    },
+                    <another_file_path>: {...},
+                    <a name for this category>: {
+                        "patterns": ["regex1", "regex2", ...],
+                        ... (other arguments)
+                    }
+                }
+
+            If the "patterns" entry is present, then the name is not considered and is only used as
+            a helper for the user. When a "patterns" entry is detected, the other arguments are
+            applied to all files whose relative name matches one of the given regular expression
+            patterns. If a file could match multiple patterns of different groups, only the first
+            one is considered.
+
+            Note that all entries in this ``dict`` are optional.
+        return_raw_diffs: If set to ``True``, only the raw differences are returned instead
+            of a formatted report.
+        export_formatted_files: If set to ``True`` or a not empty string, create a
+            new directory with formatted compared data files. If a string is passed, this string is
+            used as suffix for the new directory. If `True` is passed, the suffix is
+            ``_FORMATTED``.
+        max_workers: Maximum number of worker threads/processes for parallel execution. If None,
+            defaults to min(32, (os.cpu_count() or 1) + 4) as per executor default.
+        executor_type: Type of executor to use for parallel execution. 'thread' uses
+            ThreadPoolExecutor (better for I/O-bound tasks), 'process' uses ProcessPoolExecutor
+            (better for CPU-bound tasks), 'sequential' disables parallel execution.
+    """
+
+    include_patterns: Optional[Iterable[str]] = attrs.field(
+        default=None, converter=_convert_iterable_to_tuple
+    )
+    exclude_patterns: Optional[Iterable[str]] = attrs.field(
+        default=None, converter=_convert_iterable_to_tuple
+    )
+    comparators: Optional[Dict[Optional[str], ComparatorType]] = attrs.field(
+        default=None, validator=attrs.validators.optional(_validate_comparators)
+    )
+    specific_args: Optional[Dict[str, Dict[str, Any]]] = attrs.field(
+        default=None, validator=attrs.validators.optional(_validate_specific_args)
+    )
+    return_raw_diffs: bool = attrs.field(default=False)
+    export_formatted_files: Union[bool, str] = attrs.field(
+        default=False, validator=_validate_export_formatted_files
+    )
+    executor_type: Literal["sequential", "thread", "process"] = attrs.field(
+        default="sequential"
+    )
+    max_workers: Optional[int] = attrs.field(default=None)
+
+    # Compiled patterns - computed once, no caching complexity needed
+    compiled_include_patterns: Tuple[Pattern[str], ...] = attrs.field(init=False)
+    compiled_exclude_patterns: Tuple[Pattern[str], ...] = attrs.field(init=False)
+    pattern_specific_args: Dict[Pattern[str], Dict[str, Any]] = attrs.field(
+        init=False, repr=False
+    )
+
+    def __attrs_post_init__(self):
+        """Initialize computed fields after attrs initialization."""
+        # Validate and compile patterns - with frozen, we compile once and store directly
+        try:
+            compiled_include = self._compile_patterns(self.include_patterns)
+            object.__setattr__(self, "compiled_include_patterns", compiled_include)
+        except ValueError as e:
+            raise ValueError(f"Error in include_patterns: {e}") from e
+
+        try:
+            compiled_exclude = self._compile_patterns(self.exclude_patterns)
+            object.__setattr__(self, "compiled_exclude_patterns", compiled_exclude)
+        except ValueError as e:
+            raise ValueError(f"Error in exclude_patterns: {e}") from e
+
+        # Setup specific args and pattern specific args
+        if self.specific_args is None:
+            # Use object.__setattr__ to modify the field even if it's frozen
+            object.__setattr__(self, "specific_args", {})
+
+        # Setup pattern specific args
+        pattern_specific_args = {}
+        if self.specific_args:  # Check if it's not None
+            for file_path, v in self.specific_args.items():
+                if "patterns" in v:
+                    patterns = v.pop("patterns", [])
+                    for pattern in patterns:
+                        try:
+                            compiled_pattern = self._compile_pattern(pattern)
+                            pattern_specific_args[compiled_pattern] = v
+                        except ValueError as e:
+                            raise ValueError(
+                                f"Error in specific_args['{file_path}']['patterns']: {e}"
+                            ) from e
+
+        object.__setattr__(self, "pattern_specific_args", pattern_specific_args)
+
+        # Setup comparators
+        if self.comparators is None:
+            object.__setattr__(self, "comparators", get_comparators())
+
+    def _compile_pattern(self, pattern: str) -> Pattern[str]:
+        """Compile a regex pattern."""
+        try:
+            return re.compile(pattern)
+        except re.error as e:
+            raise ValueError(f"Invalid regex pattern: '{pattern}'") from e
+
+    def _compile_patterns(
+        self, patterns: Optional[Iterable[str]]
+    ) -> Tuple[Pattern[str], ...]:
+        """Compile regex patterns from any iterable to tuple."""
+        if patterns is None:
+            return ()
+        return tuple(self._compile_pattern(pattern) for pattern in patterns)
+
+    # Note: compiled_include_patterns, compiled_exclude_patterns, and pattern_specific_args
+    # are now direct attributes set in __attrs_post_init__, no properties needed!
+
+    def should_ignore_file(self, relative_path: str) -> bool:
+        """Check if a file should be ignored."""
+        # Check inclusion patterns first
+        if self.compiled_include_patterns:
+            included = any(
+                pattern.match(relative_path)
+                for pattern in self.compiled_include_patterns
+            )
+            if not included:
+                return True
+
+        # Check exclusion patterns
+        return any(
+            pattern.match(relative_path) for pattern in self.compiled_exclude_patterns
+        )