dir-content-diff 1.12.1__tar.gz → 1.14.0__tar.gz

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/.github/workflows/commitlint.yml

@@ -11,8 +11,8 @@ jobs:
     env:
       PR_TITLE: ${{ github.event.pull_request.title }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v5
         with:
           node-version: '20'
       - run: npm install -g --force commitlint @commitlint/cli commitlint-plugin-cleanfeet

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/.github/workflows/publish-sdist.yml

@@ -10,9 +10,9 @@ jobs:
     name: Build and publish on PyPI
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
       - name: Set up Python 3.10
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: "3.10"
       - name: Build a wheel and a source tarball

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/.github/workflows/run-tox.yml

@@ -24,9 +24,9 @@ jobs:
             min_versions: "min_versions"
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v6
       with:
         python-version: ${{ matrix.python-version }}
     - name: Get current month

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/.gitignore

@@ -30,6 +30,7 @@ test_venv
 docs/build
 docs/source/generated
 docs/source/examples
+docs/source/README_processed.md
 
 # reports
 reports

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## [1.14.0](https://github.com/BlueBrain/dir-content-diff/compare/1.13.0..1.14.0)
+
+> 13 September 2025
+
+### New Features
+
+- Add parallel iteration over compared files (Adrien Berchet - [#89](https://github.com/BlueBrain/dir-content-diff/pull/89))
+
+### Documentation Changes
+
+- Improve README to describe parallel execution (Adrien Berchet - [#91](https://github.com/BlueBrain/dir-content-diff/pull/91))
+
+### Refactoring and Updates
+
+- Reorganize the code (Adrien Berchet - [#90](https://github.com/BlueBrain/dir-content-diff/pull/90))
+
+## [1.13.0](https://github.com/BlueBrain/dir-content-diff/compare/1.12.1..1.13.0)
+
+> 12 September 2025
+
+### New Features
+
+- Add patterns to ignore files in the reference directory (Adrien Berchet - [#86](https://github.com/BlueBrain/dir-content-diff/pull/86))
+
+### Documentation Changes
+
+- Fix format in Sphinx (Adrien Berchet - [#85](https://github.com/BlueBrain/dir-content-diff/pull/85))
+- Clarify how reference and compared directories are processed (Adrien Berchet - [#83](https://github.com/BlueBrain/dir-content-diff/pull/83))
+
+### Performance Improvements
+
+- Build comparator registry only once before iterating over all files (Adrien Berchet - [#87](https://github.com/BlueBrain/dir-content-diff/pull/87))
+
 ## [1.12.1](https://github.com/BlueBrain/dir-content-diff/compare/1.12.0..1.12.1)
 
 > 3 July 2025

{dir_content_diff-1.12.1/dir_content_diff.egg-info → dir_content_diff-1.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dir-content-diff
-Version: 1.12.1
+Version: 1.14.0
 Summary: Simple tool to compare directory contents.
 Author: Blue Brain Project, EPFL
 License: Apache License 2.0
@@ -21,6 +21,7 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 License-File: AUTHORS.md
+Requires-Dist: attrs>=21.3.0
 Requires-Dist: click>=8
 Requires-Dist: dictdiffer>=0.8
 Requires-Dist: dicttoxml>=1.7.12
@@ -75,7 +76,7 @@ Dynamic: provides-extra
 This project provides simple tools to compare the content of a directory against a reference
 directory.
 
-This is useful to check the results of a process that generate several files, like a luigi
+This is useful to check the results of a process that generates several files, like a luigi
 workflow for example.
 
 
@@ -101,7 +102,9 @@ the files, so it is not able to report which values are different.
 
 ### Compare two directories
 
-If one wants to compare two directories with the following structures:
+#### Basic Directory Comparison
+
+Let's compare two directories with the following structures:
 
 ```bash
 └── reference_dir
@@ -120,6 +123,9 @@ If one wants to compare two directories with the following structures:
     └── file_1.c
 ```
 
+The `reference` directory contains all the files that should be checked in the `compared` directory,
+which means that extraneous files in the `compared` directory are just ignored.
+
 These two directories can be compared with the following code:
 
 ```python
@@ -128,7 +134,17 @@ import dir_content_diff
 dir_content_diff.compare_trees("reference_dir", "compared_dir")
 ```
 
-This code will return an empty dictionary because no difference was detected.
+> [!WARNING]
+> The order of the parameters is important: the first path is considered as the `reference`
+> directory while the second one is the `compared` directory. Inverting the parameters may return
+> a different result (in this example it would return that the file `sub_file_3.b` is missing).
+
+If all the files are identical, this code will return an empty dictionary because no difference
+was detected. As mentioned previously, this is because `dir-content-diff` is only looking for files
+in the compared directory that are also present in the reference directory, so the file
+`sub_file_3.b` is just ignored in this case.
+
+#### Using Custom Comparators
 
 If ``reference_dir/file_1.c`` is the following JSON-like file:
 
@@ -171,6 +187,8 @@ The previous code will output the following dictionary:
 }
 ```
 
+#### Assertion-based Comparison
+
 It is also possible to check whether the two directories are equal or not with the following code:
 
 ```python
@@ -189,7 +207,9 @@ Changed the value of '[a]' from 1 to 2.
 Changed the value of '[b][0]' from 1 to 10.
 ```
 
-Finally, the comparators have parameters that can be passed either to be used for all files of a
+#### Advanced Configuration Options
+
+The comparators have parameters that can be passed either to be used for all files of a
 given extension or only for a specific file:
 
 ```python
@@ -217,6 +237,8 @@ dir_content_diff.assert_equal_trees(
 
 Each comparator has different arguments that are detailed in the documentation.
 
+##### File-specific Comparators
+
 It's also possible to specify a arbitrary comparator for a specific file:
 
 ```python
@@ -228,7 +250,9 @@ specific_args = {
 }
 ```
 
-And last but not least, it's possible to use regular expressions to associate specific arguments to
+##### Pattern-based Configuration
+
+Another possibility is to use regular expressions to associate specific arguments to
 a set of files:
 
 ```python
@@ -240,6 +264,91 @@ specific_args = {
 }
 ```
 
+##### File Filtering
+
+Last but not least, it's possible to filter files from the reference directory (for example
+because the reference directory contains temporary files that should not be compared). For
+example, the following code will ignore all files whose name does not start with `file_` and does
+not ends with `_tmp.yaml`:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    include_patterns=[r"file_.*"],
+    exclude_patterns=[r".*_tmp\.yaml"],
+)
+```
+
+
+### Parallel Execution
+
+By default, `dir-content-diff` runs file comparisons sequentially. However, for improved performance when comparing large numbers of files, parallel execution is available using either thread-based or process-based concurrency.
+
+#### Configuration Options
+
+Parallel execution can be configured using the following parameters:
+
+- **`executor_type`**: Controls the type of parallel execution:
+  - `"sequential"` (default): No parallel execution, files are compared one by one
+  - `"thread"`: Uses `ThreadPoolExecutor` (recommended for I/O-bound tasks)
+  - `"process"`: Uses `ProcessPoolExecutor` (recommended for CPU-intensive comparisons)
+
+- **`max_workers`**: Maximum number of worker threads/processes. If `None` (default), it defaults to `min(32, (os.cpu_count() or 1) + 4)`.
+
+#### Usage Examples
+
+Enable thread-based parallel execution:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="thread",
+    max_workers=8
+)
+```
+
+Enable process-based parallel execution with automatic worker count:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="process"
+)
+```
+
+Using a configuration object:
+
+```python
+import dir_content_diff
+
+config = dir_content_diff.ComparisonConfig(
+    executor_type="thread",
+    max_workers=4
+)
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    config=config
+)
+```
+
+#### Performance Considerations
+
+- **Thread-based execution** (`executor_type="thread"`) is generally recommended for most use cases as file comparisons are typically I/O-bound operations
+- **Process-based execution** (`executor_type="process"`) may be beneficial when using computationally intensive comparators or when dealing with very large files
+- Parallel execution is automatically disabled for single file comparisons and falls back to sequential execution when only one file needs to be compared
+- The optimal number of workers depends on your system's capabilities and the nature of your files; too many workers may actually decrease performance due to overhead
+
 
 ### Export formatted data
 

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/README.md

@@ -10,7 +10,7 @@
 This project provides simple tools to compare the content of a directory against a reference
 directory.
 
-This is useful to check the results of a process that generate several files, like a luigi
+This is useful to check the results of a process that generates several files, like a luigi
 workflow for example.
 
 
@@ -36,7 +36,9 @@ the files, so it is not able to report which values are different.
 
 ### Compare two directories
 
-If one wants to compare two directories with the following structures:
+#### Basic Directory Comparison
+
+Let's compare two directories with the following structures:
 
 ```bash
 └── reference_dir
@@ -55,6 +57,9 @@ If one wants to compare two directories with the following structures:
     └── file_1.c
 ```
 
+The `reference` directory contains all the files that should be checked in the `compared` directory,
+which means that extraneous files in the `compared` directory are just ignored.
+
 These two directories can be compared with the following code:
 
 ```python
@@ -63,7 +68,17 @@ import dir_content_diff
 dir_content_diff.compare_trees("reference_dir", "compared_dir")
 ```
 
-This code will return an empty dictionary because no difference was detected.
+> [!WARNING]
+> The order of the parameters is important: the first path is considered as the `reference`
+> directory while the second one is the `compared` directory. Inverting the parameters may return
+> a different result (in this example it would return that the file `sub_file_3.b` is missing).
+
+If all the files are identical, this code will return an empty dictionary because no difference
+was detected. As mentioned previously, this is because `dir-content-diff` is only looking for files
+in the compared directory that are also present in the reference directory, so the file
+`sub_file_3.b` is just ignored in this case.
+
+#### Using Custom Comparators
 
 If ``reference_dir/file_1.c`` is the following JSON-like file:
 
@@ -106,6 +121,8 @@ The previous code will output the following dictionary:
 }
 ```
 
+#### Assertion-based Comparison
+
 It is also possible to check whether the two directories are equal or not with the following code:
 
 ```python
@@ -124,7 +141,9 @@ Changed the value of '[a]' from 1 to 2.
 Changed the value of '[b][0]' from 1 to 10.
 ```
 
-Finally, the comparators have parameters that can be passed either to be used for all files of a
+#### Advanced Configuration Options
+
+The comparators have parameters that can be passed either to be used for all files of a
 given extension or only for a specific file:
 
 ```python
@@ -152,6 +171,8 @@ dir_content_diff.assert_equal_trees(
 
 Each comparator has different arguments that are detailed in the documentation.
 
+##### File-specific Comparators
+
 It's also possible to specify a arbitrary comparator for a specific file:
 
 ```python
@@ -163,7 +184,9 @@ specific_args = {
 }
 ```
 
-And last but not least, it's possible to use regular expressions to associate specific arguments to
+##### Pattern-based Configuration
+
+Another possibility is to use regular expressions to associate specific arguments to
 a set of files:
 
 ```python
@@ -175,6 +198,91 @@ specific_args = {
 }
 ```
 
+##### File Filtering
+
+Last but not least, it's possible to filter files from the reference directory (for example
+because the reference directory contains temporary files that should not be compared). For
+example, the following code will ignore all files whose name does not start with `file_` and does
+not ends with `_tmp.yaml`:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    include_patterns=[r"file_.*"],
+    exclude_patterns=[r".*_tmp\.yaml"],
+)
+```
+
+
+### Parallel Execution
+
+By default, `dir-content-diff` runs file comparisons sequentially. However, for improved performance when comparing large numbers of files, parallel execution is available using either thread-based or process-based concurrency.
+
+#### Configuration Options
+
+Parallel execution can be configured using the following parameters:
+
+- **`executor_type`**: Controls the type of parallel execution:
+  - `"sequential"` (default): No parallel execution, files are compared one by one
+  - `"thread"`: Uses `ThreadPoolExecutor` (recommended for I/O-bound tasks)
+  - `"process"`: Uses `ProcessPoolExecutor` (recommended for CPU-intensive comparisons)
+
+- **`max_workers`**: Maximum number of worker threads/processes. If `None` (default), it defaults to `min(32, (os.cpu_count() or 1) + 4)`.
+
+#### Usage Examples
+
+Enable thread-based parallel execution:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="thread",
+    max_workers=8
+)
+```
+
+Enable process-based parallel execution with automatic worker count:
+
+```python
+import dir_content_diff
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    executor_type="process"
+)
+```
+
+Using a configuration object:
+
+```python
+import dir_content_diff
+
+config = dir_content_diff.ComparisonConfig(
+    executor_type="thread",
+    max_workers=4
+)
+
+dir_content_diff.compare_trees(
+    "reference_dir",
+    "compared_dir",
+    config=config
+)
+```
+
+#### Performance Considerations
+
+- **Thread-based execution** (`executor_type="thread"`) is generally recommended for most use cases as file comparisons are typically I/O-bound operations
+- **Process-based execution** (`executor_type="process"`) may be beneficial when using computationally intensive comparators or when dealing with very large files
+- Parallel execution is automatically disabled for single file comparisons and falls back to sequential execution when only one file needs to be compared
+- The optimal number of workers depends on your system's capabilities and the nature of your files; too many workers may actually decrease performance due to overhead
+
 
 ### Export formatted data
 

dir_content_diff-1.14.0/dir_content_diff/__init__.py

@@ -0,0 +1,53 @@
+"""dir-content-diff package.
+
+Simple tool to compare directory contents.
+"""
+
+# LICENSE HEADER MANAGED BY add-license-header
+# Copyright (c) 2023-2025 Blue Brain Project, EPFL.
+#
+# This file is part of dir-content-diff.
+# See https://github.com/BlueBrain/dir-content-diff for further info.
+#
+# SPDX-License-Identifier: Apache-2.0
+# LICENSE HEADER MANAGED BY add-license-header
+
+import importlib.metadata
+
+# Import base comparators
+from dir_content_diff.base_comparators import BaseComparator
+from dir_content_diff.base_comparators import DefaultComparator
+
+# Import core comparison functionality
+from dir_content_diff.config import ComparisonConfig
+from dir_content_diff.core import ComparatorType
+from dir_content_diff.core import assert_equal_trees
+from dir_content_diff.core import compare_files
+from dir_content_diff.core import compare_trees
+from dir_content_diff.core import export_formatted_file
+from dir_content_diff.core import pick_comparator
+
+# Import comparator registry functionality
+from dir_content_diff.registry import get_comparators
+from dir_content_diff.registry import register_comparator
+from dir_content_diff.registry import reset_comparators
+from dir_content_diff.registry import unregister_comparator
+
+__version__ = importlib.metadata.version("dir-content-diff")
+
+# For backward compatibility, ensure all public functions are available
+__all__ = [
+    "ComparisonConfig",
+    "ComparatorType",
+    "assert_equal_trees",
+    "compare_files",
+    "compare_trees",
+    "export_formatted_file",
+    "pick_comparator",
+    "get_comparators",
+    "register_comparator",
+    "reset_comparators",
+    "unregister_comparator",
+    "BaseComparator",
+    "DefaultComparator",
+]

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/dir_content_diff/base_comparators.py

@@ -513,7 +513,7 @@ class XmlComparator(DictComparator):
             elif str(text).lower() == "false":
                 res = False
             else:
-                raise ValueError("Bool attributes expect 'true' or 'false'.")
+                raise ValueError("Boolean attributes expect 'true' or 'false'.")
         elif value_type == "list":
             res = []
         elif value_type == "dict":

{dir_content_diff-1.12.1 → dir_content_diff-1.14.0}/dir_content_diff/cli/__init__.py

@@ -13,15 +13,17 @@ import json
 import logging
 import sys
 from pathlib import Path
+from typing import Optional
+from typing import Union
 
 import click
 from yaml import safe_load
 
-from dir_content_diff import _DEFAULT_EXPORT_SUFFIX
 from dir_content_diff import compare_files
 from dir_content_diff import compare_trees
 from dir_content_diff import export_formatted_file
 from dir_content_diff import pick_comparator
+from dir_content_diff.core import _DEFAULT_EXPORT_SUFFIX
 from dir_content_diff.util import LOGGER
 
 
@@ -102,6 +104,19 @@ def load_config(ctx, param, value):  # pylint: disable=unused-argument
     default=False,
     help="Sort the differences by file name.",
 )
+@click.option(
+    "--executor-type",
+    type=click.Choice(["sequential", "thread", "process"]),
+    default="sequential",
+    help="Type of executor for execution. 'sequential' for single-threaded, "
+    "'thread' for I/O-bound tasks, 'process' for CPU-bound tasks.",
+)
+@click.option(
+    "--max-workers",
+    type=int,
+    help="Maximum number of worker threads/processes for parallel execution. "
+    "Only used with 'thread' or 'process' executor types.",
+)
 @click.version_option()
 @click.pass_context
 def main(ctx, *args, **kwargs):
@@ -120,16 +135,31 @@ def main(ctx, *args, **kwargs):
 
     ref = Path(kwargs.pop("reference_input"))
     comp = Path(kwargs.pop("compared_input"))
+
+    # Extract execution options
+    executor_type = kwargs.pop("executor_type", "sequential")
+    max_workers = kwargs.pop("max_workers", None)
+
     input_diff(
         ref,
         comp,
         ctx.config,
         kwargs.pop("export_formatted_files", False),
         kwargs.pop("sort_diffs", False),
+        executor_type,
+        max_workers,
     )
 
 
-def input_diff(ref, comp, config, export_formatted_files=False, sort_diffs=False):
+def input_diff(
+    ref: Union[str, Path],
+    comp: Union[str, Path],
+    config,
+    export_formatted_files: Union[bool, str] = False,
+    sort_diffs: bool = False,
+    executor_type: str = "sequential",
+    max_workers: Optional[int] = None,
+):
     """Compute and display differences from given inputs."""
     ref = Path(ref)
     comp = Path(comp)
@@ -145,6 +175,8 @@ def input_diff(ref, comp, config, export_formatted_files=False, sort_diffs=False
             comp,
             specific_args=config,
             export_formatted_files=export_formatted_files,
+            executor_type=executor_type,
+            max_workers=max_workers,
         )
     else:
         comparator_name = config.pop("comparator", None)