gerrit-clone 2.2.4__tar.gz → 2.3.0__tar.gz

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/.pre-commit-config.yaml

@@ -74,14 +74,14 @@ repos:
         types: [yaml]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: a56c0b927e6465d37cae3e97d35d4d18ab2b96cd  # frozen: v0.16.9
+    rev: f12be1ebaa5351c1fc76472de98db2c3446c8253  # frozen: v0.16.10
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: 7ff8d35ae36a7d2b968f2f90b4c723e292e594ee  # frozen: v2.3.1
+    rev: 2834ec6639549dd6796205c8f011dedcd587288b  # frozen: v2.4.0
     hooks:
       - id: mypy
         additional_dependencies:
@@ -126,12 +126,12 @@ repos:
       - id: codespell
 
   - repo: https://github.com/DetachHead/basedpyright-prek-mirror
-    rev: 706e6acebe9e88c2c28d1527d3085a14b11794c4  # frozen: 1.40.1
+    rev: a11caebebdbf797e1e96611123db279b334a7675  # frozen: 1.40.2
     hooks:
       - id: basedpyright
 
   - repo: https://github.com/lfreleng-actions/gha-workflow-linter
-    rev: 0c582e431022f1c7004196fb4ea32744d1fe4843  # frozen: v1.7.0
+    rev: b23b60e5884e985320a030e6114032152e2373cd  # frozen: v1.7.1
     hooks:
       - id: gha-workflow-linter
 

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gerrit-clone
-Version: 2.2.4
+Version: 2.3.0
 Summary: A multi-threaded CLI tool for bulk cloning repositories from Gerrit servers
 Project-URL: Homepage, https://github.com/lfreleng-actions/gerrit-clone-action
 Project-URL: Documentation, https://github.com/lfreleng-actions/gerrit-clone-action#readme
@@ -237,24 +237,35 @@ nothing. If the
 command finds no repositories at all, it says so and exits with status 1
 rather than reporting success.
 
-A repository that content filtering (`--remove-files`, `--git-filter`,
-`--redact-secrets`) has rewritten gets refreshed only by a run whose filters
-include the ones that rewrote it, since fetching forces the original history
-back, filtered content included. Each repository records which filters
-rewrote it, keeping each `--git-filter` token as a SHA-256 digest only.
-Refresh copies such a mirror, fetches and filters the copy, and publishes
-its refs only when every step succeeds, so a failure leaves the mirror
-unchanged. A run without those filters skips the mirror and says which
-filters it lacks; the same applies to re-running `clone` and `mirror`. A
-filtered working copy (`--no-mirror`) is always skipped: re-clone it to
-update it. Every remote of a filtered repository refuses pushes.
-Releases up to v2.2.4 recorded no filters, so refresh skips any
-repository they filtered, whatever filters you pass, and asks you to
-re-clone it. It recognises one by the traces those releases left: the
+Content filtering (`--remove-files`, `--git-filter`, `--redact-secrets`) is a
+decision about a whole clone tree, and later runs keep to it. The first run
+that filters writes the decision to `.gerrit-clone/filter-policy.json` at the
+root of the tree. Every later `clone`, `refresh` or `mirror` in that tree
+applies it, even without the options: to repositories cloned later, to files
+that appear upstream later, and to a repository you delete and clone again.
+Options add to the decision; leaving one out never removes a filter. The file
+keeps each `--git-filter` token as a SHA-256 digest only, so a run must pass
+those tokens again. If a run lacks a project's tokens, `refresh` skips that
+project, `clone` exits with a filtering failure, and `mirror` aborts the
+whole batch before it pushes anything; each gives the number of tokens the
+project needs. A file the tool cannot read stops the run.
+
+Each repository also records which filters rewrote it. Fetching would force
+the original history back, filtered content included, so refresh copies each
+mirror it filters, fetches and filters the copy, and publishes its refs only
+when every step succeeds; a failure leaves the mirror unchanged. A filtered
+working copy (`--no-mirror`) always gets skipped: re-clone it to update it.
+Every remote of a filtered repository refuses pushes. A tree filtered before
+the tree-level file existed gains one from these per-repository records on
+its next run. Releases up to v2.2.4 recorded no filters at all, so refresh
+skips any repository they filtered, whatever filters you pass, and asks you
+to re-clone it, and `mirror --overwrite` leaves it out of the run rather
+than cloning or pushing it: delete it yourself, then clone it again with the
+filters it needs. The tool recognises one by the traces those releases left: the
 commit map a `git filter-repo` rewrite leaves in the git directory, or a
 branch tip committed by the file-removal fallback (`Remove filtered files
 for platform sync`).
-A `--dry-run` applies no content filters.
+A `--dry-run` applies no content filters and records nothing.
 
 Refresh all repositories in the current directory:
 
@@ -915,8 +926,16 @@ export GITHUB_TOKEN=github_pat_your_token_here
 
 **With `--overwrite` flag**:
 
-- Removes and re-clones local repositories
-- Still skips existing GitHub repositories unless `--recreate` is also used
+- Removes and re-clones local repositories, then pushes each one
+  (`git push --mirror`) to its GitHub repository, existing ones included;
+  the tree's recorded content filters still apply first
+- Leaves out of the run any repository a release up to v2.2.4
+  content-filtered, and any directory holding one: those releases recorded
+  no filters to apply again, so the run neither re-clones nor pushes it,
+  `--recreate` leaves its GitHub repository alone, and the manifest lists
+  it as skipped
+- Without `--recreate`, reuses existing GitHub repositories rather than
+  deleting them
 
 **With both `--recreate` and `--overwrite`**:
 

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/README.md

@@ -195,24 +195,35 @@ nothing. If the
 command finds no repositories at all, it says so and exits with status 1
 rather than reporting success.
 
-A repository that content filtering (`--remove-files`, `--git-filter`,
-`--redact-secrets`) has rewritten gets refreshed only by a run whose filters
-include the ones that rewrote it, since fetching forces the original history
-back, filtered content included. Each repository records which filters
-rewrote it, keeping each `--git-filter` token as a SHA-256 digest only.
-Refresh copies such a mirror, fetches and filters the copy, and publishes
-its refs only when every step succeeds, so a failure leaves the mirror
-unchanged. A run without those filters skips the mirror and says which
-filters it lacks; the same applies to re-running `clone` and `mirror`. A
-filtered working copy (`--no-mirror`) is always skipped: re-clone it to
-update it. Every remote of a filtered repository refuses pushes.
-Releases up to v2.2.4 recorded no filters, so refresh skips any
-repository they filtered, whatever filters you pass, and asks you to
-re-clone it. It recognises one by the traces those releases left: the
+Content filtering (`--remove-files`, `--git-filter`, `--redact-secrets`) is a
+decision about a whole clone tree, and later runs keep to it. The first run
+that filters writes the decision to `.gerrit-clone/filter-policy.json` at the
+root of the tree. Every later `clone`, `refresh` or `mirror` in that tree
+applies it, even without the options: to repositories cloned later, to files
+that appear upstream later, and to a repository you delete and clone again.
+Options add to the decision; leaving one out never removes a filter. The file
+keeps each `--git-filter` token as a SHA-256 digest only, so a run must pass
+those tokens again. If a run lacks a project's tokens, `refresh` skips that
+project, `clone` exits with a filtering failure, and `mirror` aborts the
+whole batch before it pushes anything; each gives the number of tokens the
+project needs. A file the tool cannot read stops the run.
+
+Each repository also records which filters rewrote it. Fetching would force
+the original history back, filtered content included, so refresh copies each
+mirror it filters, fetches and filters the copy, and publishes its refs only
+when every step succeeds; a failure leaves the mirror unchanged. A filtered
+working copy (`--no-mirror`) always gets skipped: re-clone it to update it.
+Every remote of a filtered repository refuses pushes. A tree filtered before
+the tree-level file existed gains one from these per-repository records on
+its next run. Releases up to v2.2.4 recorded no filters at all, so refresh
+skips any repository they filtered, whatever filters you pass, and asks you
+to re-clone it, and `mirror --overwrite` leaves it out of the run rather
+than cloning or pushing it: delete it yourself, then clone it again with the
+filters it needs. The tool recognises one by the traces those releases left: the
 commit map a `git filter-repo` rewrite leaves in the git directory, or a
 branch tip committed by the file-removal fallback (`Remove filtered files
 for platform sync`).
-A `--dry-run` applies no content filters.
+A `--dry-run` applies no content filters and records nothing.
 
 Refresh all repositories in the current directory:
 
@@ -873,8 +884,16 @@ export GITHUB_TOKEN=github_pat_your_token_here
 
 **With `--overwrite` flag**:
 
-- Removes and re-clones local repositories
-- Still skips existing GitHub repositories unless `--recreate` is also used
+- Removes and re-clones local repositories, then pushes each one
+  (`git push --mirror`) to its GitHub repository, existing ones included;
+  the tree's recorded content filters still apply first
+- Leaves out of the run any repository a release up to v2.2.4
+  content-filtered, and any directory holding one: those releases recorded
+  no filters to apply again, so the run neither re-clones nor pushes it,
+  `--recreate` leaves its GitHub repository alone, and the manifest lists
+  it as skipped
+- Without `--recreate`, reuses existing GitHub repositories rather than
+  deleting them
 
 **With both `--recreate` and `--overwrite`**:
 

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/src/gerrit_clone/cli_clone_run.py

@@ -21,12 +21,7 @@ from rich.text import Text
 from gerrit_clone import __version__, cli_hooks
 from gerrit_clone import cli_clone_setup as setup
 from gerrit_clone.cli_app import is_github_actions_context
-from gerrit_clone.content_filter import (
-    apply_content_filters,
-    is_shallow_repository,
-    normalize_file_patterns,
-    parse_git_filter_spec,
-)
+from gerrit_clone.content_stage import filter_repository
 from gerrit_clone.error_codes import DiscoveryError, ExitCode
 from gerrit_clone.rich_status import (
     handle_crash_display,
@@ -37,7 +32,8 @@ from gerrit_clone.rich_status import (
 if TYPE_CHECKING:
     from gerrit_clone.cli_clone_models import CloneRequest
     from gerrit_clone.cli_session import CliSession
-    from gerrit_clone.models import BatchResult, CloneResult, Config
+    from gerrit_clone.content_spec import ContentFilterSpec
+    from gerrit_clone.models import BatchResult, Config
 
 
 def run_clone(request: CloneRequest, session: CliSession) -> None:
@@ -70,7 +66,7 @@ def run_clone(request: CloneRequest, session: CliSession) -> None:
         setup.show_startup_banner(console, config)
 
     batch_result = _clone_repositories(config)
-    _apply_content_filters(request, console, batch_result)
+    _apply_content_filters(request, console, batch_result, config.content_filters)
     _report_results(request, session, batch_result)
     exit_code = _determine_exit_code(session, batch_result)
 
@@ -105,116 +101,53 @@ def _clone_repositories(config: Config) -> BatchResult:
 
 
 def _apply_content_filters(
-    request: CloneRequest, console: Console, batch_result: BatchResult
+    request: CloneRequest,
+    console: Console,
+    batch_result: BatchResult,
+    spec: ContentFilterSpec | None,
 ) -> None:
-    """Apply content filters to successfully cloned repositories, if requested."""
-    remove_file_patterns = (
-        normalize_file_patterns([request.remove_files])
-        if request.remove_files
-        else None
-    )
-    git_filter_projects = (
-        parse_git_filter_spec(request.git_filter) if request.git_filter else None
-    )
-    if not (remove_file_patterns or git_filter_projects or request.redact_secrets):
+    """Filter every successful clone, as each project's filters decide."""
+    if spec is None:
         return
 
     if not request.quiet:
         console.print("[cyan]🔧 Applying content filters...[/cyan]")
     filter_success = filter_fail = 0
     for cr in batch_result.results:
+        if cr.skipped and cr.path and spec.missing_tokens(cr.path):
+            # The refresh pass refused it for the tokens, before any
+            # fetch.  Clone reports that as the filtering failure it is,
+            # whether or not upstream had moved on.
+            filter_fail += 1
+            if not request.quiet:
+                console.print(
+                    f"[yellow]⚠️  Filter failed for {cr.project.name}: "
+                    f"{cr.error_message}[/yellow]"
+                )
+            continue
         if not cr.success or not cr.path:
             continue
         if cr.content_filtered:
             # Re-filtered as part of its staged refresh already.
             filter_success += 1
             continue
-        succeeded, failed = _filter_repository(
-            request, console, cr, remove_file_patterns, git_filter_projects
+        reason = filter_repository(
+            spec, cr.path, spec.project_name(cr.path), request.clone_timeout
         )
-        filter_success += succeeded
-        filter_fail += failed
-    if not request.quiet:
-        console.print(
-            f"[cyan]Content filtering: {filter_success} succeeded, {filter_fail} failed[/cyan]"
-        )
-    if filter_fail > 0:
-        raise typer.Exit(ExitCode.GENERAL_ERROR)
-
-
-def _filter_repository(
-    request: CloneRequest,
-    console: Console,
-    result: CloneResult,
-    remove_file_patterns: list[str] | None,
-    git_filter_projects: dict[str, list[str]] | None,
-) -> tuple[int, int]:
-    """Filter one cloned repository.
-
-    Returns:
-        Tuple of (successes, failures) to add to the run totals
-    """
-    # Fail closed on shallow repositories when history-
-    # dependent filters are requested: --git-filter /
-    # --redact-secrets rely on the full commit history, so
-    # a shallow repo can hide older secrets (and a later
-    # unshallow fetch could reintroduce blocked content),
-    # giving a false sense of safety.  Probe each repo
-    # individually rather than only checking config.depth:
-    # a repo that already existed locally (e.g. cloned
-    # earlier with --depth) is shallow even when this run
-    # sets no --depth.  --remove-files targets file paths
-    # present at the branch tips and does not depend on
-    # full history being available (though it may still
-    # rewrite history via git filter-repo when that tool
-    # is present), so it is still applied; only the unsafe
-    # history-scanning filters are dropped for the shallow
-    # repo.
-    repo_git_filter = git_filter_projects
-    repo_redact = request.redact_secrets
-    history_filters_skipped = False
-    if (git_filter_projects or request.redact_secrets) and (
-        is_shallow_repository(result.path)
-    ):
+        if reason is None:
+            filter_success += 1
+            continue
+        filter_fail += 1
         if not request.quiet:
             console.print(
-                "[red]❌ Refusing to run --git-filter / "
-                "--redact-secrets on shallow repo "
-                f"{result.project.name}: truncated history can hide "
-                "older secrets. Re-clone without --depth.[/red]"
+                f"[yellow]⚠️  Filter failed for {cr.project.name}: {reason}[/yellow]"
             )
-        # The requested redaction/rewrite did not run, so
-        # this repo counts as a filtering failure even if
-        # the safe --remove-files step below succeeds.
-        history_filters_skipped = True
-        repo_git_filter = None
-        repo_redact = False
-        if not remove_file_patterns:
-            # Nothing history-independent left to do.
-            return 0, 1
-    success, error = apply_content_filters(
-        result.path,
-        result.project.name,
-        remove_patterns=remove_file_patterns,
-        git_filter_projects=repo_git_filter,
-        redact_secrets=repo_redact,
-        timeout=request.clone_timeout,
-    )
-    if history_filters_skipped:
-        # Already counted as a failure above; don't also count
-        # the safe --remove-files step as a success.
-        if not success and not request.quiet:
-            console.print(
-                f"[yellow]⚠️  Filter failed for {result.project.name}: {error}[/yellow]"
-            )
-        return 0, 1
-    if success:
-        return 1, 0
     if not request.quiet:
         console.print(
-            f"[yellow]⚠️  Filter failed for {result.project.name}: {error}[/yellow]"
+            f"[cyan]Content filtering: {filter_success} succeeded, {filter_fail} failed[/cyan]"
         )
-    return 0, 1
+    if filter_fail > 0:
+        raise typer.Exit(ExitCode.GENERAL_ERROR)
 
 
 def _report_results(

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/src/gerrit_clone/cli_clone_setup.py

@@ -21,7 +21,9 @@ from rich.text import Text
 from gerrit_clone import cli_hooks
 from gerrit_clone.cli_app import format_version_string
 from gerrit_clone.config import ConfigurationError, load_config
-from gerrit_clone.content_policy import ContentFilterSpec
+from gerrit_clone.content_intent import IntentError
+from gerrit_clone.content_intent_resolve import resolve_filters
+from gerrit_clone.content_spec import ContentFilterSpec
 from gerrit_clone.error_codes import ExitCode
 from gerrit_clone.file_logging import (
     cli_args_to_dict,
@@ -318,13 +320,19 @@ def build_config(
     except ConfigurationError as e:
         _report_configuration_error(session, e)
         raise typer.Exit(ExitCode.CONFIGURATION_ERROR) from e
-    config.content_filters = ContentFilterSpec.from_options(
+    options = ContentFilterSpec.from_options(
         request.remove_files, request.git_filter, request.redact_secrets, config.path
     )
+    try:
+        # Written down before anything is cloned or refreshed.
+        config.content_filters = resolve_filters(config.path, options, persist=True)
+    except IntentError as e:
+        _report_configuration_error(session, e)
+        raise typer.Exit(ExitCode.CONFIGURATION_ERROR) from e
     return config
 
 
-def _report_configuration_error(session: CliSession, error: ConfigurationError) -> None:
+def _report_configuration_error(session: CliSession, error: Exception) -> None:
     """Log and display a configuration error before aborting."""
     if session.file_logger:
         session.file_logger.error("Configuration error: %s", str(error))

{gerrit_clone-2.2.4 → gerrit_clone-2.3.0}/src/gerrit_clone/cli_mirror_run.py

@@ -28,6 +28,7 @@ from gerrit_clone.cli_mirror_setup import (
     validate_request,
 )
 from gerrit_clone.content_filter import normalize_file_patterns, parse_git_filter_spec
+from gerrit_clone.content_intent import IntentError
 from gerrit_clone.error_codes import ExitCode
 from gerrit_clone.mirror_manager import (
     MirrorBatchResult,
@@ -43,6 +44,7 @@ if TYPE_CHECKING:
 
     from gerrit_clone.cli_mirror_models import MirrorRequest
     from gerrit_clone.cli_session import CliSession
+    from gerrit_clone.mirror_models import MirrorResult
     from gerrit_clone.models import Config, Project
 
 
@@ -102,11 +104,17 @@ def run_mirror(request: MirrorRequest, session: CliSession) -> None:
         redact_secrets=request.redact_secrets,
     )
 
+    if not projects_to_mirror:
+        # Nothing to mirror, but the run's filters still decide for the
+        # projects a later run selects: the manager records them.
+        _mirror(mirror_manager, [], console)
+        raise typer.Exit(0)
+
     started_at = datetime.now(UTC)
     if not request.quiet:
         console.print("🚀 Starting mirror operation...")
 
-    results = mirror_manager.mirror_projects(projects_to_mirror)
+    results = _mirror(mirror_manager, projects_to_mirror, console)
 
     completed_at = datetime.now(UTC)
 
@@ -130,6 +138,17 @@ def run_mirror(request: MirrorRequest, session: CliSession) -> None:
     _finish(request, session, batch_result)
 
 
+def _mirror(
+    mirror_manager: MirrorManager, projects: list[Project], console: Console
+) -> list[MirrorResult]:
+    """Mirror *projects*; a tree whose filter intent is unreadable stops the run."""
+    try:
+        return mirror_manager.mirror_projects(projects)
+    except IntentError as exc:
+        console.print(f"[red]❌ {exc}[/red]")
+        raise typer.Exit(ExitCode.CONFIGURATION_ERROR) from exc
+
+
 def _select_projects(
     request: MirrorRequest,
     console: Console,
@@ -137,12 +156,16 @@ def _select_projects(
     project_filters: list[str],
     exclude_filters: list[str],
 ) -> list[Project]:
-    """Discover projects and apply the include/exclude filters."""
+    """Discover projects and apply the include/exclude filters.
+
+    Returns:
+        The projects to mirror; empty, after saying so, if none are.
+    """
     all_projects, _discovery_stats = cli_hooks.discover_projects(config)
 
     if not all_projects:
         console.print("[yellow]No projects found on Gerrit server[/yellow]")
-        raise typer.Exit(0)
+        return []
 
     # Filter projects by include/exclude patterns
     if project_filters or exclude_filters:
@@ -156,7 +179,7 @@ def _select_projects(
 
     if not projects_to_mirror:
         console.print("[yellow]No projects matched the specified filters[/yellow]")
-        raise typer.Exit(0)
+        return []
 
     if not request.quiet:
         console.print(