batchfetch 1.2.6__tar.gz → 1.2.7__tar.gz

{batchfetch-1.2.6 → batchfetch-1.2.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: batchfetch
-Version: 1.2.6
+Version: 1.2.7
 Summary: Efficiently clone and pull multiple Git repositories.
 Home-page: https://github.com/jamescherti/batchfetch
 Author: James Cherti
@@ -22,27 +22,24 @@ License-File: LICENSE
 
 ## Introduction
 
-Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously.
+Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously. With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
 
-With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
+But why use Batchfetch? Because it is extremely fast, cloning repositories quickly by running Git operations in parallel. It intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories. Additionally, it allows specifying the revision (for Git), ensuring that the cloned repository matches the exact version you require.
 
 Batchfetch is ideal for quickly cloning or pulling multiple Git repositories. It is also useful for cloning various addons, such as Vim plugins, Emacs packages, Ansible roles, Ansible collections, and other addons available on websites like GitHub, Codeberg, and GitLab.
 
-## Features:
-- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
-- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
-- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
-- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
-
 ## Installation
 
+Here is how to install *batchfetch* using [pip](https://pypi.org/project/pip/):
 ```
 pip install --user batchfetch
 ```
 
-The pip command above will install the `batchfetch` executable in the directory `~/.local/bin/`.
+The pip command above installs the *batchfetch* executable in the `~/.local/bin/` directory. Omitting the `--user` flag will install it system-wide.
+
+## Usage
 
-## Example
+### Example of a `batchfetch.yaml` file
 
 Here is an example of a `batchfetch.yaml` file:
 
@@ -52,43 +49,55 @@ Here is an example of a `batchfetch.yaml` file:
 tasks:
   # Clone the default branch of the general.el repository to the
   # './general.el' directory
-  - git: https://github.com/noctuid/general.el
+  - git: https://github.com/jamescherti/compile-angel.el
 
   # Clone the tag 1.5 of the consult repository to the './consult'
   # directory
-  - git: https://github.com/minad/consult
-    revision: "1.5"
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
 
   # Clone the s.el repository to the './another-name.el' directory
-  - git: https://github.com/magnars/s.el
-    path: another-name.el
-    revision: dda84d38fffdaf0c9b12837b504b402af910d01d
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
 
-  # Delete './impatient-mode'
-  - git: https://github.com/skeeto/impatient-mode
+  # Delete the local copy of the following repository
+  - git: https://github.com/jamescherti/dir-config.el
     delete: true
 ```
 
 Execute the `batchfetch` command from the same directory as `batchfetch.yml` to make it clone or update the local copies of the repositories above.
 
-## Usage
+## Command-line options
 
 Here are the various options that `batchfetch` provides, along with descriptions of their usage:
 
 ```
-usage: batchfetch [--option] [args]
+usage: batchfetch [--option]
 
-Command line interface.
-
-positional arguments:
-  N                     Specify the batchfetch YAML file(s) (default: './batchfetch.yaml').
+Efficiently clone/pull multiple Git repositories in parallel.
 
 options:
   -h, --help            show this help message and exit
-  -j JOBS, --jobs JOBS  Run up to N Number of parallel processes (Default: 5).
+  -f FILE, --file FILE  Specify the batchfetch YAML file (default:
+                        './batchfetch.yaml').
+  -C DIRECTORY, --directory DIRECTORY
+                        Change the working directory before reading the
+                        batchfetch.yaml file. If not specified, the directory is
+                        set to the parent directory of the batchfetch.yaml file.
+  -j JOBS, --jobs JOBS  Run up to N parallel processes (default: 5).
+                        Alternatively, the BATCHFETCH_JOBS environment variable
+                        can be used to configure the number of jobs.
   -v, --verbose         Enable verbose mode.
 ```
 
+## Features
+- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
+- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
+- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
+- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
+- Detect files that should not be present in directories managed by batchfetch, known as untracked files.
+
 ## Frequently Asked Questions
 
 ### What are untracked files?
@@ -114,6 +123,43 @@ tasks:
 
 By default, *batchfetch.yaml* is the only untracked file that is ignored. The user does not need to add it to the *ignore_untracked_paths* option.
 
+### How is the Git local paths handled?
+
+When "path:" is specified, that's the path that is used.
+
+When "path:" is not specified, Batchfetch attempts to determine the path name by extracting the repository name from the URI (e.g., `https://domain.com/repo` becomes `repo`). If the URL ends with a `.git` extension, it removes the extension (e.g., `https://domain.com/repo.git` becomes `repo`).
+
+### How does Batchfetch detect when a git fetch is necessary?
+
+Batchfetch is fast, not only because it runs Git commands in parallel, but also because it intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories.
+
+When the user has specifies a revision (branch or commit reference), Batchfetch only performs a `git fetch` if that revision does not exist locally. If the revision is already up to date, it simply proceeds to the next repository in the queue.
+
+That's why it is highly recommended to always specify the revision to speed up Batchfetch, if speed is important to you. Here is an example of a `batchfetch.yaml` file where the branch (`1.1.0`) or commit reference (`b9c6d9b6134b4981760893254f804a371ffbc899`) is specified:
+``` yaml
+tasks:
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
+
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
+```
+
+### How to execute a command before and after a task?
+
+To execute a command both before and after a specific task, you can define the `exec_before` and `exec_after` directives within the task configuration. These directives specify commands to be executed at the respective stages of the task lifecycle.
+
+Here is an example:
+``` yaml
+---
+tasks:
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    exec_before: ["sh", "-c", "echo exec_before_task"]
+    exec_after: ["sh", "-c", "echo exec_after_task"]
+```
+
 ## License
 
 Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)

batchfetch-1.2.7/README.md

@@ -0,0 +1,156 @@
+# Batchfetch - Efficiently clone or pull multiple Git repositories in parallel
+
+## Introduction
+
+Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously. With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
+
+But why use Batchfetch? Because it is extremely fast, cloning repositories quickly by running Git operations in parallel. It intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories. Additionally, it allows specifying the revision (for Git), ensuring that the cloned repository matches the exact version you require.
+
+Batchfetch is ideal for quickly cloning or pulling multiple Git repositories. It is also useful for cloning various addons, such as Vim plugins, Emacs packages, Ansible roles, Ansible collections, and other addons available on websites like GitHub, Codeberg, and GitLab.
+
+## Installation
+
+Here is how to install *batchfetch* using [pip](https://pypi.org/project/pip/):
+```
+pip install --user batchfetch
+```
+
+The pip command above installs the *batchfetch* executable in the `~/.local/bin/` directory. Omitting the `--user` flag will install it system-wide.
+
+## Usage
+
+### Example of a `batchfetch.yaml` file
+
+Here is an example of a `batchfetch.yaml` file:
+
+```yaml
+---
+
+tasks:
+  # Clone the default branch of the general.el repository to the
+  # './general.el' directory
+  - git: https://github.com/jamescherti/compile-angel.el
+
+  # Clone the tag 1.5 of the consult repository to the './consult'
+  # directory
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
+
+  # Clone the s.el repository to the './another-name.el' directory
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
+
+  # Delete the local copy of the following repository
+  - git: https://github.com/jamescherti/dir-config.el
+    delete: true
+```
+
+Execute the `batchfetch` command from the same directory as `batchfetch.yml` to make it clone or update the local copies of the repositories above.
+
+## Command-line options
+
+Here are the various options that `batchfetch` provides, along with descriptions of their usage:
+
+```
+usage: batchfetch [--option]
+
+Efficiently clone/pull multiple Git repositories in parallel.
+
+options:
+  -h, --help            show this help message and exit
+  -f FILE, --file FILE  Specify the batchfetch YAML file (default:
+                        './batchfetch.yaml').
+  -C DIRECTORY, --directory DIRECTORY
+                        Change the working directory before reading the
+                        batchfetch.yaml file. If not specified, the directory is
+                        set to the parent directory of the batchfetch.yaml file.
+  -j JOBS, --jobs JOBS  Run up to N parallel processes (default: 5).
+                        Alternatively, the BATCHFETCH_JOBS environment variable
+                        can be used to configure the number of jobs.
+  -v, --verbose         Enable verbose mode.
+```
+
+## Features
+- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
+- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
+- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
+- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
+- Detect files that should not be present in directories managed by batchfetch, known as untracked files.
+
+## Frequently Asked Questions
+
+### What are untracked files?
+
+The parent directory of the "path:" value defines the managed directory, where the directory of each path is considered as the managed directory.
+
+For example, if the "path:" value is `file/my-project`, the managed directory will be `file/`. Any file within `file/` that is not managed by batchfetch will be considered an untracked file.
+
+When *batchfetch* encounters an untracked file, it displays an error message to inform users about paths that are not managed by the system. The message provides clear instructions on how to handle these paths by adding them to the `options.ignore_untracked_paths` list, enabling users to manage untracked files effectively.
+
+Here is an example of a *batchfetch.yaml* file that enables *batchfetch* to accept a list of untracked files:
+
+``` yaml
+options:
+  ignore_untracked_paths:
+    - ./test
+    - /absolute/path
+    - ../relative/path
+
+tasks:
+  - git: https://github.com/user/project
+```
+
+By default, *batchfetch.yaml* is the only untracked file that is ignored. The user does not need to add it to the *ignore_untracked_paths* option.
+
+### How is the Git local paths handled?
+
+When "path:" is specified, that's the path that is used.
+
+When "path:" is not specified, Batchfetch attempts to determine the path name by extracting the repository name from the URI (e.g., `https://domain.com/repo` becomes `repo`). If the URL ends with a `.git` extension, it removes the extension (e.g., `https://domain.com/repo.git` becomes `repo`).
+
+### How does Batchfetch detect when a git fetch is necessary?
+
+Batchfetch is fast, not only because it runs Git commands in parallel, but also because it intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories.
+
+When the user has specifies a revision (branch or commit reference), Batchfetch only performs a `git fetch` if that revision does not exist locally. If the revision is already up to date, it simply proceeds to the next repository in the queue.
+
+That's why it is highly recommended to always specify the revision to speed up Batchfetch, if speed is important to you. Here is an example of a `batchfetch.yaml` file where the branch (`1.1.0`) or commit reference (`b9c6d9b6134b4981760893254f804a371ffbc899`) is specified:
+``` yaml
+tasks:
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
+
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
+```
+
+### How to execute a command before and after a task?
+
+To execute a command both before and after a specific task, you can define the `exec_before` and `exec_after` directives within the task configuration. These directives specify commands to be executed at the respective stages of the task lifecycle.
+
+Here is an example:
+``` yaml
+---
+tasks:
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    exec_before: ["sh", "-c", "echo exec_before_task"]
+    exec_after: ["sh", "-c", "echo exec_after_task"]
+```
+
+## License
+
+Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)
+
+This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with this program.
+
+## Links
+
+- [batchfetch @GitHub](https://github.com/jamescherti/batchfetch)
+- [batchfetch @Pypi](https://pypi.org/project/batchfetch/)

{batchfetch-1.2.6 → batchfetch-1.2.7}/batchfetch/batchfetch_cli.py

@@ -25,7 +25,7 @@ import subprocess
 import sys
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from pathlib import Path
-from typing import Any, Dict, Set
+from typing import Any, Dict, Set, Union
 
 import colorama
 import yaml  # type: ignore
@@ -40,9 +40,10 @@ from .batchfetch_git import BatchFetchGit
 class BatchFetchCli:
     """Command-line-interface that downloads."""
 
-    def __init__(self, max_workers: int, verbose: bool = False):
+    def __init__(self, max_workers: int, verbose: bool, check_untracked: bool):
         self.cfg: dict = {}
         self.folder = Path(".")
+        self.check_untracked = check_untracked
         self.tracked_paths: Dict[Path, Set[str]] = {}
         self.ignore_untracked_paths: Set[Path] = set()
         self.verbose = verbose
@@ -101,7 +102,8 @@ class BatchFetchCli:
                 self.ignore_untracked_paths.add(Path(path).absolute())
                 self.ignore_untracked_paths.add(Path(path).resolve())
                 untracked_paths = None
-                if "options" in yaml_dict:
+                if "options" in yaml_dict and \
+                        "ignore_untracked_paths" in yaml_dict["options"]:
                     untracked_paths = \
                         yaml_dict["options"]["ignore_untracked_paths"]
 
@@ -155,10 +157,12 @@ class BatchFetchCli:
                 print(f"Schema error: {err}.", file=sys.stderr)
                 sys.exit(1)
 
-            dest_path = Path(batchfetch_instance["path"]).resolve()
-            if str(dest_path) in dict_local_dir:
+            dest_path = Path(batchfetch_instance["path"]).absolute()
+            dest_path2 = Path(batchfetch_instance["path"]).resolve()
+            if str(dest_path) in dict_local_dir \
+                    or str(dest_path2) in dict_local_dir:
                 err_str = ("More than one task have the " +
-                           f"destination path '{dest_path}' (" +
+                           f"destination path (" +
                            str(task[keyword]) + " and " +
                            str(dict_local_dir[(str(dest_path))]) +
                            ")")
@@ -232,7 +236,8 @@ class BatchFetchCli:
 
             return False
         else:
-            self._find_untracked_paths()
+            if self.check_untracked:
+                self._find_untracked_paths()
 
             if num_success == 0:
                 print("Nothing to do.")
@@ -269,7 +274,24 @@ class BatchFetchCli:
 
 def parse_args():
     """Parse the command line arguments."""
-    desc = __doc__
+    # Jobs
+    try:
+        jobs = os.environ["BATCHFETCH_JOBS"]
+    except KeyError:
+        jobs = 5
+    else:
+        if jobs:
+            jobs = int(jobs)
+
+    # Check untracked
+    try:
+        check_untracked = os.environ["BATCHFETCH_CHECK_UNTRACKED"]
+    except KeyError:
+        check_untracked = False
+    else:
+        check_untracked = bool(check_untracked)
+
+    desc = "Efficiently clone/pull multiple Git repositories in parallel."
     usage = "%(prog)s [--option]"
     parser = argparse.ArgumentParser(description=desc, usage=usage)
 
@@ -281,15 +303,40 @@ def parse_args():
                               "(default: './batchfetch.yaml')."))
 
     parser.add_argument(
-        "-j", "--jobs", default="5", required=False,
-        help="Run up to N Number of parallel processes (Default: 5).",
+        "-C",
+        "--directory",
+        default=None,
+        required=False,
+        help=("Change the working directory before reading the "
+              "batchfetch.yaml file. If not specified, the directory is "
+              "set to the parent directory of the batchfetch.yaml file.")
     )
 
+    parser.add_argument(
+        "-j",
+        "--jobs",
+        default=jobs,
+        type=int,
+        required=False,
+        help=("Run up to N parallel processes (default: 5). "
+              "Alternatively, the BATCHFETCH_JOBS environment variable can be "
+              "used to configure the number of jobs."))
+
     parser.add_argument(
         "-v", "--verbose", action="store_true", default=False,
         help="Enable verbose mode.",
     )
 
+    parser.add_argument(
+        "-u",
+        "--check-untracked",
+        default=check_untracked,
+        action="store_true",
+        required=False,
+        help=("Abort if untracked files or directories exist. "
+              "Alternatively, set the BATCHFETCH_CHECK_UNTRACKED=1 "
+              "environment variable to enable this check."))
+
     args = parser.parse_args()
 
     if not args.file:
@@ -303,11 +350,16 @@ def parse_args():
     return args
 
 
-def run_batchfetch_procedure(file: Path, args) -> int:
+def run_batchfetch_procedure(file: Path,
+                             directory: Union[None, Path],
+                             verbose: bool,
+                             jobs: int,
+                             check_untracked: bool) -> int:
     errno = 0
-    batchfetch_cli = BatchFetchCli(verbose=args.verbose,
-                                   max_workers=int(args.jobs))
-    os.chdir(file.parent)
+    batchfetch_cli = BatchFetchCli(verbose=verbose,
+                                   max_workers=int(jobs),
+                                   check_untracked=check_untracked)
+    os.chdir(directory)
     batchfetch_cli.load(file)
 
     try:
@@ -337,14 +389,27 @@ def command_line_interface():
         args = parse_args()
         done = []
         file = Path(args.file)
-        file_resolved = file.resolve()
+        file_resolved = file.absolute()
         if not file_resolved:
             print(f"Error: cannot resolve the path {file}",
                   file=sys.stderr)
             sys.exit(1)
 
         done.append(file_resolved)
-        errno |= run_batchfetch_procedure(file, args)
+
+        args.directory = args.directory if args.directory else file.parent
+        if args.verbose and args.jobs:
+            print(f"[FILE] {file}")
+            print(f"[DIR] {args.directory}")
+            print(f"[JOBS] {args.jobs}")
+            print(f"[CHECK UNTRACKED] {args.check_untracked}")
+            print()
+
+        errno |= run_batchfetch_procedure(file=file,
+                                          directory=args.directory,
+                                          verbose=args.verbose,
+                                          jobs=args.jobs,
+                                          check_untracked=args.check_untracked)
 
         sys.exit(errno)
     except BrokenPipeError:

{batchfetch-1.2.6 → batchfetch-1.2.7}/batchfetch/batchfetch_git.py

@@ -93,9 +93,12 @@ class BatchFetchGit(TaskBatchFetch):
             self.values[self.main_key].rstrip("/")
 
         if "path" not in self.values:
-            self.values["path"] = \
-                posixpath.basename(
-                    self.values[self.main_key])  # type: ignore
+            path = \
+                posixpath.basename(self.values[self.main_key])  # type: ignore
+            # Remove .git from the file name
+            if path.endswith(".git"):
+                path = path[:-4]
+            self.values["path"] = path
 
     def update(self):
         """Clone or update a Git repository."""
@@ -120,7 +123,7 @@ class BatchFetchGit(TaskBatchFetch):
 
         self.add_output(f"[GIT {update_type}] {self[self.main_key]}"
                         + (f" (Ref: {self['revision']})"
-                         if self["revision"] else "") + "\n")
+                           if self["revision"] else "") + "\n")
 
         try:
             # Delete
@@ -203,8 +206,8 @@ class BatchFetchGit(TaskBatchFetch):
                 # directly to a commit. You need to resolve it to the
                 # commit it points to. Using `git rev-parse
                 # <tagname>^{commit}` allows getting the right revision.
-                commit_ref = self._git_tags(self["revision"]
-                                            + "^{commit}")[0]
+                commit_ref = self._git_rev_parse_verify(self["revision"]
+                                                        + "^{commit}")[0]
                 self.branch_commit_ref = commit_ref.strip()
             except GitRevisionDoesNotExist:
                 pass
@@ -242,9 +245,7 @@ class BatchFetchGit(TaskBatchFetch):
     def _repo_fetch(self):
         # Merge
         do_git_fetch = self["git_pull"]
-        if self.is_branch:
-            do_git_fetch = True
-        elif not self["revision"]:
+        if not self["revision"]:
             do_git_fetch = True
             self.add_output(self.indent_spaces
                             + "[INFO] Git fetch origin reason: "
@@ -256,12 +257,9 @@ class BatchFetchGit(TaskBatchFetch):
             try:
                 # Check if the revision such as
                 # 0560fe21d1173b2221fd8c600fab818f7eecbad4 exist
-                commit_ref = self._git_tags(self["revision"])[0]
+                commit_ref = self._git_rev_parse_verify(self["revision"])[0]
                 commit_ref = commit_ref.strip()
             except GitRevisionDoesNotExist:
-                pass
-
-            if not commit_ref and not self.is_branch:
                 do_git_fetch = True
                 self.add_output(
                     self.indent_spaces
@@ -269,6 +267,7 @@ class BatchFetchGit(TaskBatchFetch):
                     + f"The revision does not exist: {self['revision']}"
                     + "\n")
 
+            # The revision exists, but if it a branch, git pull anyway
             if not do_git_fetch:
                 try:
                     # Check if the branch is a tag or a branch
@@ -284,31 +283,6 @@ class BatchFetchGit(TaskBatchFetch):
                 except subprocess.CalledProcessError:
                     pass
 
-            if not do_git_fetch and commit_ref:
-                # This is to check if a tag has been changed
-                try:
-                    # If the tag is annotated, it points to a tag object,
-                    # not directly to a commit. You need to resolve it to
-                    # the commit it points to. Using `git rev-parse
-                    # <tagname>^{commit}` allows getting the right
-                    # revision.
-                    commit_ref_head = self._git_tags("HEAD^{commit}")[0]
-                    commit_ref_head = commit_ref_head.strip()
-                except GitRevisionDoesNotExist:
-                    # HEAD is detached
-                    commit_ref_head = None
-
-                # The wanted commit revision does not exist
-                # Or the commit ref of HEAD hasn't changed
-                if not commit_ref_head or commit_ref_head != commit_ref:
-                    self.add_output(
-                        self.indent_spaces
-                        + "[INFO] Git fetch origin reason: "
-                        f"Commit ref head '{commit_ref_head}' != "
-                        f"commit ref '{commit_ref_head}'"
-                        "\n")
-                    do_git_fetch = True
-
         if not do_git_fetch:
             self.add_output(self.indent_spaces + "[INFO] git fetch ignored\n")
             return False
@@ -379,6 +353,7 @@ class BatchFetchGit(TaskBatchFetch):
         return origin_url
 
     def _git_is_local_branch(self, branch: str) -> bool:
+        "Return True if it is a local branch that exists."
         try:
             stdout, _ = run_simple(["git", "rev-parse", "--symbolic-full-name",
                                     branch], env=self.env,
@@ -395,11 +370,11 @@ class BatchFetchGit(TaskBatchFetch):
 
         return False
 
-    def _git_tags(self, branch: str) -> List[str]:
+    def _git_rev_parse_verify(self, revision: str) -> List[str]:
         stdout: List[str] = []
         error = False
         try:
-            stdout, _ = run_simple(["git", "rev-parse", "--verify", branch],
+            stdout, _ = run_simple(["git", "rev-parse", "--verify", revision],
                                    env=self.env,
                                    cwd=self.git_local_dir)
             if not stdout:
@@ -409,7 +384,7 @@ class BatchFetchGit(TaskBatchFetch):
 
         if error:
             raise GitRevisionDoesNotExist(
-                f"The revision '{branch}' does not exist.")
+                f"The revision '{revision}' does not exist.")
 
         return stdout
 
@@ -429,13 +404,14 @@ class BatchFetchGit(TaskBatchFetch):
             # branch is a commit revision instead of a tag
             try:
                 # Check if the branch exists
-                git_ref_branch = self._git_tags("origin/"
-                                                + self["revision"]
-                                                + "^{commit}")[0]
+                git_ref_branch = self._git_rev_parse_verify("origin/"
+                                                            + self["revision"]
+                                                            + "^{commit}")[0]
             except GitRevisionDoesNotExist:
                 # Check if the commit ref exists
                 try:
-                    git_ref_branch = self._git_tags(self["revision"])[0]
+                    git_ref_branch = self._git_rev_parse_verify(self["revision"])[
+                        0]
                 except GitRevisionDoesNotExist as err:
                     raise BatchFetchError(f"The branch '{self['revision']}' "
                                           "does not exist.") from err

{batchfetch-1.2.6 → batchfetch-1.2.7}/batchfetch/helpers.py

@@ -112,7 +112,7 @@ def run_indent(cmd: Union[List[str], str], spaces: int = 4,
     cmd = shlex.split(cmd) if isinstance(cmd, str) else cmd
     stdout, stderr = run_simple(cmd=cmd, **kwargs)
     stdout = indent_raw_output([f"[RUN] {list2cmdline(cmd)}"], spaces) + \
-        indent_raw_output(stdout, spaces)
+        indent_raw_output(stdout, spaces + spaces)
     stderr = indent_raw_output(stderr, spaces)
 
     return (stdout, stderr)

{batchfetch-1.2.6 → batchfetch-1.2.7}/batchfetch.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: batchfetch
-Version: 1.2.6
+Version: 1.2.7
 Summary: Efficiently clone and pull multiple Git repositories.
 Home-page: https://github.com/jamescherti/batchfetch
 Author: James Cherti
@@ -22,27 +22,24 @@ License-File: LICENSE
 
 ## Introduction
 
-Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously.
+Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously. With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
 
-With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
+But why use Batchfetch? Because it is extremely fast, cloning repositories quickly by running Git operations in parallel. It intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories. Additionally, it allows specifying the revision (for Git), ensuring that the cloned repository matches the exact version you require.
 
 Batchfetch is ideal for quickly cloning or pulling multiple Git repositories. It is also useful for cloning various addons, such as Vim plugins, Emacs packages, Ansible roles, Ansible collections, and other addons available on websites like GitHub, Codeberg, and GitLab.
 
-## Features:
-- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
-- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
-- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
-- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
-
 ## Installation
 
+Here is how to install *batchfetch* using [pip](https://pypi.org/project/pip/):
 ```
 pip install --user batchfetch
 ```
 
-The pip command above will install the `batchfetch` executable in the directory `~/.local/bin/`.
+The pip command above installs the *batchfetch* executable in the `~/.local/bin/` directory. Omitting the `--user` flag will install it system-wide.
+
+## Usage
 
-## Example
+### Example of a `batchfetch.yaml` file
 
 Here is an example of a `batchfetch.yaml` file:
 
@@ -52,43 +49,55 @@ Here is an example of a `batchfetch.yaml` file:
 tasks:
   # Clone the default branch of the general.el repository to the
   # './general.el' directory
-  - git: https://github.com/noctuid/general.el
+  - git: https://github.com/jamescherti/compile-angel.el
 
   # Clone the tag 1.5 of the consult repository to the './consult'
   # directory
-  - git: https://github.com/minad/consult
-    revision: "1.5"
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
 
   # Clone the s.el repository to the './another-name.el' directory
-  - git: https://github.com/magnars/s.el
-    path: another-name.el
-    revision: dda84d38fffdaf0c9b12837b504b402af910d01d
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
 
-  # Delete './impatient-mode'
-  - git: https://github.com/skeeto/impatient-mode
+  # Delete the local copy of the following repository
+  - git: https://github.com/jamescherti/dir-config.el
     delete: true
 ```
 
 Execute the `batchfetch` command from the same directory as `batchfetch.yml` to make it clone or update the local copies of the repositories above.
 
-## Usage
+## Command-line options
 
 Here are the various options that `batchfetch` provides, along with descriptions of their usage:
 
 ```
-usage: batchfetch [--option] [args]
+usage: batchfetch [--option]
 
-Command line interface.
-
-positional arguments:
-  N                     Specify the batchfetch YAML file(s) (default: './batchfetch.yaml').
+Efficiently clone/pull multiple Git repositories in parallel.
 
 options:
   -h, --help            show this help message and exit
-  -j JOBS, --jobs JOBS  Run up to N Number of parallel processes (Default: 5).
+  -f FILE, --file FILE  Specify the batchfetch YAML file (default:
+                        './batchfetch.yaml').
+  -C DIRECTORY, --directory DIRECTORY
+                        Change the working directory before reading the
+                        batchfetch.yaml file. If not specified, the directory is
+                        set to the parent directory of the batchfetch.yaml file.
+  -j JOBS, --jobs JOBS  Run up to N parallel processes (default: 5).
+                        Alternatively, the BATCHFETCH_JOBS environment variable
+                        can be used to configure the number of jobs.
   -v, --verbose         Enable verbose mode.
 ```
 
+## Features
+- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
+- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
+- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
+- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
+- Detect files that should not be present in directories managed by batchfetch, known as untracked files.
+
 ## Frequently Asked Questions
 
 ### What are untracked files?
@@ -114,6 +123,43 @@ tasks:
 
 By default, *batchfetch.yaml* is the only untracked file that is ignored. The user does not need to add it to the *ignore_untracked_paths* option.
 
+### How is the Git local paths handled?
+
+When "path:" is specified, that's the path that is used.
+
+When "path:" is not specified, Batchfetch attempts to determine the path name by extracting the repository name from the URI (e.g., `https://domain.com/repo` becomes `repo`). If the URL ends with a `.git` extension, it removes the extension (e.g., `https://domain.com/repo.git` becomes `repo`).
+
+### How does Batchfetch detect when a git fetch is necessary?
+
+Batchfetch is fast, not only because it runs Git commands in parallel, but also because it intelligently detects whether a `git fetch` is needed, further speeding up the process of downloading data from repositories.
+
+When the user has specifies a revision (branch or commit reference), Batchfetch only performs a `git fetch` if that revision does not exist locally. If the revision is already up to date, it simply proceeds to the next repository in the queue.
+
+That's why it is highly recommended to always specify the revision to speed up Batchfetch, if speed is important to you. Here is an example of a `batchfetch.yaml` file where the branch (`1.1.0`) or commit reference (`b9c6d9b6134b4981760893254f804a371ffbc899`) is specified:
+``` yaml
+tasks:
+  - git: https://github.com/jamescherti/outline-indent.el
+    revision: "1.1.0"
+
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    revision: b9c6d9b6134b4981760893254f804a371ffbc899
+```
+
+### How to execute a command before and after a task?
+
+To execute a command both before and after a specific task, you can define the `exec_before` and `exec_after` directives within the task configuration. These directives specify commands to be executed at the respective stages of the task lifecycle.
+
+Here is an example:
+``` yaml
+---
+tasks:
+  - git: https://github.com/jamescherti/easysession.el
+    path: easysession
+    exec_before: ["sh", "-c", "echo exec_before_task"]
+    exec_after: ["sh", "-c", "echo exec_after_task"]
+```
+
 ## License
 
 Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)

{batchfetch-1.2.6 → batchfetch-1.2.7}/setup.py

@@ -22,7 +22,7 @@ from setuptools import find_packages, setup
 
 setup(
     name="batchfetch",
-    version="1.2.6",
+    version="1.2.7",
     packages=find_packages(),
     description="Efficiently clone and pull multiple Git repositories.",
     license="GPLv3",

batchfetch-1.2.6/README.md

@@ -1,110 +0,0 @@
-# Batchfetch - Efficiently clone or pull multiple Git repositories in parallel
-
-## Introduction
-
-Batchfetch is a command-line tool designed to clone, fetch, and merge multiple Git repositories simultaneously.
-
-With Batchfetch, you no longer need to manually manage each repository one by one. It automates the tedious aspects of repository management, freeing you up to focus on what truly matters: your workflow.
-
-Batchfetch is ideal for quickly cloning or pulling multiple Git repositories. It is also useful for cloning various addons, such as Vim plugins, Emacs packages, Ansible roles, Ansible collections, and other addons available on websites like GitHub, Codeberg, and GitLab.
-
-## Features:
-- Git Clone and Fetch/Merge: Clones the repositories and their submodules, ensuring that all the repositories are always up-to-date by fetching and merging changes.
-- Parallel Operations: Utilizes threads to simultaneously Git clone or pull multiple repositories, dramatically reducing wait times.
-- User-Friendly Interface: Provides simple and straightforward command-line options that make it easy to get started and effectively manage your repositories.
-- Custom Configuration: Allows the use of a YAML configuration file to specify and manage the repositories you interact with, enabling repeatable setups and consistent environments.
-
-## Installation
-
-```
-pip install --user batchfetch
-```
-
-The pip command above will install the `batchfetch` executable in the directory `~/.local/bin/`.
-
-## Example
-
-Here is an example of a `batchfetch.yaml` file:
-
-```yaml
----
-
-tasks:
-  # Clone the default branch of the general.el repository to the
-  # './general.el' directory
-  - git: https://github.com/noctuid/general.el
-
-  # Clone the tag 1.5 of the consult repository to the './consult'
-  # directory
-  - git: https://github.com/minad/consult
-    revision: "1.5"
-
-  # Clone the s.el repository to the './another-name.el' directory
-  - git: https://github.com/magnars/s.el
-    path: another-name.el
-    revision: dda84d38fffdaf0c9b12837b504b402af910d01d
-
-  # Delete './impatient-mode'
-  - git: https://github.com/skeeto/impatient-mode
-    delete: true
-```
-
-Execute the `batchfetch` command from the same directory as `batchfetch.yml` to make it clone or update the local copies of the repositories above.
-
-## Usage
-
-Here are the various options that `batchfetch` provides, along with descriptions of their usage:
-
-```
-usage: batchfetch [--option] [args]
-
-Command line interface.
-
-positional arguments:
-  N                     Specify the batchfetch YAML file(s) (default: './batchfetch.yaml').
-
-options:
-  -h, --help            show this help message and exit
-  -j JOBS, --jobs JOBS  Run up to N Number of parallel processes (Default: 5).
-  -v, --verbose         Enable verbose mode.
-```
-
-## Frequently Asked Questions
-
-### What are untracked files?
-
-The parent directory of the "path:" value defines the managed directory, where the directory of each path is considered as the managed directory.
-
-For example, if the "path:" value is `file/my-project`, the managed directory will be `file/`. Any file within `file/` that is not managed by batchfetch will be considered an untracked file.
-
-When *batchfetch* encounters an untracked file, it displays an error message to inform users about paths that are not managed by the system. The message provides clear instructions on how to handle these paths by adding them to the `options.ignore_untracked_paths` list, enabling users to manage untracked files effectively.
-
-Here is an example of a *batchfetch.yaml* file that enables *batchfetch* to accept a list of untracked files:
-
-``` yaml
-options:
-  ignore_untracked_paths:
-    - ./test
-    - /absolute/path
-    - ../relative/path
-
-tasks:
-  - git: https://github.com/user/project
-```
-
-By default, *batchfetch.yaml* is the only untracked file that is ignored. The user does not need to add it to the *ignore_untracked_paths* option.
-
-## License
-
-Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)
-
-This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
-
-This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with this program.
-
-## Links
-
-- [batchfetch @GitHub](https://github.com/jamescherti/batchfetch)
-- [batchfetch @Pypi](https://pypi.org/project/batchfetch/)