glpkg 1.1.0__tar.gz → 1.3.0__tar.gz

{glpkg-1.1.0/src/glpkg.egg-info → glpkg-1.3.0}/PKG-INFO

@@ -1,25 +1,19 @@
 Metadata-Version: 2.4
 Name: glpkg
-Version: 1.1.0
+Version: 1.3.0
 Summary: Tool to make GitLab generic package registry operations easy.
 Author-email: bugproduction <bugproduction@outlook.com>
 License-Expression: MIT
 Project-URL: Repository, https://gitlab.com/bugproduction/glpkg.git
 Keywords: GitLab,packages,registry,generic
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.md
-Provides-Extra: dev
-Requires-Dist: black; extra == "dev"
-Requires-Dist: build; extra == "dev"
-Requires-Dist: pytest; extra == "dev"
-Requires-Dist: pytest-cov; extra == "dev"
-Requires-Dist: setuptools; extra == "dev"
-Requires-Dist: twine; extra == "dev"
 Dynamic: license-file
 
-# glpkg - GitLab Generic Package registry tools 
+# glpkg - GitLab Generic Package tools 
 
-glpkg is a tool that makes it easy to work with [GitLab generic package registry](https://docs.gitlab.com/user/packages/generic_packages/).
+glpkg is a tool that makes it easy to work with [GitLab generic packages](https://docs.gitlab.com/user/packages/generic_packages/).
 
 
 ## Installation
@@ -47,7 +41,11 @@ By default, the used GitLab host is gitlab.com. If you use a self-hosted GitLab,
 
 To authenticate with the package registry in any of the commands below, use `--token readapitoken123` argument where the `readapitoken123` is a [personal](https://docs.gitlab.com/user/profile/personal_access_tokens/#create-a-personal-access-token) or [project](https://docs.gitlab.com/user/project/settings/project_access_tokens/#create-a-project-access-token) access token, with read API scope. In case the package registry is public, you can omit this argument.
 
-The above arguments are omitted in the examples below to focus on the functions. Add the arguments to change the host or to authenticate with the registry.
+Alternatively you can use a token stored in your `.netrc` file by setting `--netrc` argument.
+
+> If you use the tool in GitLab CI, read [below](#Use-in-GitLab-pipelines) on how to use the `CI_JOB_TOKEN`.
+
+The arguments related to the GitLab host or authentication (`--token`, `--netrc`, and `--ci`) are omitted in the examples below to focus on the commands.
 
 In general, run `glpkg --help` when needed.
 
@@ -74,7 +72,7 @@ mypackagename   2.0
 
 ### Download generic package
 
-To download everything from a specific generic package, run
+To download all files from a specific version of a generic package, run
 
 ```bash
 glpkg download --project 12345 --name mypackagename --version 1.0
@@ -85,7 +83,13 @@ Where:
 - `mypackagename` is the name of the generic package
 - `1.0` is the version of the generic package from which the files are downloaded
 
-The files will be downloaded in the current working directory. Any pre-existing files will be overridden without warning.
+By default the files will be downloaded in the current working directory. To download the files to another directory, add argument `--destination` to the command. In all cases, as long as you have permissions to the destination directory, any pre-existing files will be overridden without warning.
+
+To download only a specific file from the package, add `--file` argument.
+
+```bash
+glpkg download --project 12345 --name mypackagename --version 1.5 --file the_only_one --destination /temp
+```
 
 > If a package has multiple files with the same filename, the tool can only download the newest file. This is a restriction of GitLab API.
 
@@ -101,13 +105,21 @@ Where:
 - `12345` is your projects ID ([Find the Project ID](https://docs.gitlab.com/user/project/working_with_projects/#find-the-project-id)) or the path of the project (like `namespace/project`)
 - `mypackagename` is the name of the generic package
 - `1.0` is the version of the generic package to which the file is uploaded
-- `my-file.txt` is the file that is uploaded to the generic package. Currently, only relative paths are supported, and the relative path (e.g. `folder/file.txt`) is preserved when uploading the file to the registry.
+- `my-file.txt` is the file that is uploaded to the generic package. Only relative paths are supported, and the relative path (e.g. `folder/file.txt`) is preserved when uploading the file to the package.
 
 > A GitLab generic package may have multiple files with the same file name. However, it likely is not a great idea, as they cannot be downloaded separately from the GitLab API.
 
+To upload multiple files, or to upload a single file from a different directory, use `--source` argument. If no `--file` argument is set, all of the files in the source directory are uploaded, recursively. As an example, to upload all files from a `upload` folder to the package:
+
+```bash
+glpkg upload --project 12345 --name mypackagename --version 1.0 --source upload
+```
+
+Note: a file in `upload` folder will be uploaded to the root of the package. If you want to upload the file to a `dir` folder in the package, make a structure in the upload folder, like `upload/dir/`.
+
 ### Use in GitLab pipelines
 
-If you use the tool in a GitLab pipeline, using `--ci` argument uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID and to use a personal or project access token instead of CI_JOB_TOKEN.
+If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID or to use a personal or project access token instead of `CI_JOB_TOKEN`.
 
 In other words, you don't need to give the `--host`, `--project`, or `--token` arguments if you are interacting with the package registry of the project where the pipeline is running. Example: uploading `my-file.txt` to generic package `mypackagename` version `1.0` in the project package registry in CI:
 
@@ -122,5 +134,4 @@ To use the `CI_JOB_TOKEN` with package registry of another projects, add `--proj
 
 The tool is not perfect (yet) and has limitations. The following limitations are known, but more can exist:
 
-- Uploading files must be done one-by-one.
 - Only project registries are supported for now.

{glpkg-1.1.0 → glpkg-1.3.0}/README.md

@@ -1,6 +1,6 @@
-# glpkg - GitLab Generic Package registry tools 
+# glpkg - GitLab Generic Package tools 
 
-glpkg is a tool that makes it easy to work with [GitLab generic package registry](https://docs.gitlab.com/user/packages/generic_packages/).
+glpkg is a tool that makes it easy to work with [GitLab generic packages](https://docs.gitlab.com/user/packages/generic_packages/).
 
 
 ## Installation
@@ -28,7 +28,11 @@ By default, the used GitLab host is gitlab.com. If you use a self-hosted GitLab,
 
 To authenticate with the package registry in any of the commands below, use `--token readapitoken123` argument where the `readapitoken123` is a [personal](https://docs.gitlab.com/user/profile/personal_access_tokens/#create-a-personal-access-token) or [project](https://docs.gitlab.com/user/project/settings/project_access_tokens/#create-a-project-access-token) access token, with read API scope. In case the package registry is public, you can omit this argument.
 
-The above arguments are omitted in the examples below to focus on the functions. Add the arguments to change the host or to authenticate with the registry.
+Alternatively you can use a token stored in your `.netrc` file by setting `--netrc` argument.
+
+> If you use the tool in GitLab CI, read [below](#Use-in-GitLab-pipelines) on how to use the `CI_JOB_TOKEN`.
+
+The arguments related to the GitLab host or authentication (`--token`, `--netrc`, and `--ci`) are omitted in the examples below to focus on the commands.
 
 In general, run `glpkg --help` when needed.
 
@@ -55,7 +59,7 @@ mypackagename   2.0
 
 ### Download generic package
 
-To download everything from a specific generic package, run
+To download all files from a specific version of a generic package, run
 
 ```bash
 glpkg download --project 12345 --name mypackagename --version 1.0
@@ -66,7 +70,13 @@ Where:
 - `mypackagename` is the name of the generic package
 - `1.0` is the version of the generic package from which the files are downloaded
 
-The files will be downloaded in the current working directory. Any pre-existing files will be overridden without warning.
+By default the files will be downloaded in the current working directory. To download the files to another directory, add argument `--destination` to the command. In all cases, as long as you have permissions to the destination directory, any pre-existing files will be overridden without warning.
+
+To download only a specific file from the package, add `--file` argument.
+
+```bash
+glpkg download --project 12345 --name mypackagename --version 1.5 --file the_only_one --destination /temp
+```
 
 > If a package has multiple files with the same filename, the tool can only download the newest file. This is a restriction of GitLab API.
 
@@ -82,13 +92,21 @@ Where:
 - `12345` is your projects ID ([Find the Project ID](https://docs.gitlab.com/user/project/working_with_projects/#find-the-project-id)) or the path of the project (like `namespace/project`)
 - `mypackagename` is the name of the generic package
 - `1.0` is the version of the generic package to which the file is uploaded
-- `my-file.txt` is the file that is uploaded to the generic package. Currently, only relative paths are supported, and the relative path (e.g. `folder/file.txt`) is preserved when uploading the file to the registry.
+- `my-file.txt` is the file that is uploaded to the generic package. Only relative paths are supported, and the relative path (e.g. `folder/file.txt`) is preserved when uploading the file to the package.
 
 > A GitLab generic package may have multiple files with the same file name. However, it likely is not a great idea, as they cannot be downloaded separately from the GitLab API.
 
+To upload multiple files, or to upload a single file from a different directory, use `--source` argument. If no `--file` argument is set, all of the files in the source directory are uploaded, recursively. As an example, to upload all files from a `upload` folder to the package:
+
+```bash
+glpkg upload --project 12345 --name mypackagename --version 1.0 --source upload
+```
+
+Note: a file in `upload` folder will be uploaded to the root of the package. If you want to upload the file to a `dir` folder in the package, make a structure in the upload folder, like `upload/dir/`.
+
 ### Use in GitLab pipelines
 
-If you use the tool in a GitLab pipeline, using `--ci` argument uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID and to use a personal or project access token instead of CI_JOB_TOKEN.
+If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID or to use a personal or project access token instead of `CI_JOB_TOKEN`.
 
 In other words, you don't need to give the `--host`, `--project`, or `--token` arguments if you are interacting with the package registry of the project where the pipeline is running. Example: uploading `my-file.txt` to generic package `mypackagename` version `1.0` in the project package registry in CI:
 
@@ -103,5 +121,4 @@ To use the `CI_JOB_TOKEN` with package registry of another projects, add `--proj
 
 The tool is not perfect (yet) and has limitations. The following limitations are known, but more can exist:
 
-- Uploading files must be done one-by-one.
 - Only project registries are supported for now.

{glpkg-1.1.0 → glpkg-1.3.0}/pyproject.toml

@@ -12,21 +12,12 @@ license-files = ["LICENSE.md"]
 authors = [
   { name="bugproduction", email="bugproduction@outlook.com" }
 ]
+requires-python = ">= 3.9"
 dependencies = []
 
 [project.urls]
 Repository = "https://gitlab.com/bugproduction/glpkg.git"
 
-[project.optional-dependencies]
-dev = [
-    "black",
-    "build",
-    "pytest",
-    "pytest-cov",
-    "setuptools",
-    "twine"
-]
-
 [project.scripts]
 glpkg = "gitlab.__main__:cli"
 
@@ -40,3 +31,9 @@ where = ["src"]
 [tool.pytest.ini_options]
 pythonpath = ["src"]
 addopts = "--cov=src --cov-report term"
+
+[tool.pylint.'MESSAGES CONTROL']
+min-public-methods = 1
+max-args = 6
+max-positional-arguments = 6
+ignore = ["build", "test"]

glpkg-1.3.0/src/gitlab/__init__.py

@@ -0,0 +1,5 @@
+"""GitLab packages initialization module"""
+
+from gitlab.packages import Packages
+
+__version__ = "1.3.0"

glpkg-1.3.0/src/gitlab/__main__.py

@@ -0,0 +1,29 @@
+"""GitLab packages main module"""
+
+import logging
+import sys
+
+from gitlab.cli_handler import CLIHandler
+
+
+def cli() -> int:
+    """
+    Runs the main program of the glpkg.
+
+    Uses arguments from command line and executes the given command.
+
+    Return
+    ------
+    int
+        Zero when everything is fine, non-zero otherwise.
+    """
+    logging.basicConfig(
+        level=logging.DEBUG,
+        handlers=[logging.FileHandler("glpkg.log")],  # , logging.StreamHandler()],
+    )
+    handler = CLIHandler()
+    return handler.do_it()
+
+
+if __name__ == "__main__":
+    sys.exit(cli())

glpkg-1.3.0/src/gitlab/cli_handler.py

@@ -0,0 +1,361 @@
+"""CLI handler for glpkg"""
+
+import argparse
+from netrc import netrc
+import os
+import sys
+import urllib
+from gitlab import Packages, __version__
+
+
+class CLIHandler:
+    """Class to parse CLI arguments and run the requested command"""
+
+    def __init__(self):
+        """
+        Creates a new instance of CLIHandler.
+
+        Parses the arguments from command line and prepares everything
+        to be ready to executed. Just use do_it to run it!
+        """
+        parser = argparse.ArgumentParser(
+            description="Toolbox for GitLab generic packages"
+        )
+        parser.add_argument("-v", "--version", action="store_true")
+        parser.set_defaults(action=self._print_version)
+        subparsers = parser.add_subparsers()
+        list_parser = subparsers.add_parser(
+            name="list",
+            description="Lists the available version of a package from the "
+            "package registry.",
+        )
+        self._register_list_parser(list_parser)
+        download_parser = subparsers.add_parser(
+            name="download",
+            description="Downloads all files from a specific package version "
+            "to the current directory.",
+        )
+        self._register_download_parser(download_parser)
+        upload_parser = subparsers.add_parser(
+            name="upload", description="Uploads file to a specific package version."
+        )
+        self._register_upload_parser(upload_parser)
+        self.args = parser.parse_args()
+
+    def _print_version(self, _args: argparse.Namespace) -> int:
+        """
+        A handler for printing the version of the tool to the console.
+
+        Parameters
+        ----------
+        _args : argparse.Namespace
+            Unused.
+
+        Return
+        ------
+        int
+            Zero when printing to console succeeded.
+        """
+        print(__version__)
+        return 0
+
+    def do_it(self) -> int:
+        """
+        Executes the requested command.
+
+        In case of error, prints to stderr.
+
+        Return
+        ------
+        int
+            Zero when everything went fine, non-zero otherwise.
+        """
+        ret = 1
+        try:
+            ret = self.args.action(self.args)
+        except urllib.error.HTTPError as e:
+            # GitLab API returns 404 when a resource is not found
+            # but also when the user has no access to the resource
+            print("Oops! Something did go wrong.", file=sys.stderr)
+            print(e, file=sys.stderr)
+            print(
+                "Note that Error 404 may also indicate authentication issues with GitLab API.",
+                file=sys.stderr,
+            )
+            print("Check your arguments and credentials.", file=sys.stderr)
+        return ret
+
+    def _register_common_arguments(self, parser: argparse.ArgumentParser) -> None:
+        """
+        Registers common arguments to the parser:
+            - host
+            - ci
+            - project
+            - name
+            - token
+            - netrc
+
+        Parameters
+        ----------
+        parser:
+            The argparser where to register the common arguments.
+        """
+        group = parser.add_mutually_exclusive_group()
+        group.add_argument(
+            "-H",
+            "--host",
+            default="gitlab.com",
+            type=str,
+            help="The host address of GitLab instance without scheme, "
+            "for example gitlab.com. Note that only https scheme is supported.",
+        )
+        group.add_argument(
+            "-c",
+            "--ci",
+            action="store_true",
+            help="Use this in GitLab jobs. In this case CI_SERVER_HOST, CI_PROJECT_ID, "
+            "and CI_JOB_TOKEN variables from the environment are used. --project and --token "
+            "can be used to override project ID and the CI_JOB_TOKEN to a personal or project "
+            "access token.",
+        )
+        parser.add_argument(
+            "-p",
+            "--project",
+            type=str,
+            help="The project ID or path. For example 123456 or namespace/project.",
+        )
+        parser.add_argument("-n", "--name", type=str, help="The package name.")
+        group2 = parser.add_mutually_exclusive_group()
+        group2.add_argument(
+            "-t",
+            "--token",
+            type=str,
+            help="Private or project access token that is used to authenticate with "
+            "the package registry. Leave empty if the registry is public. The token "
+            "must have 'read API' or 'API' scope.",
+        )
+        group2.add_argument(
+            "--netrc",
+            action="store_true",
+            help="Set to use a token from .netrc file (~/.netrc) for the host. The "
+            ".netrc username is ignored due to API restrictions. PRIVATE-TOKEN is used "
+            "instead. Note that .netrc file access rights must be correct.",
+        )
+
+    def _register_download_parser(self, parser: argparse.ArgumentParser):
+        """
+        Registers the download command related arguments to the parser:
+            - version
+            - file
+            - destination
+            - action
+
+        Additionally, registers the common args.
+
+        Parameters
+        ----------
+        parser:
+            The argparser where to register the download arguments.
+        """
+        self._register_common_arguments(parser)
+        parser.add_argument("-v", "--version", type=str, help="The package version.")
+        parser.add_argument(
+            "-f",
+            "--file",
+            type=str,
+            help="The file to download from the package. If not defined, all "
+            "files are downloaded.",
+        )
+        parser.add_argument(
+            "-d",
+            "--destination",
+            default="",
+            type=str,
+            help="The path where the file(s) are downloaded. If not defined, "
+            "the current working directory is used.",
+        )
+        parser.set_defaults(action=self._download_handler)
+
+    def _args(self, args) -> tuple[str, str, str, str, str]:
+        """
+        Returns the connection parameters according to the args
+
+        Parameters
+        ----------
+        args:
+            The args that are used to determined the connection parameters
+
+        Returns
+        -------
+        host : str
+            The GitLab host name
+        project : str
+            The Project ID or name to use
+        name : str
+            The package name
+        token_user : str
+            The token user according to the args. If ci is used, returns `JOB-TOKEN`, else
+            `PRIVATE-TOKEN`.
+        token : str
+            The token according to the args. If token is set, returns it. If netrc is set,
+            reads the token from the .netrc file. If ci is set, reads the environment
+            variable CI_JOB_TOKEN. Otherwise returns None.
+        """
+        if args.ci:
+            host = os.environ["CI_SERVER_HOST"]
+            project = os.environ["CI_PROJECT_ID"]
+            token = os.environ["CI_JOB_TOKEN"]
+            token_user = "JOB-TOKEN"
+            if args.project:
+                project = args.project
+            if args.token:
+                token = args.token
+                token_user = "PRIVATE-TOKEN"
+        else:
+            host = args.host
+            project = args.project
+            token = args.token
+            token_user = "PRIVATE-TOKEN"
+        if args.netrc:
+            _, _, token = netrc().authenticators(host)
+            token_user = "PRIVATE-TOKEN"
+        name = args.name
+        return host, project, name, token_user, token
+
+    def _download_handler(self, args: argparse.Namespace) -> int:
+        """
+        Downloads package file(s) from GitLab package registry.
+
+        Parameters
+        ----------
+        args : argparse.Namespace
+            The parsed arguments
+
+        Returns
+        -------
+        int
+            Zero if everything goes well, non-zero otherwise
+        """
+        ret = 1
+        host, project, name, token_user, token = self._args(args)
+        version = args.version
+        destination = args.destination
+        gitlab = Packages(host, token_user, token)
+        package_id = gitlab.get_package_id(project, name, version)
+        if package_id:
+            files = []
+            if args.file:
+                files.append(args.file)
+            else:
+                files = gitlab.list_files(project, package_id)
+            for file in files:
+                ret = gitlab.download_file(project, name, version, file, destination)
+                if ret:
+                    print("Failed to download file " + file)
+                    break
+        else:
+            print("No package " + name + " version " + version + " found!")
+        return ret
+
+    def _register_list_parser(self, parser: argparse.ArgumentParser):
+        """
+        Registers the list command related arguments to the parser:
+            - action
+
+        Additionally, registers the common args.
+
+        Parameters
+        ----------
+        parser:
+            The argparser where to register the list arguments.
+        """
+        self._register_common_arguments(parser)
+        parser.set_defaults(action=self._list_packages)
+
+    def _list_packages(self, args: argparse.Namespace) -> int:
+        """
+        List package versions from GitLab package registry.
+
+        Parameters
+        ----------
+        args : argparse.Namespace
+            The parsed arguments
+
+        Returns
+        -------
+        int
+            Zero if everything goes well, non-zero otherwise
+        """
+        host, project, name, token_user, token = self._args(args)
+        gitlab = Packages(host, token_user, token)
+        packages = gitlab.list_packages(project, name)
+        print("Name" + "\t\t" + "Version")
+        for package in packages:
+            print(package["name"] + "\t" + package["version"])
+
+    def _register_upload_parser(self, parser: argparse.ArgumentParser):
+        """
+        Registers the upload command related arguments to the parser:
+            - version
+            - file
+            - action
+
+        Additionally, registers the common args.
+
+        Parameters
+        ----------
+        parser:
+            The argparser where to register the upload arguments.
+        """
+        self._register_common_arguments(parser)
+        parser.add_argument("-v", "--version", type=str, help="The package version.")
+        parser.add_argument(
+            "-f",
+            "--file",
+            type=str,
+            help="The file to be uploaded, for example my_file.txt. Note that "
+            "only relative paths (to the source) are supported and the relative "
+            "path is preserved when uploading the file. If left undefined, all files "
+            "of the source directory are uploaded. For example --source=temp --file=myfile "
+            "will upload myfile to the GitLab generic package root. However using --source=. "
+            "(or omittinge source) --file=temp/myfile will upload the file to temp folder "
+            "in the GitLab package.",
+        )
+        parser.add_argument(
+            "-s",
+            "--source",
+            type=str,
+            default="",
+            help="The source directory of the uploaded file(s). Defaults to current"
+            "working directory.",
+        )
+        parser.set_defaults(action=self._upload)
+
+    def _upload(self, args: argparse.Namespace) -> int:
+        """
+        Uploads a file to a GitLab package registry
+
+        Parameters
+        ----------
+        args : argparse.Namespace
+            The arguments from command line
+
+        Returns
+        -------
+        int
+            Zero if everything went fine, non-zero otherwise.
+        """
+        ret = 0
+        host, project, name, token_user, token = self._args(args)
+        version = args.version
+        file = args.file
+        source = args.source
+        if file:
+            # Check if the uploaded file exists.
+            if not os.path.isfile(os.path.join(source, file)):
+                print("File " + file + " does not exist!")
+                ret = 1
+        if not ret:
+            gitlab = Packages(host, token_user, token)
+            ret = gitlab.upload_file(project, name, version, file, source)
+        return ret