glpkg 1.4.2__tar.gz → 1.5.0__tar.gz

{glpkg-1.4.2 → glpkg-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: glpkg
-Version: 1.4.2
+Version: 1.5.0
 Summary: Tool to make GitLab generic package registry operations easy.
 Author-email: bugproduction <bugproduction@outlook.com>
 License-Expression: MIT
@@ -101,6 +101,29 @@ glpkg download --project 12345 --name mypackagename --version 1.5 --file the_onl
 
 > If a package has multiple files with the same filename, the tool can only download the newest file. This is a restriction of GitLab API.
 
+#### Downloading a list of packages
+
+To download multiple packages or versions from one or more projects, a package file can be used together with `--from-file` argument with `download`:
+
+```bash
+glpkg download --from-file my-packages.txt --destination my-downloads
+```
+
+The `my-packages.txt` lists all wanted packages with their versions and projects in format `<package-name>==<package-version>@<project-id>`.For example `my-packages.txt` could look like this:
+
+```
+mypackage==2.3.6-beta@namespace/project
+yourpackage==1.2.3@12345
+```
+
+The `<project-id>` can be either the path to the project in string format or the project ID in numeric.
+
+When `--from-file` argument is used, `--file`, `--project`, `--version`, and `--name` arguments are unused.
+
+> The `--destination` argument can be used to download the files from the packages to a different folder. In case packages contains files with same names, some files will be overwritten.
+
+> Whatever credentials (`--token`, `--ci`, or `--netrc`) you run the download command with will be used for all package downloads in the list. Make sure that the credential has access to all projects.
+
 ### Upload a file to a generic package
 
 To upload a file to a version of a generic package, run
@@ -158,7 +181,7 @@ The token that is used to delete packages must have at least Maintainer role in
 
 ### Use in GitLab pipelines
 
-If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID or to use a personal or project access token instead of `CI_JOB_TOKEN`.
+If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--host`, `--project`, and `--token` arguments can still be used to override the host, project ID, or to use a personal or project access token instead of `CI_JOB_TOKEN`.
 
 In other words, you don't need to give the `--host`, `--project`, or `--token` arguments if you are interacting with the package registry of the project where the pipeline is running. Example: uploading `my-file.txt` to generic package `mypackagename` version `1.0` in the project package registry in CI:
 

{glpkg-1.4.2 → glpkg-1.5.0}/README.md

@@ -88,6 +88,29 @@ glpkg download --project 12345 --name mypackagename --version 1.5 --file the_onl
 
 > If a package has multiple files with the same filename, the tool can only download the newest file. This is a restriction of GitLab API.
 
+#### Downloading a list of packages
+
+To download multiple packages or versions from one or more projects, a package file can be used together with `--from-file` argument with `download`:
+
+```bash
+glpkg download --from-file my-packages.txt --destination my-downloads
+```
+
+The `my-packages.txt` lists all wanted packages with their versions and projects in format `<package-name>==<package-version>@<project-id>`.For example `my-packages.txt` could look like this:
+
+```
+mypackage==2.3.6-beta@namespace/project
+yourpackage==1.2.3@12345
+```
+
+The `<project-id>` can be either the path to the project in string format or the project ID in numeric.
+
+When `--from-file` argument is used, `--file`, `--project`, `--version`, and `--name` arguments are unused.
+
+> The `--destination` argument can be used to download the files from the packages to a different folder. In case packages contains files with same names, some files will be overwritten.
+
+> Whatever credentials (`--token`, `--ci`, or `--netrc`) you run the download command with will be used for all package downloads in the list. Make sure that the credential has access to all projects.
+
 ### Upload a file to a generic package
 
 To upload a file to a version of a generic package, run
@@ -145,7 +168,7 @@ The token that is used to delete packages must have at least Maintainer role in
 
 ### Use in GitLab pipelines
 
-If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--project`, and `--token` arguments can still be used to override the project ID or to use a personal or project access token instead of `CI_JOB_TOKEN`.
+If you use the tool in a GitLab pipeline, setting argument `--ci` uses [GitLab predefined variables](https://docs.gitlab.com/ci/variables/predefined_variables/) to configure the tool. In this case `CI_SERVER_HOST`, `CI_PROJECT_ID`, and `CI_JOB_TOKEN` environment variables are used. The `--host`, `--project`, and `--token` arguments can still be used to override the host, project ID, or to use a personal or project access token instead of `CI_JOB_TOKEN`.
 
 In other words, you don't need to give the `--host`, `--project`, or `--token` arguments if you are interacting with the package registry of the project where the pipeline is running. Example: uploading `my-file.txt` to generic package `mypackagename` version `1.0` in the project package registry in CI:
 

{glpkg-1.4.2 → glpkg-1.5.0}/src/gitlab/__init__.py

@@ -2,4 +2,4 @@
 
 from gitlab.packages import Packages
 
-__version__ = "1.4.2"
+__version__ = "1.5.0"

{glpkg-1.4.2 → glpkg-1.5.0}/src/gitlab/cli_handler.py

@@ -3,6 +3,7 @@
 import argparse
 from netrc import netrc
 import os
+import re
 import sys
 import urllib
 from gitlab import Packages, __version__
@@ -72,6 +73,9 @@ class CLIHandler:
                 file=sys.stderr,
             )
             print("Check your arguments and credentials.", file=sys.stderr)
+        except ValueError as e:
+            print("Whopsie! Something did go wrong.", file=sys.stderr)
+            print(e, file=sys.stderr)
         return ret
 
     def _register_common_arguments(self, parser: argparse.ArgumentParser) -> None:
@@ -89,7 +93,7 @@ class CLIHandler:
         parser:
             The argparser where to register the common arguments.
         """
-        group = parser.add_mutually_exclusive_group()
+        group = parser.add_argument_group()
         group.add_argument(
             "-H",
             "--host",
@@ -148,6 +152,12 @@ class CLIHandler:
         """
         self._register_common_arguments(parser)
         parser.add_argument("-v", "--version", type=str, help="The package version.")
+        parser.add_argument(
+            "--from-file",
+            type=str,
+            help="Download all files from packages listed in the file. Each line in the "
+            "file should follow format <package-name>==<package-version>@<project>.",
+        )
         parser.add_argument(
             "-f",
             "--file",
@@ -195,6 +205,8 @@ class CLIHandler:
             project = os.environ["CI_PROJECT_ID"]
             token = os.environ["CI_JOB_TOKEN"]
             token_user = "JOB-TOKEN"
+            if args.host:
+                host = args.host
             if args.project:
                 project = args.project
             if args.token:
@@ -229,23 +241,67 @@ class CLIHandler:
         host, project, name, token_user, token = self._args(args)
         version = args.version
         destination = args.destination
+        file = args.file
+        from_file = args.from_file
         packages = Packages(host, token_user, token)
-        package_id = packages.get_id(project, name, version)
-        if package_id:
-            files = []
-            if args.file:
-                files.append(args.file)
-            else:
-                files = packages.get_files(project, package_id).keys()
-            for file in files:
-                ret = packages.download_file(project, name, version, file, destination)
+        if from_file:
+            package_list = self._read_from_file(from_file)
+            for package in package_list:
+                ret = packages.download(
+                    package["project"], package["name"], package["version"], destination
+                )
                 if ret:
-                    print("Failed to download file " + file)
                     break
+        elif file:
+            ret = packages.download_file(project, name, version, file, destination)
         else:
-            print("No package " + name + " version " + version + " found!")
+            ret = packages.download(project, name, version, destination)
+        if ret:
+            if ret == 2:
+                print("No package " + name + " version " + version + " found!")
+            else:
+                print("Failed to download file(s)!")
         return ret
 
+    def _read_from_file(self, filepath: str) -> list[dict]:
+        """
+        Reads packages file and returns a list of packages to be downloaded.
+
+        The lines in the file should be
+        <packagename>==<version>@<project>
+
+        Parameters
+        ----------
+        filepath : str
+            Filepath of the packages file
+
+        Returns
+        -------
+        int
+            Zero if everything goes well, non-zero otherwise
+        """
+        packages = []
+        pattern = (
+            r"\s*(?P<packagename>\S+)\s*=="
+            r"\s*(?P<packageversion>\S+)\s*@"
+            r"\s*(?P<project>\S+)\s*"
+        )
+        with open(filepath, "r", encoding="utf-8") as file:
+            for line in file:
+                if not line:
+                    continue
+                match = re.match(pattern, line)
+                if not match:
+                    raise ValueError(
+                        f"Package file {filepath} line {line} seems fishy! \
+                            Check that the line is <packagename>==<version>@<project>"
+                    )
+                name = match.group("packagename")
+                version = match.group("packageversion")
+                project = match.group("project")
+                packages.append({"name": name, "version": version, "project": project})
+        return packages
+
     def _register_list_parser(self, parser: argparse.ArgumentParser):
         """
         Registers the list command related arguments to the parser:
@@ -334,19 +390,21 @@ class CLIHandler:
         int
             Zero if everything went fine, non-zero otherwise.
         """
-        ret = 0
+        ret = 1
         host, project, name, token_user, token = self._args(args)
         version = args.version
         file = args.file
         source = args.source
+        packages = Packages(host, token_user, token)
         if file:
             # Check if the uploaded file exists.
             if not os.path.isfile(os.path.join(source, file)):
                 print("File " + file + " does not exist!")
                 ret = 1
-        if not ret:
-            packages = Packages(host, token_user, token)
-            ret = packages.upload_file(project, name, version, file, source)
+            else:
+                ret = packages.upload_file(project, name, version, file, source)
+        else:
+            ret = packages.upload(project, name, version, source)
         return ret
 
     def _register_delete_parser(self, parser: argparse.ArgumentParser):
@@ -397,5 +455,5 @@ class CLIHandler:
         if file:
             ret = packages.delete_file(project, name, version, file)
         else:
-            ret = packages.delete_package(project, name, version)
+            ret = packages.delete(project, name, version)
         return ret

glpkg-1.5.0/src/gitlab/lib/gitlab_api.py

@@ -0,0 +1,248 @@
+"""GitLab basic API"""
+
+from http.client import HTTPMessage
+from json import loads
+import logging
+from urllib import parse
+from gitlab.lib import http
+
+logger = logging.getLogger(__name__)
+
+
+class GitLabAPI:
+    """Class for basic GitLab API"""
+
+    def __init__(self, host: str, token_type: str, token: str):
+        """
+        Creates a new instance of class.
+
+        Parameters
+        ----------
+        host : str
+            The GitLab instance hostname, without schema.
+            The host will be used for the package API interaction.
+            For example gitlab.com.
+        token_type : str
+            The token type or "user" to authenticate with GitLab REST API.
+            For personal, project, and group tokens this is `PRIVATE-TOKEN`.
+            For `CI_JOB_TOKEN` this is `JOB-TOKEN`.
+            Can be left empty when authentication is not used.
+        token : str
+            The token (secret) to authenticate with GitLab REST API.
+            This can be a personal token, project token, or`CI_JOB_TOKEN`.
+            Leave empty when authentication is not used.
+        """
+        self.host = host
+        self.token_type = token_type
+        self.token = token
+
+    def _url(self) -> str:
+        """
+        Returns the GitLab REST API URL by using the host variable.
+
+        Returns
+        -------
+        str
+            The GitLab REST API URL, for example `https://gitlab.com/api/v4/`.
+        """
+        return f"https://{self.host}/api/v4/"
+
+    def _get_headers(self) -> dict:
+        """
+        Creates headers for a GitLab REST API call.
+
+        The headers contain token for authentication according to the
+        instance variables.
+
+        Returns
+        -------
+        dict
+            Headers for a REST API request, that contain the authentication token.
+        """
+        headers = {}
+        if self.token_type and self.token:
+            headers = {self.token_type: self.token}
+        return headers
+
+    def _parse_header_links(self, headers: HTTPMessage) -> dict:
+        """
+        Parses link field from HTTP headers to a dictionary where
+        the link "rel" value is the key.
+
+        This is useful for example with GitLab REST API to get the pagination links.
+
+        Parameters
+        ----------
+        headers : HTTPMessage
+            The HTTP response headers that contain the links
+
+        Returns
+        -------
+        dict
+            The header links in a dictionary, that can be used to for example pagination:
+            _parse_header_links(headers).get("next") returns the next page link, or None.
+        """
+        links = {}
+        header_links = headers.get("link")
+        if header_links:
+            items = header_links.split(",")
+            # Values are <uri-reference>; param1=value1; param2="value2"
+            for item in items:
+                parts = item.split(";", 1)
+                if parts:
+                    # First value should be the URI
+                    val = parts.pop(0).lstrip(" <").rstrip("> ")
+                    # The rest are the parameters; let's take the first one that has rel=
+                    # in it, split it with ; and take the first part as a value
+                    typ = parts.pop()
+                    typ = typ.split("rel=", 1).pop().split(";").pop(0).strip('" ')
+                    links[typ] = val
+        return links
+
+    def _build_query(self, **args: str) -> str:
+        """
+        Builds a query for a GitLab REST API request
+
+        Parameters
+        ----------
+        args : str
+            keyword arguments for the REST API query, like per_page=20, page=20
+
+        Returns
+        -------
+        str
+            A query string for a REST API request. Append this to
+            the request URL. Example `?per_page=20&page=20`.
+        """
+        query = ""
+        for key, value in args.items():
+            query = "&".join(filter(None, (query, f"{key}={parse.quote_plus(value)}")))
+        if query:
+            query = "?" + query
+        return query
+
+    def _build_path(self, *paths: str) -> str:
+        """
+        Returns the path to be appended to the REST API URL
+
+        Parameters
+        ----------
+        paths : str
+            The path string that are joined with "/".
+
+        Returns
+        -------
+        str
+            A URL path, for example `projects/123/`
+            or `projects/namespace%2Fproject/`
+        """
+        quoted_paths = []
+        for subpath in paths:
+            quoted_paths.append(subpath)
+        return "/".join(quoted_paths)
+
+    def get(self, *paths: str, **query_params: str) -> tuple[int, bytes, HTTPMessage]:
+        """
+        Makes a HTTP GET request to the given GitLab path, and returns
+        the response status, body, and headers.
+
+        Parameters
+        ----------
+        paths : str
+            The URL path of the HTTP request to make.
+        query_params : str,  optional
+            Dictionary of query parameters for the request, like package_name=mypackage
+
+        Returns
+        -------
+        int
+            The HTTP response code, such as 200
+        bytes
+            The HTTP response body read as bytes
+        HTTPMessage
+            The HTTP response headers
+        """
+        url = self._url() + self._build_path(*paths) + self._build_query(**query_params)
+        headers = self._get_headers()
+        return http.get(url, headers)
+
+    def get_all(self, *paths: str, **query_params: str) -> list:
+        """
+        Returns data from the REST API endpoint. In case
+        of multiple pages, all data will be returned.
+
+        Parameters
+        ----------
+        paths : str
+            The paths of the API endpoint that is called. For example projects, 123, packages
+            would be querying the projects/123/packages endpoint.
+        query_params : str, optional
+            Additional arguments for the query of the URL, for example to filter
+            results: package_name=mypackage
+
+        Returns
+        -------
+        list
+            Data from GitLab REST API endpoint with the arguments.
+        """
+        url = self._url() + self._build_path(*paths) + self._build_query(**query_params)
+        data = []
+        while url:
+            res_status, res_data, res_headers = http.get(url, self._get_headers())
+            logger.debug("Response status: %d", res_status)
+            res_data = loads(res_data)
+            logger.debug("Response data: %s", res_data)
+            data = data + res_data
+            url = self._parse_header_links(res_headers).get("next")
+        return data
+
+    def put(self, data: bytes, *paths: str, **query_params: str) -> int:
+        """
+        Makes a HTTP PUT request to the given GitLab path.
+
+        Parameters
+        ----------
+        data : bytes
+            The data to PUT.
+        paths : str
+            The URL path of the HTTP request to make.
+        query_params : str, optional
+            Additional arguments for the query of the URL, for example to filter
+            results: package_name=mypackage
+
+        Returns
+        -------
+        int
+            0 If the request was successful.
+        """
+        ret = 1
+        url = self._url() + self._build_path(*paths) + self._build_query(**query_params)
+        status, _, _ = http.put(url, data, self._get_headers())
+        if status == 201:  # 201 is created
+            ret = 0
+        return ret
+
+    def delete(self, *paths: str, **query_params: str) -> int:
+        """
+        Makes a HTTP DELETE request to the given GitLab path.
+
+        Parameters
+        ----------
+        paths : str
+            The URL path of the HTTP request to make.
+        query_params : str, optional
+            Additional arguments for the query of the URL, for example to filter
+            results: package_name=mypackage
+
+        Returns
+        -------
+        int
+            0 If the request was successful.
+        """
+        ret = 1
+        url = self._url() + self._build_path(*paths) + self._build_query(**query_params)
+        status, _, _ = http.delete(url, self._get_headers())
+        if status == 204:
+            # 204 is no content, that GL responds when file deleted
+            ret = 0
+        return ret

glpkg-1.5.0/src/gitlab/lib/http.py

@@ -0,0 +1,90 @@
+"""HTTP helper"""
+
+from http.client import HTTPMessage
+import logging
+from urllib import request
+
+logger = logging.getLogger(__name__)
+
+
+def get(url: str, headers: dict) -> tuple[int, bytes, HTTPMessage]:
+    """
+    Makes a raw GET request to the given URL, and returns
+    the response status, body, and headers.
+
+    Parameters
+    ----------
+    url : str
+        The URL of the HTTP request to make.
+    headers: dict
+        The HTTP headers used in the request.
+
+    Returns
+    -------
+    int
+        The HTTP response code, such as 200
+    bytes
+        The HTTP response body read as bytes
+    HTTPMessage
+        The HTTP response headers
+    """
+    logger.debug("Getting %s", url)
+    req = request.Request(url, headers=headers)
+    with request.urlopen(req) as response:
+        return response.status, response.read(), response.headers
+
+
+def put(url: str, data: bytes, headers: dict) -> tuple[int, bytes, HTTPMessage]:
+    """
+    Makes a raw PUT request to the given URL, and returns
+    the response status, body, and headers.
+
+    Parameters
+    ----------
+    url : str
+        The URL of the HTTP request to make.
+    data : bytes
+        The data to PUT
+    headers: dict
+        The HTTP headers used in the request.
+
+    Returns
+    -------
+    int
+        The HTTP response code, such as 200
+    bytes
+        The HTTP response body read as bytes
+    HTTPMessage
+        The HTTP response headers
+    """
+    logger.debug("Putting %s", url)
+    req = request.Request(url, method="PUT", data=data, headers=headers)
+    with request.urlopen(req) as response:
+        return response.status, response.read(), response.headers
+
+
+def delete(url, headers):
+    """
+    Makes a raw DELETE request to the given URL, and returns
+    the response status, body, and headers.
+
+    Parameters
+    ----------
+    url : str
+        The URL of the HTTP request to make.
+    headers: dict
+        The HTTP headers used in the request.
+
+    Returns
+    -------
+    int
+        The HTTP response code, such as 200
+    bytes
+        The HTTP response body read as bytes
+    HTTPMessage
+        The HTTP response headers
+    """
+    logger.debug("Deleting %s", url)
+    req = request.Request(url, method="DELETE", headers=headers)
+    with request.urlopen(req) as response:
+        return response.status, response.read(), response.headers