cloudrepo-migrate 0.1.0__tar.gz

cloudrepo_migrate-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+# The published sdist is the tool and its README. The tests need a Nexus 2 container and the
+# repository's other files, so they stay in the repository.
+prune test

cloudrepo_migrate-0.1.0/PKG-INFO

@@ -0,0 +1,62 @@
+Metadata-Version: 2.4
+Name: cloudrepo-migrate
+Version: 0.1.0
+Summary: Move a Nexus 2 server's Maven and npm repositories into CloudRepo, verifying every file.
+Project-URL: Homepage, https://www.cloudrepo.io
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# cloudrepo-migrate (early access)
+
+Moves a Nexus 2 server's repositories into CloudRepo and verifies every file.
+
+- **Maven**: every file of each hosted Maven repository, POMs and checksum files included.
+- **npm**: the latest dist-tag of each package in each hosted npm repository (older versions are counted in the report, not moved).
+- **Skipped with a reason**: proxy repositories (create a CloudRepo proxy instead), group repositories (create a CloudRepo group of the moved ones), virtual repositories, other formats, and files Nexus generates itself (`.index`, `.meta`, the root `archetype-catalog.xml`).
+
+Every file is read back from CloudRepo after upload and its sha1 and sha256 compared with the source's bytes. A checksum file the source carries (`X.sha1`, `X.sha256`, `X.sha512`, `X.md5`) must name the digest of `X`. Any mismatch fails the run and names the file.
+
+## Install
+
+```sh
+pipx install cloudrepo-migrate
+```
+
+That installs it from pypi.org and puts `cloudrepo-migrate` on the PATH (`pip install cloudrepo-migrate` in a virtual environment works too).
+
+It is one file, Python 3.8 or later, with no dependencies, so copying `cloudrepo_migrate.py` to a machine that can reach the Nexus server and running `python3 cloudrepo_migrate.py --help` works too.
+
+## Run
+
+Create each target repository in CloudRepo first (same name as the Nexus repository, or map it with `--repo SOURCE=TARGET`) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.
+
+```sh
+export NEXUS_USERNAME=deployer NEXUS_PASSWORD=...   # leave both unset for anonymous read
+export CLOUDREPO_TOKEN=...
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
+```
+
+| Flag                                 | Meaning                                                                                            |
+| ------------------------------------ | -------------------------------------------------------------------------------------------------- |
+| `--nexus-url URL`                    | The Nexus 2 base URL, ending in `/nexus` on a default install.                                     |
+| `--org ORG` or `--cloudrepo-url URL` | Your CloudRepo organization, or its full base URL (https).                                         |
+| `--repo SOURCE[=TARGET]`             | Move only these repositories (repeatable). Default: every hosted Maven and npm repository.         |
+| `--dry-run`                          | List what would move. Uploads nothing. Checks the token and targets when `CLOUDREPO_TOKEN` is set. |
+| `--report-json PATH`                 | Also write the report as JSON.                                                                     |
+| `--jobs N`                           | Parallel Maven uploads (default 4).                                                                |
+| `--work-dir DIR`                     | Where each file is spooled while it moves (default `TMPDIR`).                                      |
+| `--nexus-ca-file PEM`                | CA bundle for a Nexus with a private certificate.                                                  |
+
+With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.
+
+**Resume**: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a `409` on a release CloudRepo already holds counts as moved once its bytes match.
+
+**Exit codes**: `0` everything moved and verified; `1` at least one file failed (every failure is listed with its path; re-run to resume); `2` a usage or preflight problem, such as a missing target repository or a refused token (nothing moved).
+
+## Credentials
+
+- Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
+- The Nexus login is sent only to `--nexus-url`, the CloudRepo token only to CloudRepo.
+- Redirects are never followed, and no URL is taken from a server's answer (a packument's `dist.tarball` contributes only its file name), so neither credential can be steered to another host.
+- CloudRepo must be https (plain http only to localhost). A plain-http Nexus gets a warning.

cloudrepo_migrate-0.1.0/README.md

@@ -0,0 +1,54 @@
+# cloudrepo-migrate (early access)
+
+Moves a Nexus 2 server's repositories into CloudRepo and verifies every file.
+
+- **Maven**: every file of each hosted Maven repository, POMs and checksum files included.
+- **npm**: the latest dist-tag of each package in each hosted npm repository (older versions are counted in the report, not moved).
+- **Skipped with a reason**: proxy repositories (create a CloudRepo proxy instead), group repositories (create a CloudRepo group of the moved ones), virtual repositories, other formats, and files Nexus generates itself (`.index`, `.meta`, the root `archetype-catalog.xml`).
+
+Every file is read back from CloudRepo after upload and its sha1 and sha256 compared with the source's bytes. A checksum file the source carries (`X.sha1`, `X.sha256`, `X.sha512`, `X.md5`) must name the digest of `X`. Any mismatch fails the run and names the file.
+
+## Install
+
+```sh
+pipx install cloudrepo-migrate
+```
+
+That installs it from pypi.org and puts `cloudrepo-migrate` on the PATH (`pip install cloudrepo-migrate` in a virtual environment works too).
+
+It is one file, Python 3.8 or later, with no dependencies, so copying `cloudrepo_migrate.py` to a machine that can reach the Nexus server and running `python3 cloudrepo_migrate.py --help` works too.
+
+## Run
+
+Create each target repository in CloudRepo first (same name as the Nexus repository, or map it with `--repo SOURCE=TARGET`) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.
+
+```sh
+export NEXUS_USERNAME=deployer NEXUS_PASSWORD=...   # leave both unset for anonymous read
+export CLOUDREPO_TOKEN=...
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
+```
+
+| Flag                                 | Meaning                                                                                            |
+| ------------------------------------ | -------------------------------------------------------------------------------------------------- |
+| `--nexus-url URL`                    | The Nexus 2 base URL, ending in `/nexus` on a default install.                                     |
+| `--org ORG` or `--cloudrepo-url URL` | Your CloudRepo organization, or its full base URL (https).                                         |
+| `--repo SOURCE[=TARGET]`             | Move only these repositories (repeatable). Default: every hosted Maven and npm repository.         |
+| `--dry-run`                          | List what would move. Uploads nothing. Checks the token and targets when `CLOUDREPO_TOKEN` is set. |
+| `--report-json PATH`                 | Also write the report as JSON.                                                                     |
+| `--jobs N`                           | Parallel Maven uploads (default 4).                                                                |
+| `--work-dir DIR`                     | Where each file is spooled while it moves (default `TMPDIR`).                                      |
+| `--nexus-ca-file PEM`                | CA bundle for a Nexus with a private certificate.                                                  |
+
+With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.
+
+**Resume**: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a `409` on a release CloudRepo already holds counts as moved once its bytes match.
+
+**Exit codes**: `0` everything moved and verified; `1` at least one file failed (every failure is listed with its path; re-run to resume); `2` a usage or preflight problem, such as a missing target repository or a refused token (nothing moved).
+
+## Credentials
+
+- Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
+- The Nexus login is sent only to `--nexus-url`, the CloudRepo token only to CloudRepo.
+- Redirects are never followed, and no URL is taken from a server's answer (a packument's `dist.tarball` contributes only its file name), so neither credential can be steered to another host.
+- CloudRepo must be https (plain http only to localhost). A plain-http Nexus gets a warning.

cloudrepo_migrate-0.1.0/cloudrepo_migrate.egg-info/PKG-INFO

@@ -0,0 +1,62 @@
+Metadata-Version: 2.4
+Name: cloudrepo-migrate
+Version: 0.1.0
+Summary: Move a Nexus 2 server's Maven and npm repositories into CloudRepo, verifying every file.
+Project-URL: Homepage, https://www.cloudrepo.io
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# cloudrepo-migrate (early access)
+
+Moves a Nexus 2 server's repositories into CloudRepo and verifies every file.
+
+- **Maven**: every file of each hosted Maven repository, POMs and checksum files included.
+- **npm**: the latest dist-tag of each package in each hosted npm repository (older versions are counted in the report, not moved).
+- **Skipped with a reason**: proxy repositories (create a CloudRepo proxy instead), group repositories (create a CloudRepo group of the moved ones), virtual repositories, other formats, and files Nexus generates itself (`.index`, `.meta`, the root `archetype-catalog.xml`).
+
+Every file is read back from CloudRepo after upload and its sha1 and sha256 compared with the source's bytes. A checksum file the source carries (`X.sha1`, `X.sha256`, `X.sha512`, `X.md5`) must name the digest of `X`. Any mismatch fails the run and names the file.
+
+## Install
+
+```sh
+pipx install cloudrepo-migrate
+```
+
+That installs it from pypi.org and puts `cloudrepo-migrate` on the PATH (`pip install cloudrepo-migrate` in a virtual environment works too).
+
+It is one file, Python 3.8 or later, with no dependencies, so copying `cloudrepo_migrate.py` to a machine that can reach the Nexus server and running `python3 cloudrepo_migrate.py --help` works too.
+
+## Run
+
+Create each target repository in CloudRepo first (same name as the Nexus repository, or map it with `--repo SOURCE=TARGET`) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.
+
+```sh
+export NEXUS_USERNAME=deployer NEXUS_PASSWORD=...   # leave both unset for anonymous read
+export CLOUDREPO_TOKEN=...
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
+cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
+```
+
+| Flag                                 | Meaning                                                                                            |
+| ------------------------------------ | -------------------------------------------------------------------------------------------------- |
+| `--nexus-url URL`                    | The Nexus 2 base URL, ending in `/nexus` on a default install.                                     |
+| `--org ORG` or `--cloudrepo-url URL` | Your CloudRepo organization, or its full base URL (https).                                         |
+| `--repo SOURCE[=TARGET]`             | Move only these repositories (repeatable). Default: every hosted Maven and npm repository.         |
+| `--dry-run`                          | List what would move. Uploads nothing. Checks the token and targets when `CLOUDREPO_TOKEN` is set. |
+| `--report-json PATH`                 | Also write the report as JSON.                                                                     |
+| `--jobs N`                           | Parallel Maven uploads (default 4).                                                                |
+| `--work-dir DIR`                     | Where each file is spooled while it moves (default `TMPDIR`).                                      |
+| `--nexus-ca-file PEM`                | CA bundle for a Nexus with a private certificate.                                                  |
+
+With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.
+
+**Resume**: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a `409` on a release CloudRepo already holds counts as moved once its bytes match.
+
+**Exit codes**: `0` everything moved and verified; `1` at least one file failed (every failure is listed with its path; re-run to resume); `2` a usage or preflight problem, such as a missing target repository or a refused token (nothing moved).
+
+## Credentials
+
+- Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
+- The Nexus login is sent only to `--nexus-url`, the CloudRepo token only to CloudRepo.
+- Redirects are never followed, and no URL is taken from a server's answer (a packument's `dist.tarball` contributes only its file name), so neither credential can be steered to another host.
+- CloudRepo must be https (plain http only to localhost). A plain-http Nexus gets a warning.

cloudrepo_migrate-0.1.0/cloudrepo_migrate.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+MANIFEST.in
+README.md
+cloudrepo_migrate.py
+pyproject.toml
+cloudrepo_migrate.egg-info/PKG-INFO
+cloudrepo_migrate.egg-info/SOURCES.txt
+cloudrepo_migrate.egg-info/dependency_links.txt
+cloudrepo_migrate.egg-info/entry_points.txt
+cloudrepo_migrate.egg-info/top_level.txt

cloudrepo_migrate-0.1.0/cloudrepo_migrate.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

cloudrepo_migrate-0.1.0/cloudrepo_migrate.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+cloudrepo-migrate = cloudrepo_migrate:main

cloudrepo_migrate-0.1.0/cloudrepo_migrate.egg-info/top_level.txt

@@ -0,0 +1 @@
+cloudrepo_migrate

cloudrepo_migrate-0.1.0/cloudrepo_migrate.py

@@ -0,0 +1,729 @@
+#!/usr/bin/env python3
+"""cloudrepo-migrate: move a Nexus 2 server's repositories into CloudRepo (early access).
+
+v1 moves Nexus 2 hosted Maven repositories (every file, checksum files and POMs included)
+and hosted npm repositories (the latest dist-tag of each package) into CloudRepo
+repositories that already exist, and verifies every file after upload by reading it back
+and comparing its sha1 and sha256 with the source's bytes. Proxy, group and virtual
+repositories are listed and skipped with a reason. A re-run resumes: a file CloudRepo
+already holds with the same bytes is counted, not uploaded again.
+
+    export NEXUS_USERNAME=... NEXUS_PASSWORD=... CLOUDREPO_TOKEN=...
+    cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
+
+Credentials come only from the environment (or a prompt on a terminal): never argv, never
+printed, never in the report. The Nexus credential is sent only to --nexus-url and the
+CloudRepo token only to CloudRepo. Redirects are never followed and no URL is taken from a
+server's answer, so neither credential can be steered to another host.
+
+Exit codes: 0 everything moved and verified; 1 at least one file failed (each is listed,
+re-run to resume); 2 a usage or preflight problem (nothing moved).
+
+Python 3.8 or later, standard library only.
+"""
+from __future__ import annotations
+
+import argparse
+import base64
+import concurrent.futures
+import getpass
+import hashlib
+import http.client
+import json
+import os
+import re
+import shutil
+import ssl
+import sys
+import tempfile
+import time
+import urllib.parse
+
+VERSION = "0.1.0"
+CREATE_REPOSITORY_URL = "https://admin.cloudrepo.io/repositories/create"
+RETRY_DELAYS = (1, 4, 10)  # seconds between tries on a network error, a 5xx or a 429
+TIMEOUT = 120
+CHUNK = 1 << 20
+SIDECAR_TEXT_LIMIT = 1024
+REPO_NAME = re.compile(r"[a-z0-9_\-]{1,32}")
+STORAGE_PATH = re.compile(r"[A-Za-z0-9+_\-.!*'()/@~]+")
+NPM_NAME = re.compile(r"(?:@[a-z0-9][a-z0-9._-]*/)?[a-z0-9][a-z0-9._-]*")
+NPM_VERSION = re.compile(r"[0-9A-Za-z][0-9A-Za-z.+_-]*")
+TARBALL_FILE = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9._-]*\.tgz")
+ORG = re.compile(r"[a-z0-9][a-z0-9-]{0,62}")
+LOCAL_HOSTS = ("localhost", "127.0.0.1", "::1")
+SIDECARS = {".sha1": "sha1", ".sha256": "sha256", ".sha512": "sha512", ".md5": "md5"}
+COMPARED = ("sha1", "sha256")
+NEXUS_GENERATED = "archetype-catalog.xml"
+
+
+class Usage(Exception):
+    """A usage or preflight problem: nothing was moved (exit 2)."""
+
+
+class Refused(Exception):
+    """One file or request could not be completed; the run continues past it."""
+
+
+# --- HTTP: one origin, one credential ---------------------------------------------------
+
+
+def retrying(attempt):
+    """Run attempt(), retrying a network error, a 5xx or a 429. attempt returns a tuple
+    whose first item is the HTTP status."""
+    for delay in tuple(RETRY_DELAYS) + (None,):
+        try:
+            result = attempt()
+        except (OSError, http.client.HTTPException) as e:
+            if delay is None:
+                raise Refused(f"network error: {e.__class__.__name__}: {e}")
+            time.sleep(delay)
+            continue
+        if delay is not None and (result[0] >= 500 or result[0] == 429):
+            time.sleep(delay)
+            continue
+        return result
+    raise AssertionError("unreachable")
+
+
+class Endpoint:
+    """One origin and the only credential ever sent to it.
+
+    Built on http.client, which has no redirect machinery: a 3xx is an error naming its
+    Location, so a credential can never follow a redirect to another host."""
+
+    def __init__(self, base_url, authorization=None, ca_file=None):
+        parts = urllib.parse.urlsplit(base_url)
+        if parts.scheme not in ("http", "https") or not parts.hostname:
+            raise Usage(f"not an http or https URL: {base_url}")
+        if parts.username or parts.password or parts.query or parts.fragment:
+            raise Usage(f"give the URL without a login, query or fragment: {parts.scheme}://{parts.hostname}")
+        self.https = parts.scheme == "https"
+        self.host = parts.hostname
+        self.port = parts.port
+        self.prefix = parts.path.rstrip("/")
+        self.base = f"{parts.scheme}://{parts.netloc}{self.prefix}"
+        self._authorization = authorization
+        self._context = ssl.create_default_context(cafile=ca_file) if self.https else None
+
+    def url(self, path):
+        return f"{self.base}/{path}"
+
+    def open(self, method, path, body=None, headers=None):
+        """Send one request to base/path (path relative and already encoded); returns
+        (connection, response) for the caller to read and close."""
+        if self.https:
+            conn = http.client.HTTPSConnection(self.host, self.port, timeout=TIMEOUT, context=self._context)
+        else:
+            conn = http.client.HTTPConnection(self.host, self.port, timeout=TIMEOUT)
+        sent = {"User-Agent": f"cloudrepo-migrate/{VERSION}", "Accept": "*/*"}
+        sent.update(headers or {})
+        if self._authorization:
+            sent["Authorization"] = self._authorization
+        try:
+            conn.request(method, f"{self.prefix}/{path}", body=body, headers=sent)
+            resp = conn.getresponse()
+        except BaseException:
+            conn.close()
+            raise
+        if 300 <= resp.status < 400:
+            location = resp.getheader("Location") or "(none)"
+            conn.close()
+            raise Refused(f"{method} {self.url(path)} answered {resp.status}, a redirect to {location}; "
+                          "redirects are never followed, so a credential cannot leave its server")
+        return conn, resp
+
+    def call(self, method, path, body=None, headers=None):
+        """One request with retries; returns (status, lower-cased headers, body bytes)."""
+        def once():
+            conn, resp = self.open(method, path, body, headers)
+            try:
+                return resp.status, {k.lower(): v for k, v in resp.getheaders()}, resp.read()
+            finally:
+                conn.close()
+        return retrying(once)
+
+    def fetch(self, path, spool_dir=None):
+        """GET path, hashing as it streams, into a spool file when spool_dir is given.
+        Returns (status, spool path or None, digests or None, size)."""
+        def once():
+            conn, resp = self.open("GET", path)
+            try:
+                if resp.status != 200:
+                    resp.read()
+                    return resp.status, None, None, 0
+                hashes = new_hashes()
+                fd, spool = tempfile.mkstemp(dir=spool_dir) if spool_dir else (None, None)
+                size = 0
+                with (os.fdopen(fd, "wb") if fd is not None else open(os.devnull, "wb")) as out:
+                    while True:
+                        chunk = resp.read(CHUNK)
+                        if not chunk:
+                            break
+                        out.write(chunk)
+                        size += len(chunk)
+                        for h in hashes.values():
+                            h.update(chunk)
+                return 200, spool, hexes(hashes), size
+            finally:
+                conn.close()
+        return retrying(once)
+
+    def put_file(self, path, spool, size):
+        """PUT a spool file with an explicit length; returns the status."""
+        def once():
+            with open(spool, "rb") as body:
+                conn, resp = self.open("PUT", path, body, {"Content-Type": "application/octet-stream",
+                                                           "Content-Length": str(size)})
+                try:
+                    resp.read()
+                    return (resp.status,)
+                finally:
+                    conn.close()
+        return retrying(once)[0]
+
+
+def new_hashes():
+    hashes = {name: hashlib.new(name) for name in ("sha1", "sha256", "sha512")}
+    try:
+        hashes["md5"] = hashlib.md5()
+    except ValueError:  # a FIPS host: .md5 checksum files are then not checked
+        pass
+    return hashes
+
+
+def hexes(hashes):
+    return {name: h.hexdigest() for name, h in hashes.items()}
+
+
+def digests_of(data):
+    hashes = new_hashes()
+    for h in hashes.values():
+        h.update(data)
+    return hexes(hashes)
+
+
+def same_bytes(a, b):
+    return bool(a) and bool(b) and all(a[k] == b[k] for k in COMPARED)
+
+
+def qpath(path):
+    return urllib.parse.quote(path, safe="/")
+
+
+def npm_doc_path(name):
+    """A validated npm name as one path segment: @scope%2fname."""
+    return name.replace("/", "%2f")
+
+
+# --- Nexus 2 ----------------------------------------------------------------------------
+
+
+class Nexus2:
+    """Reads a Nexus 2 server through its REST API and content URLs."""
+
+    def __init__(self, endpoint):
+        self.ep = endpoint
+
+    def json(self, path):
+        status, _, body = self.ep.call("GET", path, headers={"Accept": "application/json"})
+        if status == 401:
+            raise Refused("Nexus refused the login (set NEXUS_USERNAME and NEXUS_PASSWORD)")
+        if status != 200:
+            raise Refused(f"Nexus answered {status} for {self.ep.url(path)}")
+        try:
+            return json.loads(body)
+        except ValueError:
+            raise Refused(f"Nexus answered something other than JSON for {self.ep.url(path)}")
+
+    def repositories(self):
+        repos = [dict(r, kind=r.get("repoType", "")) for r in self.json("service/local/repositories").get("data", [])]
+        groups = [dict(g, kind="group") for g in self.json("service/local/repo_groups").get("data", [])]
+        return repos + groups
+
+    def maven_files(self, repo_id):
+        """[(path, size)] of every file in a hosted Maven repository. What Nexus generates
+        itself is skipped, as it is not an artifact anyone deployed: its own trees (.index,
+        .meta, .nexus) and the root archetype-catalog.xml, which Nexus renders on each read
+        (measured on 2.15.2: its recorded sha1 did not match the bytes it served)."""
+        todo, seen, files = [""], set(), []
+        while todo:
+            directory = todo.pop()
+            if directory in seen:
+                continue
+            seen.add(directory)
+            listing = self.json(f"service/local/repositories/{qpath(repo_id)}/content/{qpath(directory)}")
+            for item in listing.get("data", []):
+                rel = str(item.get("relativePath", "")).strip("/")
+                segments = rel.split("/")
+                if not rel or rel == NEXUS_GENERATED or any(not s or s.startswith(".") for s in segments):
+                    continue
+                if item.get("leaf"):
+                    files.append((rel, int(item.get("sizeOnDisk") or 0)))
+                else:
+                    todo.append(rel + "/")
+        return sorted(files)
+
+    def recorded_sha1(self, repo_id, path):
+        info = self.json(f"service/local/repositories/{qpath(repo_id)}/content/{qpath(path)}?describe=info")
+        return str((info.get("data") or {}).get("sha1Hash") or "").lower()
+
+    def content(self, repo_id, path):
+        return f"content/repositories/{qpath(repo_id)}/{qpath(path)}"
+
+    def npm_names(self, repo_id):
+        listing = self.json(f"content/repositories/{qpath(repo_id)}/-/all")
+        return sorted(k for k in listing if not k.startswith("_"))
+
+    def packument(self, repo_id, name):
+        return self.json(f"content/repositories/{qpath(repo_id)}/{npm_doc_path(name)}")
+
+
+def nexus_tarball_path(repo_id, name, dist_tarball):
+    """The Nexus path of a package tarball, built from --nexus-url's repository, the
+    validated package name and only the file name at the end of dist.tarball. The host and
+    path in the packument are never requested."""
+    file = str(dist_tarball or "").rsplit("/", 1)[-1]
+    if not TARBALL_FILE.fullmatch(file):
+        raise Refused(f"the packument's dist.tarball does not end in a tarball file name: {file!r}")
+    return f"content/repositories/{qpath(repo_id)}/{name}/-/{file}"
+
+
+def classify(repo):
+    """(format to move or None, reason it is skipped)."""
+    kind, fmt = repo.get("kind", ""), repo.get("format", "")
+    if kind == "group":
+        return None, "group repository: create a CloudRepo group of the moved repositories instead"
+    if kind == "proxy":
+        return None, f"proxy of {repo.get('remoteUri') or 'a remote'}: create a CloudRepo proxy repository instead"
+    if kind in ("virtual", "shadow"):
+        return None, "virtual repository: it stores nothing of its own"
+    if kind != "hosted":
+        return None, f"{kind or 'unknown'} repository: not supported"
+    if fmt == "maven2":
+        return "maven", ""
+    if fmt == "npm":
+        return "npm", ""
+    return None, f"{fmt or 'unknown'} format is not supported yet; v1 moves Maven and npm"
+
+
+# --- CloudRepo --------------------------------------------------------------------------
+
+
+def cloudrepo_base(args):
+    if args.org:
+        if not ORG.fullmatch(args.org):
+            raise Usage(f"--org must be a CloudRepo organization id (a-z, 0-9 and -): {args.org!r}")
+        return f"https://{args.org}.mycloudrepo.io"
+    parts = urllib.parse.urlsplit(args.cloudrepo_url)
+    if parts.scheme != "https" and parts.hostname not in LOCAL_HOSTS:
+        raise Usage("--cloudrepo-url must be https (plain http only to localhost), so the token is never sent in clear")
+    return args.cloudrepo_url
+
+
+def check_token(cr):
+    try:
+        status, _, _ = cr.call("GET", "repositories/")
+    except Refused as e:
+        raise Usage(f"could not reach CloudRepo at {cr.base}: {e}")
+    if status != 200:
+        raise Usage(f"CloudRepo refused the token: GET {cr.url('repositories/')} answered {status}. "
+                    "Check CLOUDREPO_TOKEN and that the token belongs to this organization.")
+
+
+def _is_repository_not_found(body):
+    try:
+        error = json.loads(body).get("error", "")
+    except (ValueError, AttributeError):
+        return False
+    return "repository could not be found" in str(error)
+
+
+def ensure_repository(cr, name, fmt):
+    """None when CloudRepo has repository `name` of format `fmt` and the token reaches it,
+    else the problem to print.
+
+    Slice A of #6777 (a scoped repository-create route) plugs in here: create the missing
+    repository instead of reporting it."""
+    if not REPO_NAME.fullmatch(name):
+        return (f"{name!r} is not a valid CloudRepo repository name (a-z, 0-9, _ and -, at most 32); "
+                f"choose one with --repo SOURCE=TARGET")
+    try:
+        status, _, body = cr.call("GET", f"repositories/{name}/-/ping")
+    except Refused as e:
+        return f"could not check CloudRepo repository {name}: {e}"
+    if status == 404 and _is_repository_not_found(body):
+        return f"CloudRepo has no repository {name}: create it as {_a(fmt)} repository at {CREATE_REPOSITORY_URL}"
+    if status == 403:
+        return f"the token cannot reach CloudRepo repository {name}: give it write access to {name}"
+    if status not in (200, 404):
+        return f"CloudRepo answered {status} checking repository {name}"
+    found = "npm" if status == 200 else "maven"  # -/ping answers only on an npm repository
+    if found != fmt:
+        return f"CloudRepo repository {name} is not {_a(fmt)} repository: create one at {CREATE_REPOSITORY_URL}"
+    return None
+
+
+def _a(fmt):
+    return "an npm" if fmt == "npm" else "a Maven"
+
+
+# --- moving -----------------------------------------------------------------------------
+
+
+def _result(repo, path, status, reason="", size=0, digests=None, text=None):
+    return {"repository": repo, "path": path, "status": status, "reason": reason, "bytes": size,
+            "digests": digests or {}, "text": text}
+
+
+def is_metadata(path):
+    return path.rsplit("/", 1)[-1].startswith("maven-metadata")
+
+
+def move_maven_file(nexus, cr, work, src, dst, path):
+    """Move one Maven file and read it back. Never raises: returns its result."""
+    try:
+        if not STORAGE_PATH.fullmatch(path) or len(path) > 1024:
+            raise Refused("CloudRepo cannot store this path (only A-Z a-z 0-9 + _ - . ! * ' ( ) / @ ~, "
+                          "at most 1024 characters)")
+        recorded = nexus.recorded_sha1(src, path)
+        status, spool, digests, size = nexus.ep.fetch(nexus.content(src, path), work)
+        if status != 200:
+            raise Refused(f"Nexus answered {status} reading the file")
+        try:
+            if recorded and digests["sha1"] != recorded:
+                raise Refused(f"the bytes Nexus served (sha1 {digests['sha1']}) do not match the sha1 Nexus "
+                              f"recorded for the file ({recorded}); not uploaded")
+            text = None
+            if any(path.endswith(ext) for ext in SIDECARS) and size <= SIDECAR_TEXT_LIMIT:
+                with open(spool, "rb") as f:
+                    text = f.read().decode("ascii", "replace")
+            target = f"repositories/{dst}/{qpath(path)}"
+            head, headers, _ = cr.call("HEAD", target)
+            if head == 200 and headers.get("content-length") == str(size):
+                if same_bytes(cr.fetch(target)[2], digests):
+                    return _result(src, path, "present", size=size, digests=digests, text=text)
+            put = cr.put_file(target, spool, size)
+            if put == 403:
+                raise Refused(f"the token cannot publish to CloudRepo repository {dst}")
+            if put not in (200, 201, 204, 409):
+                raise Refused(f"CloudRepo answered {put} to the upload")
+            back = cr.fetch(target)
+            if back[0] != 200:
+                raise Refused(f"CloudRepo answered {back[0]} reading the file back after the upload")
+            if not same_bytes(back[2], digests):
+                raise Refused("CloudRepo already holds different bytes for this path and refuses overwrites"
+                              if put == 409 else "the bytes CloudRepo serves back differ from the source")
+            return _result(src, path, "present" if put == 409 else "moved", size=size, digests=digests, text=text)
+        finally:
+            os.unlink(spool)
+    except Refused as e:
+        return _result(src, path, "failed", str(e))
+
+
+def move_maven_repository(nexus, cr, work, src, dst, jobs, files):
+    """Every ordinary file first, then the maven-metadata files, so a build reading the
+    metadata never sees a version whose files have not landed."""
+    results = []
+    with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
+        for phase in ([p for p, _ in files if not is_metadata(p)], [p for p, _ in files if is_metadata(p)]):
+            results.extend(pool.map(lambda p: move_maven_file(nexus, cr, work, src, dst, p), phase))
+    return results
+
+
+def _sri(data):
+    return "sha512-" + base64.b64encode(hashlib.sha512(data).digest()).decode()
+
+
+def _target_has(cr, doc_path, tarball_path, version, digests):
+    status, _, body = cr.call("GET", doc_path, headers={"Accept": "application/json"})
+    if status != 200:
+        return False
+    try:
+        shasum = json.loads(body)["versions"][version]["dist"]["shasum"]
+    except (ValueError, KeyError, TypeError):
+        return False
+    return shasum == digests["sha1"] and same_bytes(cr.fetch(tarball_path)[2], digests)
+
+
+def move_npm_package(nexus, cr, src, dst, name):
+    """Move the latest dist-tag of one package and read it back. Never raises: returns
+    (result, number of older versions not moved)."""
+    older = 0
+    try:
+        if not NPM_NAME.fullmatch(name) or len(name) > 214:
+            raise Refused("not a package name CloudRepo accepts")
+        packument = nexus.packument(src, name)
+        versions = packument.get("versions") or {}
+        version = (packument.get("dist-tags") or {}).get("latest")
+        manifest = versions.get(version) if isinstance(version, str) else None
+        if not isinstance(manifest, dict) or not NPM_VERSION.fullmatch(version):
+            raise Refused("the package has no usable latest dist-tag")
+        older = len(versions) - 1
+        dist = manifest.get("dist") or {}
+        status, _, tgz = nexus.ep.call("GET", nexus_tarball_path(src, name, dist.get("tarball")))
+        if status != 200:
+            raise Refused(f"Nexus answered {status} reading the {version} tarball")
+        digests = digests_of(tgz)
+        if str(dist.get("shasum") or "").lower() != digests["sha1"]:
+            raise Refused(f"the {version} tarball does not match the packument's dist.shasum; not uploaded")
+        if dist.get("integrity", "").startswith("sha512-") and dist["integrity"] != _sri(tgz):
+            raise Refused(f"the {version} tarball does not match the packument's dist.integrity; not uploaded")
+        file = f"{name.rsplit('/', 1)[-1]}-{version}.tgz"
+        doc_path = f"repositories/{dst}/{npm_doc_path(name)}"
+        tarball_path = f"repositories/{dst}/{name}/-/{qpath(file)}"
+        path = f"{name}@{version}"
+        if _target_has(cr, doc_path, tarball_path, version, digests):
+            return _result(src, path, "present", size=len(tgz), digests=digests), older
+        stored = dict(manifest, dist=dict(dist, tarball=cr.url(tarball_path)))
+        doc = {"_id": name, "name": name, "description": manifest.get("description", ""),
+               "dist-tags": {"latest": version}, "versions": {version: stored},
+               "_attachments": {file: {"content_type": "application/octet-stream",
+                                       "data": base64.b64encode(tgz).decode(), "length": len(tgz)}}}
+        put, _, body = cr.call("PUT", doc_path, json.dumps(doc).encode(), {"Content-Type": "application/json"})
+        if put == 403:
+            raise Refused(f"the token cannot publish to CloudRepo repository {dst}")
+        if put == 400:
+            raise Refused(f"CloudRepo refused the package: {body[:200].decode('utf-8', 'replace')}")
+        if put not in (200, 201, 409):
+            raise Refused(f"CloudRepo answered {put} to the publish")
+        if not _target_has(cr, doc_path, tarball_path, version, digests):
+            raise Refused("CloudRepo already holds a different tarball for this version and refuses overwrites"
+                          if put == 409 else "the tarball CloudRepo serves back differs from the source")
+        return _result(src, path, "present" if put == 409 else "moved", size=len(tgz), digests=digests), older
+    except Refused as e:
+        return _result(src, name, "failed", str(e)), older
+
+
+def sidecar_failures(results):
+    """Pure: a moved checksum file (X.sha1, X.sha256, X.sha512, X.md5) must name the digest
+    of X's bytes. The bytes moved unchanged, so a disagreement is already in the source; it
+    is still a failure, naming X, so nobody trusts a checksum that is wrong."""
+    landed = {(r["repository"], r["path"]): r for r in results if r["status"] != "failed"}
+    failures = []
+    for (repo, path), r in sorted(landed.items()):
+        for ext, algorithm in SIDECARS.items():
+            if not path.endswith(ext) or r.get("text") is None:
+                continue
+            main = landed.get((repo, path[: -len(ext)]))
+            if not main or algorithm not in main["digests"]:
+                continue
+            claimed = (r["text"].split() or [""])[0].lower()
+            actual = main["digests"][algorithm]
+            if claimed != actual:
+                failures.append({"repository": repo, "path": main["path"],
+                                 "reason": f"its {ext[1:]} checksum file (where the source has it) says {claimed}, "
+                                           f"but the file's bytes hash to {actual}"})
+    return failures
+
+
+# --- the run ----------------------------------------------------------------------------
+
+
+def _secret(var, prompt, required):
+    value = os.environ.get(var)
+    if value:
+        return value
+    if required and sys.stdin.isatty():
+        return getpass.getpass(prompt)
+    return None
+
+
+def parse_mapping(values):
+    mapping = {}
+    for value in values:
+        src, _, dst = value.partition("=")
+        if not src:
+            raise Usage(f"--repo wants SOURCE or SOURCE=TARGET: {value!r}")
+        mapping[src] = dst or src
+    return mapping
+
+
+def _entry(repo, fmt, action, reason="", target=""):
+    return {"source": repo.get("id", ""), "target": target, "format": fmt or repo.get("format", ""),
+            "action": action, "reason": reason, "files": 0, "bytes": 0, "verified": 0,
+            "already_present": 0, "failed": 0}
+
+
+def run_nexus2(args):
+    started = time.monotonic()
+    if args.jobs < 1 or args.jobs > 32:
+        raise Usage("--jobs must be between 1 and 32")
+    if urllib.parse.urlsplit(args.nexus_url).scheme == "http":
+        print("warning: --nexus-url is plain http, so the Nexus login crosses the network unencrypted",
+              file=sys.stderr)
+    base = cloudrepo_base(args)
+    user = os.environ.get("NEXUS_USERNAME")
+    nexus_auth = None
+    if user:
+        password = _secret("NEXUS_PASSWORD", "Nexus password: ", True) or ""
+        nexus_auth = "Basic " + base64.b64encode(f"{user}:{password}".encode()).decode()
+    nexus = Nexus2(Endpoint(args.nexus_url, nexus_auth, args.nexus_ca_file))
+    token = _secret("CLOUDREPO_TOKEN", "CloudRepo token: ", not args.dry_run)
+    if not token and not args.dry_run:
+        raise Usage("set CLOUDREPO_TOKEN to a CloudRepo repository token that can write the target repositories")
+    cr = Endpoint(base, f"Bearer {token}") if token else None
+    report = {"source": nexus.ep.base, "target": base, "dry_run": bool(args.dry_run), "repositories": [],
+              "failures": []}
+
+    try:
+        repos = nexus.repositories()
+    except Refused as e:
+        raise Usage(f"could not list the Nexus repositories: {e}")
+    mapping = parse_mapping(args.repo)
+    known = {r.get("id") for r in repos}
+    unknown = sorted(set(mapping) - known)
+    if unknown:
+        raise Usage(f"Nexus has no repository {', '.join(unknown)}")
+    plan = []
+    for repo in sorted(repos, key=lambda r: str(r.get("id"))):
+        if mapping and repo.get("id") not in mapping:
+            continue
+        fmt, reason = classify(repo)
+        if fmt is None:
+            report["repositories"].append(_entry(repo, None, "skipped", reason))
+        else:
+            plan.append((repo["id"], mapping.get(repo["id"], repo["id"]), fmt))
+
+    if cr:
+        check_token(cr)
+        problems = {src: ensure_repository(cr, dst, fmt) for src, dst, fmt in plan}
+        if any(problems.values()):
+            for src, dst, fmt in plan:
+                report["repositories"].append(_entry({"id": src}, fmt, "blocked" if problems[src] else "ready",
+                                                     problems[src] or "", dst))
+            report["minutes"] = _minutes(started)
+            return 2, report
+
+    work = tempfile.mkdtemp(prefix="cloudrepo-migrate-", dir=args.work_dir)  # mode 0700
+    results = []
+    try:
+        for src, dst, fmt in plan:
+            entry = _entry({"id": src}, fmt, "would move" if args.dry_run else "moved", target=dst)
+            report["repositories"].append(entry)
+            try:
+                if fmt == "maven":
+                    files = nexus.maven_files(src)
+                    if args.dry_run:
+                        entry.update(files=len(files), bytes=sum(size for _, size in files))
+                        continue
+                    moved = move_maven_repository(nexus, cr, work, src, dst, args.jobs, files)
+                else:
+                    names = nexus.npm_names(src)
+                    if args.dry_run:
+                        entry.update(files=len(names), reason="latest dist-tag of each package")
+                        continue
+                    moved, older = [], 0
+                    for name in names:
+                        result, skipped = move_npm_package(nexus, cr, src, dst, name)
+                        moved.append(result)
+                        older += skipped
+                    if older:
+                        entry["reason"] = (f"{older} older version{'s' if older != 1 else ''} not moved "
+                                           "(v1 moves the latest dist-tag only)")
+            except Refused as e:
+                entry["failed"] += 1
+                report["failures"].append({"repository": src, "path": "", "reason": f"could not list: {e}"})
+                continue
+            results.extend(moved)
+            for r in moved:
+                entry["files"] += 1
+                if r["status"] == "failed":
+                    entry["failed"] += 1
+                    report["failures"].append({"repository": src, "path": r["path"], "reason": r["reason"]})
+                else:
+                    entry["bytes"] += r["bytes"]
+                    entry["verified"] += 1
+                    entry["already_present"] += r["status"] == "present"
+    finally:
+        shutil.rmtree(work, ignore_errors=True)
+
+    for failure in sidecar_failures(results):
+        report["failures"].append(failure)
+        next(e for e in report["repositories"] if e["source"] == failure["repository"])["failed"] += 1
+    report["minutes"] = _minutes(started)
+    return (1 if report["failures"] else 0), report
+
+
+def _minutes(started):
+    return round((time.monotonic() - started) / 60, 1)
+
+
+def totals(report):
+    moving = [e for e in report["repositories"] if e["action"] in ("moved", "would move")]
+    keys = ("files", "bytes", "verified", "already_present")
+    t = {k: sum(e[k] for e in moving) for k in keys}
+    t.update(repositories=len(moving), failed=len(report["failures"]))
+    return t
+
+
+def render(report):
+    t = report["totals"]
+    lines = [f"cloudrepo-migrate nexus2: {report['source']} -> {report['target']}"
+             + (" (dry run, nothing uploaded)" if report["dry_run"] else "")]
+    for e in report["repositories"]:
+        where = f"{e['source']} -> {e['target']}" if e["target"] else e["source"]
+        note = f"; {e['reason']}" if e["reason"] else ""
+        if e["action"] == "moved":
+            lines.append(f"  moved       {where} ({e['format']}): {e['files']} files, {e['bytes']} bytes, "
+                         f"{e['verified']} verified ({e['already_present']} already present), {e['failed']} failed{note}")
+        elif e["action"] == "would move":
+            lines.append(f"  would move  {where} ({e['format']}): {e['files']} files, {e['bytes']} bytes{note}")
+        else:
+            lines.append(f"  {e['action']:<11} {where} ({e['format']}){': ' + e['reason'] if e['reason'] else ''}")
+    if report["failures"]:
+        lines.append("Failures (re-run the same command to resume):")
+        lines.extend(f"  {f['repository']} {f['path']}: {f['reason']}" for f in report["failures"])
+    if any(e["action"] == "blocked" for e in report["repositories"]):
+        lines.append("Nothing moved: fix the blocked repositories above, then run again.")
+    elif report["dry_run"]:
+        lines.append(f"Would move {t['repositories']} repositories, {t['files']} files, {t['bytes']} bytes.")
+    else:
+        lines.append(f"Moved {t['repositories']} repositories, {t['files']} files, {t['bytes']} bytes in "
+                     f"{report['minutes']} minutes; {t['verified']} of {t['files']} verified, {t['failed']} failed.")
+    return "\n".join(lines)
+
+
+def parser():
+    p = argparse.ArgumentParser(
+        prog="cloudrepo-migrate",
+        description="Move repositories into CloudRepo, verifying every file. Credentials come from the "
+                    "environment: NEXUS_USERNAME and NEXUS_PASSWORD (leave unset for anonymous), CLOUDREPO_TOKEN.")
+    p.add_argument("--version", action="version", version=f"cloudrepo-migrate {VERSION}")
+    sub = p.add_subparsers(dest="source", required=True)
+    n = sub.add_parser("nexus2", help="move a Nexus 2 server's hosted Maven and npm repositories")
+    n.add_argument("--nexus-url", required=True, help="the Nexus base URL, such as https://nexus.example.com/nexus")
+    target = n.add_mutually_exclusive_group(required=True)
+    target.add_argument("--org", help="your CloudRepo organization id")
+    target.add_argument("--cloudrepo-url", help="the CloudRepo base URL, instead of --org")
+    n.add_argument("--repo", action="append", default=[], metavar="SOURCE[=TARGET]",
+                   help="move only this Nexus repository (repeatable); TARGET defaults to SOURCE")
+    n.add_argument("--dry-run", action="store_true", help="list what would move; upload nothing")
+    n.add_argument("--report-json", metavar="PATH", help="also write the report as JSON")
+    n.add_argument("--jobs", type=int, default=4, help="parallel Maven uploads (default 4)")
+    n.add_argument("--work-dir", metavar="DIR", help="where files are spooled while they move (default: TMPDIR)")
+    n.add_argument("--nexus-ca-file", metavar="PEM", help="CA bundle for a Nexus with a private certificate")
+    return p
+
+
+def main(argv=None):
+    try:
+        args = parser().parse_args(argv)
+    except SystemExit as e:
+        return int(e.code or 0)
+    try:
+        code, report = run_nexus2(args)
+    except Usage as e:
+        print(f"cloudrepo-migrate: {e}", file=sys.stderr)
+        return 2
+    report["totals"] = totals(report)
+    print(render(report))
+    if args.report_json:
+        with open(args.report_json, "w", encoding="utf-8") as f:
+            json.dump(report, f, indent=2)
+            f.write("\n")
+    return code
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cloudrepo_migrate-0.1.0/pyproject.toml

@@ -0,0 +1,26 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cloudrepo-migrate"
+# One version, VERSION in cloudrepo_migrate.py, which `--version` and the User-Agent also print.
+# setuptools reads the literal without importing the module.
+dynamic = ["version"]
+description = "Move a Nexus 2 server's Maven and npm repositories into CloudRepo, verifying every file."
+# The README is the project page on pypi.org.
+readme = "README.md"
+requires-python = ">=3.8"
+dependencies = []
+
+[project.urls]
+Homepage = "https://www.cloudrepo.io"
+
+[project.scripts]
+cloudrepo-migrate = "cloudrepo_migrate:main"
+
+[tool.setuptools]
+py-modules = ["cloudrepo_migrate"]
+
+[tool.setuptools.dynamic]
+version = { attr = "cloudrepo_migrate.VERSION" }

cloudrepo_migrate-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+