devpi-server 6.11.0__tar.gz → 6.12.1__tar.gz

{devpi_server-6.11.0 → devpi-server-6.12.1}/CHANGELOG

@@ -2,6 +2,39 @@
 
 .. towncrier release notes start
 
+6.12.1 (2024-07-24)
+===================
+
+Bug Fixes
+---------
+
+- Support Python 3.13 by depending on legacy-cgi.
+
+- Preserve query string when proxying requests from replica to primary. This fixes force removal on non-volatile indexes and probably other bugs.
+
+- Fix #1044: Correctly update cache expiry time when mirrored server returns 304 Not Modified.
+
+
+
+6.12.0 (2024-06-25)
+===================
+
+Features
+--------
+
+- Added ``devpiserver_on_toxresult_store`` hook to allow blocking or skipping a toxresult upload on more specific conditions as ``acl_toxresult_upload`` would allow.
+
+- Added ``devpiserver_on_toxresult_upload_forbidden`` hook to allow returning a custom message and result (403 or 200).
+
+
+
+Bug Fixes
+---------
+
+- Return json data if toxresult upload is forbidden.
+
+
+
 6.11.0 (2024-04-20)
 ===================
 

{devpi_server-6.11.0 → devpi-server-6.12.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: devpi-server
-Version: 6.11.0
+Version: 6.12.1
 Summary: devpi-server: reliable private and pypi.org caching server
 Home-page: https://devpi.net
 Maintainer: Florian Schulze
@@ -27,25 +27,10 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.7
 License-File: LICENSE
 License-File: AUTHORS
-Requires-Dist: py>=1.4.23
-Requires-Dist: httpx
-Requires-Dist: argon2-cffi
-Requires-Dist: attrs>=21.3.0
-Requires-Dist: defusedxml
-Requires-Dist: devpi_common<5,>3.6.0
-Requires-Dist: itsdangerous>=0.24
-Requires-Dist: platformdirs
-Requires-Dist: pyramid>=2
-Requires-Dist: waitress>=1.0.1
-Requires-Dist: repoze.lru>=0.6
-Requires-Dist: passlib[argon2]
-Requires-Dist: pluggy<2.0,>=0.6.0
-Requires-Dist: ruamel.yaml
-Requires-Dist: strictyaml
-Requires-Dist: lazy
 
 =============================================================================
 devpi-server: server for private package indexes and PyPI caching
@@ -119,89 +104,86 @@ Changelog
 
 .. towncrier release notes start
 
-6.11.0 (2024-04-20)
+6.12.1 (2024-07-24)
 ===================
 
-Features
---------
-
-- The ``devpi-fsck`` script now returns an error code when there have been missing files or checksum errors.
-
-- Fix #983: Add plugin hook for mirror authentication header.
-
-
-
 Bug Fixes
 ---------
 
-- Preserve last modified of docs and toxresults during export/import.
+- Support Python 3.13 by depending on legacy-cgi.
 
-- Fix #1033: Use ``int`` for ``--mirror-cache-expiry`` to fix value of ``proxy_cache_valid`` in nginx caching config.
+- Preserve query string when proxying requests from replica to primary. This fixes force removal on non-volatile indexes and probably other bugs.
 
+- Fix #1044: Correctly update cache expiry time when mirrored server returns 304 Not Modified.
 
 
-6.10.0 (2023-12-19)
+
+6.12.0 (2024-06-25)
 ===================
 
 Features
 --------
 
-- Use ``Authorization`` header instead of adding username/password to URL when fetching from mirror.
+- Added ``devpiserver_on_toxresult_store`` hook to allow blocking or skipping a toxresult upload on more specific conditions as ``acl_toxresult_upload`` would allow.
 
-- Fix #998: Use the pure Python httpx library instead of aiohttp to prevent delays in supporting newest Python releases.
+- Added ``devpiserver_on_toxresult_upload_forbidden`` hook to allow returning a custom message and result (403 or 200).
 
 
 
 Bug Fixes
 ---------
 
-- Fix #996: support hashes other than sha256 in application/vnd.pypi.simple.v1+json responses.
+- Return json data if toxresult upload is forbidden.
 
-- Only compare hostname instead of full URL prefix when parsing mirror packages to fix mirrors with basic authentication and absolute URLs. See #1006
 
 
+6.11.0 (2024-04-20)
+===================
 
-6.9.2 (2023-08-06)
-==================
+Features
+--------
 
-Bug Fixes
----------
+- The ``devpi-fsck`` script now returns an error code when there have been missing files or checksum errors.
 
-- Prevent duplicates when adding values to lists in index configuration with ``+=`` operator.
+- Fix #983: Add plugin hook for mirror authentication header.
 
 
-6.9.1 (2023-07-02)
-==================
 
 Bug Fixes
 ---------
 
-- Prevent error in find_pre_existing_file in case of incomplete metadata.
+- Preserve last modified of docs and toxresults during export/import.
 
-- Fix #980: Remove long deprecated backward compatibility for old pluggy versions to fix error with pluggy 1.1.0.
+- Fix #1033: Use ``int`` for ``--mirror-cache-expiry`` to fix value of ``proxy_cache_valid`` in nginx caching config.
 
 
-6.9.0 (2023-05-23)
-==================
+
+6.10.0 (2023-12-19)
+===================
 
 Features
 --------
 
-- Support export directory layout for ``--replica-file-search-path`` option.
+- Use ``Authorization`` header instead of adding username/password to URL when fetching from mirror.
+
+- Fix #998: Use the pure Python httpx library instead of aiohttp to prevent delays in supporting newest Python releases.
 
-- Fix #931: Add ``mirror_no_project_list`` setting for mirror indexes that have no full project list like google cloud artifacts or if you want to prevent downloading the full list for huge indexes like PyPI.
 
 
 Bug Fixes
 ---------
 
-- Keep a reference to async tasks to avoid their removal mid execution.
+- Fix #996: support hashes other than sha256 in application/vnd.pypi.simple.v1+json responses.
+
+- Only compare hostname instead of full URL prefix when parsing mirror packages to fix mirrors with basic authentication and absolute URLs. See #1006
+
 
-- Support changed default of ``enforce_content_length`` in urllib3 >= 2.
 
-- Fix #934: Properly set PATH_INFO when outside URL is used with sub-path.
+6.9.2 (2023-08-06)
+==================
 
-- Fix #945: Adapt FatalError to be usable as an async HTTP response when updating a project on a mirror.
+Bug Fixes
+---------
 
-- Fix wrong hash metadata introduced in 6.5.0 for toxresults which prevents replication. The metadata can be fixed by an export/import cycle.
+- Prevent duplicates when adding values to lists in index configuration with ``+=`` operator.
 

devpi-server-6.12.1/devpi_server/__init__.py

@@ -0,0 +1 @@
+__version__ = '6.12.1'

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/config.py

@@ -321,27 +321,15 @@ def add_deploy_options(parser, pluginmanager):
 
     parser.addoption(
         "--argon2-memory-cost", type=int, default=DEFAULT_ARGON2_MEMORY_COST,
-        help="Argon2 memory cost parameter for key derivation. "
-             "This is *not* for the user password hashes. "
-             "There should be no need to touch this setting, "
-             "except you really know what this is about! "
-             "Replicas need to use the same parameters as the master.")
+        help=argparse.SUPPRESS)
 
     parser.addoption(
         "--argon2-parallelism", type=int, default=DEFAULT_ARGON2_PARALLELISM,
-        help="Argon2 parallelism parameter for key derivation. "
-             "This is *not* for the user password hashes. "
-             "There should be no need to touch this setting, "
-             "except you really know what this is about! "
-             "Replicas need to use the same parameters as the master.")
+        help=argparse.SUPPRESS)
 
     parser.addoption(
         "--argon2-time-cost", type=int, default=DEFAULT_ARGON2_TIME_COST,
-        help="Argon2 time cost parameter for key derivation. "
-             "This is *not* for the user password hashes. "
-             "There should be no need to touch this setting, "
-             "except you really know what this is about! "
-             "Replicas need to use the same parameters as the master.")
+        help=argparse.SUPPRESS)
 
     parser.addoption(
         "--requests-only", action="store_true",
@@ -567,7 +555,7 @@ class MyArgumentParser(argparse.ArgumentParser):
             default = action.default
             if isinstance(action, argparse._StoreFalseAction):
                 default = not default
-            if action.help and action.default != '==SUPPRESS==':
+            if action.help and action.help is not argparse.SUPPRESS and action.default is not argparse.SUPPRESS:
                 action.help += " [%s]" % default
 
     def addgroup(self, *args, **kwargs):

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/hookspecs.py

@@ -1,5 +1,13 @@
-
+from __future__ import annotations
 from pluggy import HookspecMarker
+from typing import Optional
+from typing import TYPE_CHECKING
+
+
+if TYPE_CHECKING:
+    from devpi_server.views import ToxResultHandling
+    from pyramid.request import Request
+
 
 hookspec = HookspecMarker("devpiserver")
 
@@ -255,6 +263,28 @@ def devpiserver_on_replicated_file(stage, project, version, link, serial, back_s
     """Called when a file was downloaded from master on replica."""
 
 
+@hookspec(firstresult=True)
+def devpiserver_on_toxresult_store(request: Request, tox_result_handling: ToxResultHandling) -> Optional[ToxResultHandling]:
+    """Called when a toxresult is about to be stored.
+
+    Stops at first non-None result.
+
+    :returns: A ToxResultHandling object which determines how the upload is processed.
+    """
+
+
+@hookspec(firstresult=True)
+def devpiserver_on_toxresult_upload_forbidden(request: Request, tox_result_handling: ToxResultHandling) -> Optional[str]:
+    """Called when the permission check for toxresult upload failed.
+
+    Stops at first non-None result.
+
+    :returns: A ToxResultHandling object which determines whether an error
+    with message (default using ``block``) or a success with a message
+    (using ``skip``) is returned.
+    """
+
+
 @hookspec
 def devpiserver_metrics(request):
     """ called for status view.

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/keyfs.py

@@ -673,13 +673,9 @@ class Transaction(object):
     def exists(self, typedkey):
         if typedkey in self.cache:
             val = self.cache[typedkey]
-            if val in (absent, deleted):
-                return False
-            return True
+            return val not in (absent, deleted)
         (serial, val) = self.get_original(typedkey)
-        if val in (absent, deleted):
-            return False
-        return True
+        return val not in (absent, deleted)
 
     def delete(self, typedkey):
         if not self.write:
@@ -692,7 +688,7 @@ class Transaction(object):
             raise self.keyfs.ReadOnly()
         # sanity check for dictionaries: we always want to have unicode
         # keys, not bytes
-        if typedkey.type == dict:
+        if typedkey.type is dict:
             check_unicode_keys(val)
         assert val is not None
         self.cache[typedkey] = val

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/mirror.py

@@ -794,8 +794,8 @@ class MirrorStage(BaseStage):
                 f"timeout after {self.timeout} seconds while getting data for {project!r}")
         except self.UpstreamNotModified as e:
             if links is not None:
-                self.keyfs.tx.on_commit_success(partial(
-                    self.cache_retrieve_times.refresh, project, e.etag))
+                # immediately update the cache
+                self.cache_retrieve_times.refresh(project, e.etag)
                 return self.SimpleLinks(links)
             if e.etag is None:
                 threadlog.error(

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/model.py

@@ -42,9 +42,9 @@ def join_links_data(links, requires_python, yanked):
     # build list of (key, href, require_python, yanked) tuples
     result = []
     links = zip_longest(links, requires_python, yanked, fillvalue=None)
-    for link, require_python, yanked in links:
+    for link, require_python, link_yanked in links:
         key, href = link
-        result.append((key, href, require_python, yanked))
+        result.append((key, href, require_python, link_yanked))
     return result
 
 
@@ -1615,11 +1615,10 @@ class LinkStore:
 
     @property
     def metadata(self):
-        metadata = {}
-        for k, v in get_mutable_deepcopy(self.verdata).items():
-            if not k.startswith("+"):
-                metadata[k] = v
-        return metadata
+        return {
+            k: v
+            for k, v in get_mutable_deepcopy(self.verdata).items()
+            if not k.startswith("+")}
 
     def get_file_entry(self, relpath):
         return self.filestore.get_file_entry(relpath)

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/replica.py

@@ -6,6 +6,7 @@ import secrets
 import threading
 import time
 import traceback
+from contextlib import suppress
 from functools import partial
 from pluggy import HookimplMarker
 from pyramid.httpexceptions import HTTPNotFound, HTTPAccepted, HTTPBadRequest
@@ -1174,18 +1175,22 @@ class BodyFileWrapper:
         self.len = length
 
 
-def proxy_request_to_master(xom, request, stream=False):
+def proxy_request_to_master(xom, request, *, stream=False):
     master_url = xom.config.master_url
-    url = master_url.joinpath(request.path).url
+    request_url = URL(request.url)
+    url = (
+        master_url
+        .joinpath(request_url.path)
+        .replace(query=request_url.query)
+        .url)
     assert url.startswith(master_url.url)
     http = xom._httpsession
     with threadlog.around("info", "relaying: %s %s", request.method, url):
         try:
             headers = clean_request_headers(request)
-            try:
+            length = None
+            with suppress(ValueError, TypeError):
                 length = int(headers.get('Content-Length'))
-            except (ValueError, TypeError):
-                length = None
             if length:
                 body = BodyFileWrapper(request.body_file, length)
             else:
@@ -1197,7 +1202,8 @@ def proxy_request_to_master(xom, request, stream=False):
                                 allow_redirects=False,
                                 timeout=xom.config.args.proxy_timeout)
         except http.Errors as e:
-            raise UpstreamError("proxy-write-to-master %s: %s" % (url, e))
+            msg = f"proxy-write-to-master {url}: {e}"
+            raise UpstreamError(msg) from e
 
 
 def proxy_write_to_master(xom, request):
@@ -1231,10 +1237,11 @@ def proxy_write_to_master(xom, request):
                     headers=headers)
 
 
-def proxy_view_to_master(context, request):
+def proxy_view_to_master(_context, request):
     xom = request.registry["xom"]
     tx = getattr(xom.keyfs, "tx", None)
-    assert getattr(tx, "write", False) is False, "there should be no write transaction"
+    if getattr(tx, "write", False):
+        raise RuntimeError("there should be no write transaction")
     return proxy_write_to_master(xom, request)
 
 

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/vendor/_pip.py

@@ -14,12 +14,12 @@ class HTMLPage(object):
     """Represents one page, along with its URL"""
 
     # FIXME: these regexes are horrible hacks:
-    _homepage_re = re.compile(r'<th>\s*home\s*page', re.I)
-    _download_re = re.compile(r'<th>\s*download\s+url', re.I)
+    _homepage_re = re.compile(r'<th>\s*home\s*page', re.IGNORECASE)
+    _download_re = re.compile(r'<th>\s*download\s+url', re.IGNORECASE)
     # These aren't so awful:
-    _rel_re = re.compile(r"""<[^>]*\srel\s*=\s*['"]?([^'">]+)[^>]*>""", re.I)
-    _href_re = re.compile('href=(?:"([^"]*)"|\'([^\']*)\'|([^>\\s\\n]*))', re.I | re.S)
-    _base_re = re.compile(r"""<base\s+href\s*=\s*['"]?([^'">]+)""", re.I)
+    _rel_re = re.compile(r"""<[^>]*\srel\s*=\s*['"]?([^'">]+)[^>]*>""", re.IGNORECASE)
+    _href_re = re.compile('href=(?:"([^"]*)"|\'([^\']*)\'|([^>\\s\\n]*))', re.IGNORECASE | re.DOTALL)
+    _base_re = re.compile(r"""<base\s+href\s*=\s*['"]?([^'">]+)""", re.IGNORECASE)
 
     def __init__(self, content, url, headers=None):
         self.content = content
@@ -113,7 +113,7 @@ class HTMLPage(object):
             url = self.clean_link(urljoin(self.base_url, url))
             yield Link(url, self)
 
-    _clean_re = re.compile(r'[^a-z0-9$&+,/:;=?@.#%_\\|-]', re.I)
+    _clean_re = re.compile(r'[^a-z0-9$&+,/:;=?@.#%_\\|-]', re.IGNORECASE)
 
     def clean_link(self, url):
         """Makes sure a link is fully encoded.  That is, if a ' ' shows up in

{devpi_server-6.11.0 → devpi-server-6.12.1}/devpi_server/views.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 import contextlib
 import os
 import re
@@ -50,6 +51,7 @@ from .readonly import get_mutable_deepcopy
 from .log import thread_push_log, thread_pop_log, threadlog
 
 from .auth import Auth
+import attrs
 
 devpiweb_hookimpl = HookimplMarker("devpiweb")
 server_version = devpi_server.__version__
@@ -79,9 +81,7 @@ def _select_simple_content_type(request):
 
 
 def is_simple_json(request):
-    if _select_simple_content_type(request) == SIMPLE_API_V1_JSON:
-        return True
-    return False
+    return _select_simple_content_type(request) == SIMPLE_API_V1_JSON
 
 
 def is_requested_by_installer(request):
@@ -412,6 +412,34 @@ def devpiserver_authcheck_forbidden(request):
         return True
 
 
+TOXRESULT_UPLOAD_FORBIDDEN = (
+    "No permission to upload tox results. "
+    "You can use the devpi test --no-upload option to skip the upload.")
+
+
+@attrs.define(frozen=True)
+class ToxResultHandling:
+    _block: bool = attrs.field(default=False, alias="_block")
+    _skip: bool = attrs.field(default=False, alias="_skip")
+    msg: None | str = None
+
+    def block(self, msg=TOXRESULT_UPLOAD_FORBIDDEN):
+        return ToxResultHandling(_block=True, msg=msg)
+
+    def skip(self, msg=None):
+        return ToxResultHandling(_skip=True, msg=msg)
+
+
+@hookimpl(trylast=True)
+def devpiserver_on_toxresult_store(request, tox_result_handling):
+    return tox_result_handling
+
+
+@hookimpl(trylast=True)
+def devpiserver_on_toxresult_upload_forbidden(request, tox_result_handling):
+    return tox_result_handling
+
+
 def version_in_filename(version, filename):
     if version is None:
         # no version set, so skip check
@@ -419,9 +447,7 @@ def version_in_filename(version, filename):
     if version in filename:
         return True
     # PEP 427 escaped wheels
-    if re.sub(r"[^\w\d.]+", "_", version, flags=re.UNICODE) in filename:
-        return True
-    return False
+    return re.sub(r"[^\w\d.]+", "_", version, flags=re.UNICODE) in filename
 
 
 class PyPIView:
@@ -530,14 +556,25 @@ class PyPIView:
 
     @view_config(
         route_name="/{user}/{index}/+f/{relpath:.*}",
-        request_method="POST",
-        permission="toxresult_upload")
+        request_method="POST")
     def post_toxresult(self):
-        stage = self.context.stage
-        relpath = self.request.path_info.strip("/")
-        link = stage.get_link_from_entrypath(relpath)
-        if link is None or link.rel != "releasefile":
-            apireturn(404, message="no release file found at %s" % relpath)
+        if not self.request.has_permission("toxresult_upload"):
+            default_tox_result_handling = ToxResultHandling().block(TOXRESULT_UPLOAD_FORBIDDEN)
+            tox_result_handling = self.xom.config.hook.devpiserver_on_toxresult_upload_forbidden(
+                request=self.request, tox_result_handling=default_tox_result_handling)
+        else:
+            stage = self.context.stage
+            relpath = self.request.path_info.strip("/")
+            link = stage.get_link_from_entrypath(relpath)
+            if link is None or link.rel != "releasefile":
+                apireturn(404, message="no release file found at %s" % relpath)
+            default_tox_result_handling = ToxResultHandling()
+            tox_result_handling = self.xom.config.hook.devpiserver_on_toxresult_store(
+                request=self.request, tox_result_handling=default_tox_result_handling)
+        if tox_result_handling._block:
+            apireturn(403, tox_result_handling.msg)
+        if tox_result_handling._skip:
+            apireturn(200, tox_result_handling.msg)
         # the getjson call validates that we got valid json
         getjson(self.request)
         # but we store the original body
@@ -555,9 +592,7 @@ class PyPIView:
     def _use_absolute_urls(self):
         if self.xom.config.args.absolute_urls:
             return True
-        if 'HTTP_X_DEVPI_ABSOLUTE_URLS' in self.request.environ:
-            return True
-        return False
+        return 'HTTP_X_DEVPI_ABSOLUTE_URLS' in self.request.environ
 
     @view_config(route_name="/{user}/{index}/+simple/{project}")
     def simple_list_project_redirect(self):
@@ -745,18 +780,18 @@ class PyPIView:
         # depending on remote networks
         content_type = _select_simple_content_type(self.request)
         if content_type == SIMPLE_API_V1_JSON:
-            app_iter = self._simple_list_all_json_v1(stage, stage_results)
+            app_iter = self._simple_list_all_json_v1(stage_results)
         elif is_requested_by_installer(self.request):
-            app_iter = self._simple_list_all_installer(stage, stage_results)
+            app_iter = self._simple_list_all_installer(stage_results)
         else:
-            app_iter = self._simple_list_all(stage, stage_results)
+            app_iter = self._simple_list_all(stage.name, stage_results)
         return Response(
             app_iter=buffered_iterator(app_iter),
             content_type=content_type,
             vary=set(["Accept", "User-Agent"]))
 
-    def _simple_list_all(self, stage, stage_results):
-        title = "%s: simple list (including inherited indices)" % (stage.name)
+    def _simple_list_all(self, stage_name, stage_results):
+        title = f"{stage_name}: simple list (including inherited indices)"
         yield f"<!DOCTYPE html><html><head><title>{title}</title></head><body><h1>{title}</h1>".encode("utf-8")
         last_index = len(stage_results) - 1
         seen = set()
@@ -773,7 +808,7 @@ class PyPIView:
                         seen.add(name)
         yield "</body></html>".encode("utf-8")
 
-    def _simple_list_all_installer(self, stage, stage_results):
+    def _simple_list_all_installer(self, stage_results):
         yield "<!DOCTYPE html><html><body>".encode("utf-8")
         last_index = len(stage_results) - 1
         seen = set()
@@ -785,7 +820,7 @@ class PyPIView:
                         seen.add(name)
         yield "</body></html>".encode("utf-8")
 
-    def _simple_list_all_json_v1(self, stage, stage_results):
+    def _simple_list_all_json_v1(self, stage_results):
         yield '{"meta":{"api-version":"1.0"},"projects":['.encode("utf-8")
         last_index = len(stage_results) - 1
         seen = set()