arvados-python-client 2.7.4__tar.gz → 3.0.0__tar.gz

{arvados-python-client-2.7.4/arvados_python_client.egg-info → arvados-python-client-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 1.1
+Metadata-Version: 1.2
 Name: arvados-python-client
-Version: 2.7.4
+Version: 3.0.0
 Summary: Arvados client library
 Home-page: https://arvados.org
 Author: Arvados
@@ -55,24 +55,43 @@ Description: .. Copyright (C) The Arvados Authors. All rights reserved.
         
           arv-get --version
         
-        Installing on Debian systems
-        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-        
-        1. Add this Arvados repository to your sources list::
-        
-             deb http://apt.arvados.org/ buster main
-        
-        2. Update your package list.
-        
-        3. Install the ``python3-arvados-python-client`` package.
+        Installing on Debian and Ubuntu systems
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+        
+        Arvados publishes packages for Debian 11 "bullseye," Debian 12 "bookworm," Ubuntu 20.04 "focal," and Ubuntu 22.04 "jammy." You can install the Python SDK package on any of these distributions by running the following commands::
+        
+          sudo install -d /etc/apt/keyrings
+          sudo curl -fsSL -o /etc/apt/keyrings/arvados.asc https://apt.arvados.org/pubkey.gpg
+          sudo tee /etc/apt/sources.list.d/arvados.sources >/dev/null <<EOF
+          Types: deb
+          URIs: https://apt.arvados.org/$(lsb_release -cs)
+          Suites: $(lsb_release -cs)
+          Components: main
+          Signed-by: /etc/apt/keyrings/arvados.asc
+          EOF
+          sudo apt update
+          sudo apt install python3-arvados-python-client
+        
+        Installing on Red Hat, AlmaLinux, and Rocky Linux
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+        
+        Arvados publishes packages for RHEL 8 and distributions based on it. Note that these packages depend on, and will automatically enable, the Python 3.9 module. You can install the Python SDK package on any of these distributions by running the following commands::
+        
+          sudo tee /etc/yum.repos.d/arvados.repo >/dev/null <<'EOF'
+          [arvados]
+          name=Arvados
+          baseurl=http://rpm.arvados.org/RHEL/$releasever/os/$basearch/
+          gpgcheck=1
+          gpgkey=http://rpm.arvados.org/RHEL/RPM-GPG-KEY-arvados
+          EOF
+          sudo dnf install python3-arvados-python-client
         
         Configuration
         -------------
         
         This client software needs two pieces of information to connect to
         Arvados: the DNS name of the API server, and an API authorization
-        token.  You can set these in environment variables, or the file
-        ``$HOME/.config/arvados/settings.conf``.  `The Arvados user
+        token. `The Arvados user
         documentation
         <http://doc.arvados.org/user/reference/api-tokens.html>`_ describes
         how to find this information in the Arvados Workbench, and install it
@@ -89,3 +108,4 @@ Description: .. Copyright (C) The Arvados Authors. All rights reserved.
         
 Platform: UNKNOWN
 Classifier: Programming Language :: Python :: 3
+Requires-Python: ~=3.8

{arvados-python-client-2.7.4 → arvados-python-client-3.0.0}/README.rst

@@ -46,24 +46,43 @@ You can test the change by running::
 
   arv-get --version
 
-Installing on Debian systems
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-1. Add this Arvados repository to your sources list::
-
-     deb http://apt.arvados.org/ buster main
-
-2. Update your package list.
-
-3. Install the ``python3-arvados-python-client`` package.
+Installing on Debian and Ubuntu systems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Arvados publishes packages for Debian 11 "bullseye," Debian 12 "bookworm," Ubuntu 20.04 "focal," and Ubuntu 22.04 "jammy." You can install the Python SDK package on any of these distributions by running the following commands::
+
+  sudo install -d /etc/apt/keyrings
+  sudo curl -fsSL -o /etc/apt/keyrings/arvados.asc https://apt.arvados.org/pubkey.gpg
+  sudo tee /etc/apt/sources.list.d/arvados.sources >/dev/null <<EOF
+  Types: deb
+  URIs: https://apt.arvados.org/$(lsb_release -cs)
+  Suites: $(lsb_release -cs)
+  Components: main
+  Signed-by: /etc/apt/keyrings/arvados.asc
+  EOF
+  sudo apt update
+  sudo apt install python3-arvados-python-client
+
+Installing on Red Hat, AlmaLinux, and Rocky Linux
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Arvados publishes packages for RHEL 8 and distributions based on it. Note that these packages depend on, and will automatically enable, the Python 3.9 module. You can install the Python SDK package on any of these distributions by running the following commands::
+
+  sudo tee /etc/yum.repos.d/arvados.repo >/dev/null <<'EOF'
+  [arvados]
+  name=Arvados
+  baseurl=http://rpm.arvados.org/RHEL/$releasever/os/$basearch/
+  gpgcheck=1
+  gpgkey=http://rpm.arvados.org/RHEL/RPM-GPG-KEY-arvados
+  EOF
+  sudo dnf install python3-arvados-python-client
 
 Configuration
 -------------
 
 This client software needs two pieces of information to connect to
 Arvados: the DNS name of the API server, and an API authorization
-token.  You can set these in environment variables, or the file
-``$HOME/.config/arvados/settings.conf``.  `The Arvados user
+token. `The Arvados user
 documentation
 <http://doc.arvados.org/user/reference/api-tokens.html>`_ describes
 how to find this information in the Arvados Workbench, and install it

arvados-python-client-3.0.0/arvados/__init__.py

@@ -0,0 +1,69 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+"""Arvados Python SDK
+
+This module provides the entire Python SDK for Arvados. The most useful modules
+include:
+
+* arvados.api - This module provides the `arvados.api.api` function to
+  construct an Arvados REST API client, as well as other classes and functions
+  that support it. You can call the `arvados.api` module just like a function
+  as a shortcut for calling `arvados.api.api`.
+
+* arvados.api_resources - The methods on an Arvados REST API client are
+  generated dynamically at runtime. This module documents those methods and
+  return values for the current version of Arvados. This module does not
+  implement anything so you don't need to import it, but it's a helpful
+  reference to understand how to use the Arvados REST API client.
+
+* arvados.collection - The `arvados.collection.Collection` class provides a
+  high-level interface to read and write collections. It coordinates sending
+  data to and from Keep, and synchronizing updates with the collection object.
+
+* arvados.util - Utility functions to use mostly in conjunction with the API
+  client object and the results it returns.
+
+Other submodules provide lower-level functionality.
+"""
+
+import logging as stdliblog
+import os
+import sys
+import types
+
+from collections import UserDict
+
+from . import api, errors, util
+from .api import api_from_config, http_cache
+from .collection import CollectionReader
+from arvados.keep import *
+from .logging import log_format, log_date_format, log_handler
+from .retry import RetryLoop
+
+# Backwards compatibility shims: these modules used to get pulled in after
+# `import arvados` with previous versions of the SDK. We must keep the names
+# accessible even though there's no longer any functional need for them.
+from . import cache
+from . import safeapi
+
+# Previous versions of the PySDK used to say `from .api import api`.  This
+# made it convenient to call the API client constructor, but difficult to
+# access the rest of the `arvados.api` module. The magic below fixes that
+# bug while retaining backwards compatibility: `arvados.api` is now the
+# module and you can import it normally, but we make that module callable so
+# all the existing code that says `arvados.api('v1', ...)` still works.
+class _CallableAPIModule(api.__class__):
+    __call__ = staticmethod(api.api)
+api.__class__ = _CallableAPIModule
+
+# Override logging module pulled in via `from ... import *`
+# so users can `import arvados.logging`.
+logging = sys.modules['arvados.logging']
+
+# Set up Arvados logging based on the user's configuration.
+# All Arvados code should log under the arvados hierarchy.
+logger = stdliblog.getLogger('arvados')
+logger.addHandler(log_handler)
+logger.setLevel(stdliblog.DEBUG if config.get('ARVADOS_DEBUG')
+                else stdliblog.WARNING)

arvados-python-client-3.0.0/arvados/_internal/__init__.py

@@ -0,0 +1,83 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+"""Arvados internal utilities
+
+Everything in `arvados._internal` is support code for the Arvados Python SDK
+and tools. Nothing in this module is intended to be part of the public-facing
+SDK API. Classes and functions in this module may be changed or removed at any
+time.
+"""
+
+import functools
+import re
+import time
+import warnings
+
+class Timer:
+    def __init__(self, verbose=False):
+        self.verbose = verbose
+
+    def __enter__(self):
+        self.start = time.time()
+        return self
+
+    def __exit__(self, *args):
+        self.end = time.time()
+        self.secs = self.end - self.start
+        self.msecs = self.secs * 1000  # millisecs
+        if self.verbose:
+            print('elapsed time: %f ms' % self.msecs)
+
+
+def deprecated(version=None, preferred=None):
+    """Mark a callable as deprecated in the SDK
+
+    This will wrap the callable to emit as a DeprecationWarning
+    and add a deprecation notice to its docstring.
+
+    If the following arguments are given, they'll be included in the
+    notices:
+
+    * preferred: str | None --- The name of an alternative that users should
+      use instead.
+
+    * version: str | None --- The version of Arvados when the callable is
+      scheduled to be removed.
+    """
+    if version is None:
+        version = ''
+    else:
+        version = f' and scheduled to be removed in Arvados {version}'
+    if preferred is None:
+        preferred = ''
+    else:
+        preferred = f' Prefer {preferred} instead.'
+    def deprecated_decorator(func):
+        fullname = f'{func.__module__}.{func.__qualname__}'
+        parent, _, name = fullname.rpartition('.')
+        if name == '__init__':
+            fullname = parent
+        warning_msg = f'{fullname} is deprecated{version}.{preferred}'
+        @functools.wraps(func)
+        def deprecated_wrapper(*args, **kwargs):
+            warnings.warn(warning_msg, DeprecationWarning, 2)
+            return func(*args, **kwargs)
+        # Get func's docstring without any trailing newline or empty lines.
+        func_doc = re.sub(r'\n\s*$', '', func.__doc__ or '')
+        match = re.search(r'\n([ \t]+)\S', func_doc)
+        indent = '' if match is None else match.group(1)
+        warning_doc = f'\n\n{indent}.. WARNING:: Deprecated\n{indent}   {warning_msg}'
+        # Make the deprecation notice the second "paragraph" of the
+        # docstring if possible. Otherwise append it.
+        docstring, count = re.subn(
+            rf'\n[ \t]*\n{indent}',
+            f'{warning_doc}\n\n{indent}',
+            func_doc,
+            count=1,
+        )
+        if not count:
+            docstring = f'{func_doc.lstrip()}{warning_doc}'
+        deprecated_wrapper.__doc__ = docstring
+        return deprecated_wrapper
+    return deprecated_decorator

arvados-python-client-3.0.0/arvados/_internal/basedirs.py

@@ -0,0 +1,186 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+"""Base directories utility module
+
+This module provides a set of classes useful to search and manipulate base
+directory defined by systemd and the XDG specification. Most users will just
+instantiate and use `BaseDirectories`.
+"""
+
+import dataclasses
+import enum
+import itertools
+import logging
+import os
+import shlex
+import stat
+
+from pathlib import Path, PurePath
+from typing import (
+    Iterator,
+    Mapping,
+    Optional,
+    Union,
+)
+
+logger = logging.getLogger('arvados')
+
+@dataclasses.dataclass
+class BaseDirectorySpec:
+    """Parse base directories
+
+    A BaseDirectorySpec defines all the environment variable keys and defaults
+    related to a set of base directories (cache, config, state, etc.). It
+    provides pure methods to parse environment settings into valid paths.
+    """
+    systemd_key: str
+    xdg_home_key: str
+    xdg_home_default: PurePath
+    xdg_dirs_key: Optional[str] = None
+    xdg_dirs_default: str = ''
+
+    @staticmethod
+    def _abspath_from_env(env: Mapping[str, str], key: str) -> Optional[Path]:
+        try:
+            path = Path(env[key])
+        except (KeyError, ValueError):
+            ok = False
+        else:
+            ok = path.is_absolute()
+        return path if ok else None
+
+    @staticmethod
+    def _iter_abspaths(value: str) -> Iterator[Path]:
+        for path_s in value.split(':'):
+            path = Path(path_s)
+            if path.is_absolute():
+                yield path
+
+    def iter_systemd(self, env: Mapping[str, str]) -> Iterator[Path]:
+        return self._iter_abspaths(env.get(self.systemd_key, ''))
+
+    def iter_xdg(self, env: Mapping[str, str], subdir: PurePath) -> Iterator[Path]:
+        yield self.xdg_home(env, subdir)
+        if self.xdg_dirs_key is not None:
+            for path in self._iter_abspaths(env.get(self.xdg_dirs_key) or self.xdg_dirs_default):
+                yield path / subdir
+
+    def xdg_home(self, env: Mapping[str, str], subdir: PurePath) -> Path:
+        return (
+            self._abspath_from_env(env, self.xdg_home_key)
+            or self.xdg_home_default_path(env)
+        ) / subdir
+
+    def xdg_home_default_path(self, env: Mapping[str, str]) -> Path:
+        return (self._abspath_from_env(env, 'HOME') or Path.home()) / self.xdg_home_default
+
+    def xdg_home_is_customized(self, env: Mapping[str, str]) -> bool:
+        xdg_home = self._abspath_from_env(env, self.xdg_home_key)
+        return xdg_home is not None and xdg_home != self.xdg_home_default_path(env)
+
+
+class BaseDirectorySpecs(enum.Enum):
+    """Base directory specifications
+
+    This enum provides easy access to the standard base directory settings.
+    """
+    CACHE = BaseDirectorySpec(
+        'CACHE_DIRECTORY',
+        'XDG_CACHE_HOME',
+        PurePath('.cache'),
+    )
+    CONFIG = BaseDirectorySpec(
+        'CONFIGURATION_DIRECTORY',
+        'XDG_CONFIG_HOME',
+        PurePath('.config'),
+        'XDG_CONFIG_DIRS',
+        '/etc/xdg',
+    )
+    STATE = BaseDirectorySpec(
+        'STATE_DIRECTORY',
+        'XDG_STATE_HOME',
+        PurePath('.local', 'state'),
+    )
+
+
+class BaseDirectories:
+    """Resolve paths from a base directory spec
+
+    Given a BaseDirectorySpec, this class provides stateful methods to find
+    existing files and return the most-preferred directory for writing.
+    """
+    _STORE_MODE = stat.S_IFDIR | stat.S_IWUSR
+
+    def __init__(
+            self,
+            spec: Union[BaseDirectorySpec, BaseDirectorySpecs, str],
+            env: Mapping[str, str]=os.environ,
+            xdg_subdir: Union[os.PathLike, str]='arvados',
+    ) -> None:
+        if isinstance(spec, str):
+            spec = BaseDirectorySpecs[spec].value
+        elif isinstance(spec, BaseDirectorySpecs):
+            spec = spec.value
+        self._spec = spec
+        self._env = env
+        self._xdg_subdir = PurePath(xdg_subdir)
+
+    def search(self, name: str) -> Iterator[Path]:
+        any_found = False
+        for search_path in itertools.chain(
+                self._spec.iter_systemd(self._env),
+                self._spec.iter_xdg(self._env, self._xdg_subdir),
+        ):
+            path = search_path / name
+            if path.exists():
+                yield path
+                any_found = True
+        # The rest of this function is dedicated to warning the user if they
+        # have a custom XDG_*_HOME value that prevented the search from
+        # succeeding. This should be rare.
+        if any_found or not self._spec.xdg_home_is_customized(self._env):
+            return
+        default_home = self._spec.xdg_home_default_path(self._env)
+        default_path = Path(self._xdg_subdir / name)
+        if not (default_home / default_path).exists():
+            return
+        if self._spec.xdg_dirs_key is None:
+            suggest_key = self._spec.xdg_home_key
+            suggest_value = default_home
+        else:
+            suggest_key = self._spec.xdg_dirs_key
+            cur_value = self._env.get(suggest_key, '')
+            value_sep = ':' if cur_value else ''
+            suggest_value = f'{cur_value}{value_sep}{default_home}'
+        logger.warning(
+            "\
+%s was not found under your configured $%s (%s), \
+but does exist at the default location (%s) - \
+consider running this program with the environment setting %s=%s\
+",
+            default_path,
+            self._spec.xdg_home_key,
+            self._spec.xdg_home(self._env, ''),
+            default_home,
+            suggest_key,
+            shlex.quote(suggest_value),
+        )
+
+    def storage_path(
+            self,
+            subdir: Union[str, os.PathLike]=PurePath(),
+            mode: int=0o700,
+    ) -> Path:
+        for path in self._spec.iter_systemd(self._env):
+            try:
+                mode = path.stat().st_mode
+            except OSError:
+                continue
+            if (mode & self._STORE_MODE) == self._STORE_MODE:
+                break
+        else:
+            path = self._spec.xdg_home(self._env, self._xdg_subdir)
+        path /= subdir
+        path.mkdir(parents=True, exist_ok=True, mode=mode)
+        return path

{arvados-python-client-2.7.4/arvados → arvados-python-client-3.0.0/arvados/_internal}/diskcache.py

@@ -39,7 +39,7 @@ class DiskCacheSlot(object):
         # just a few pages at a time), reducing the number of page
         # faults and improving performance by 4x compared to not
         # calling madvise.
-        if self.content and hasattr(mmap.mmap, 'madvise'):
+        if self.content:
             self.content.madvise(mmap.MADV_WILLNEED)
         return self.content
 

{arvados-python-client-2.7.4/arvados → arvados-python-client-3.0.0/arvados/_internal}/http_to_keep.py

@@ -2,24 +2,21 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-from __future__ import division
-from future import standard_library
-standard_library.install_aliases()
-
-import email.utils
-import time
+import calendar
+import dataclasses
 import datetime
-import re
-import arvados
-import arvados.collection
-import urllib.parse
+import email.utils
 import logging
-import calendar
+import re
+import time
+import typing
 import urllib.parse
+
 import pycurl
-import dataclasses
-import typing
-from arvados._pycurlhelper import PyCurlHelper
+
+import arvados
+import arvados.collection
+from .pycurl import PyCurlHelper
 
 logger = logging.getLogger('arvados.http_import')
 

arvados-python-client-3.0.0/arvados/_internal/report_template.py

@@ -0,0 +1,114 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+try:
+    from html import escape
+except ImportError:
+    from cgi import escape
+
+import json
+from typing import ItemsView
+
+class ReportTemplate(object):
+    """Base class for HTML reports produced by Arvados reporting tools.
+
+    Used by crunchstat-summary and cluster-activity.
+
+    """
+
+    STYLE = '''
+    <style>
+        body {
+          background: #fafafa;
+          font-family: "Roboto", "Helvetica", "Arial", sans-serif;
+          font-size: 0.875rem;
+          color: rgba(0, 0, 0, 0.87);
+          font-weight: 400;
+        }
+        .card {
+          background: #ffffff;
+          box-shadow: 0px 1px 5px 0px rgba(0,0,0,0.2),0px 2px 2px 0px rgba(0,0,0,0.14),0px 3px 1px -2px rgba(0,0,0,0.12);
+          border-radius: 4px;
+          margin: 20px;
+        }
+        .content {
+          padding: 2px 16px 8px 16px;
+        }
+        table {
+          border-spacing: 0px;
+        }
+        tr {
+          height: 36px;
+          text-align: left;
+        }
+        th {
+          padding-right: 4em;
+          border-top: 1px solid rgba(224, 224, 224, 1);
+        }
+        td {
+          padding-right: 2em;
+          border-top: 1px solid rgba(224, 224, 224, 1);
+        }
+        #chart {
+          margin-left: -20px;
+        }
+    </style>
+    '''
+
+    def __init__(self, label):
+        self.label = label
+        self.cards = []
+
+    def cardlist(self, items):
+        if not isinstance(items, list):
+            items = [items]
+
+        return "\n".join(
+                """
+                <div class="card">
+                  <div class="content">
+{}
+                  </div>
+                </div>""".format(i) for i in items)
+
+    def html(self):
+        return '''<!doctype html>
+<html>
+  <head>
+    <title>{label}</title>
+
+{js}
+
+{style}
+
+{header}
+
+  </head>
+
+  <body>
+  <div class="card">
+    <div class="content">
+      <h1>{label}</h1>
+    </div>
+  </div>
+
+{cards}
+
+  </body>
+</html>
+        '''.format(label=escape(self.label),
+                   js=self.js(),
+                   style=self.style(),
+                   header=self.headHTML(),
+                   cards=self.cardlist(self.cards))
+
+    def js(self):
+        return ''
+
+    def style(self):
+        return self.STYLE
+
+    def headHTML(self):
+        """Return extra HTML text to include in HEAD."""
+        return ''