taskcluster-taskgraph 22.0.0__tar.gz → 23.1.0__tar.gz

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/.pre-commit-config.yaml

@@ -13,13 +13,13 @@ repos:
         exclude: template
       - id: check-added-large-files
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.8
+    rev: v0.15.12
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.11.2
+    rev: 0.11.8
     hooks:
       - id: uv-lock
   - repo: https://github.com/adrienverge/yamllint.git

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Change Log
 
+## [23.1.0] - 2026-05-26
+
+### Added
+
+- Updated worker types in the `.taskcluster.yml` template for the d2g migration
+
+### Fixed
+
+- `TaskGraph.from_json` now skips dependencies pointing at external taskIds (e.g. from replaced or cached tasks) when rebuilding edges, instead of producing edges to non-existent graph nodes
+- Schema validation errors now display a clean message instead of a noisy traceback
+- Pinned `taskcluster-client` to work around an upstream issue
+- `fetch_graph_and_labels` now handles `HTTPError` instances where `response` is `None`
+
+## [23.0.0] - 2026-05-04
+
+### Added
+
+- Support for defining schemas as dicts (in addition to the class-based approach)
+- New `disabled_actions` graph configuration to prevent specific actions from being generated
+
+### Changed
+
+- BREAKING CHANGE: Converted parameter schema from voluptuous to msgspec
+
+### Fixed
+
+- `taskgraph full -J --tasks <regex>` now displays full dependencies instead of a filtered subset
+- `fetch-content` now percent-encodes artifact names, fixing downloads of artifacts containing spaces or other special characters
+
 ## [22.0.0] - 2026-04-16
 
 ### Fixed

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: taskcluster-taskgraph
-Version: 22.0.0
+Version: 23.1.0
 Summary: Build taskcluster taskgraphs
 Project-URL: Repository, https://github.com/taskcluster/taskgraph
 Project-URL: Issues, https://github.com/taskcluster/taskgraph/issues

taskcluster_taskgraph-23.1.0/docs/howto/define-schemas.rst

@@ -0,0 +1,156 @@
+Define Schemas
+==============
+
+:doc:`Transforms </concepts/transforms>` can define schemas to validate task data at each step of the
+pipeline. Taskgraph uses `msgspec`_ under the hood, and provides a
+:class:`~taskgraph.util.schema.Schema` base class that integrates with
+:meth:`TransformSequence.add_validate() <taskgraph.transforms.base.TransformSequence.add_validate>`.
+
+There are two ways to define a schema: the **class based** approach and the
+**dict based** approach. Both produce equivalent results; which one you prefer
+is a matter of style.
+
+.. _msgspec: https://jcristharif.com/msgspec/
+
+
+Class Based Schemas
+-------------------
+
+Subclass :class:`~taskgraph.util.schema.Schema` and declare fields as class
+attributes with type annotations:
+
+.. code-block:: python
+
+   from typing import Optional
+   from taskgraph.transforms.base import TransformSequence
+   from taskgraph.util.schema import Schema
+
+   class MySubConfig(Schema):
+       total_num: int
+       fields: list[str] = []
+
+   class MySchema(Schema, forbid_unknown_fields=False):
+       config: Optional[MySubConfig] = None
+
+   transforms = TransformSequence()
+   transforms.add_validate(MySchema)
+
+A few things to note:
+
+- Field names use ``snake_case`` in Python but are **automatically renamed to
+  ``kebab-case``** in YAML. So ``total_num`` in Python matches
+  ``total-num`` in YAML.
+- ``Optional[T]`` fields default to ``None`` unless you supply an explicit
+  default.
+- Fields without a default are **required**.
+- ``forbid_unknown_fields=True`` (the default) causes validation to fail if the
+  task data contains keys that are not declared in the schema. Set it to
+  ``False`` on outer schemas so that fields belonging to later transforms are
+  not rejected.
+
+
+Dict Based Schemas
+------------------
+
+Call :meth:`Schema.from_dict() <taskgraph.util.schema.Schema.from_dict>` with a
+dictionary mapping field names to ``type`` or ``(type, default)`` tuples:
+
+.. code-block:: python
+
+   from typing import Optional, Union
+   from taskgraph.transforms.base import TransformSequence
+   from taskgraph.util.schema import Schema
+
+   MySchema = Schema.from_dict(
+       {
+           "config": Schema.from_dict(
+               {
+                   "total-num": int,
+                   "fields": list[str] = []
+               },
+               optional=True,
+           ),
+       },
+       forbid_unknown_fields=False,
+   )
+
+   transforms = TransformSequence()
+   transforms.add_validate(MySchema)
+
+This example is equivalent to the first example. One advantage with the dict based approach
+is that you can write keys in **kebab-case** directly.
+
+Field specifications follow these rules:
+
+- A bare type (e.g. ``str``) means the field is required.
+- ``Optional[T]`` means the field is optional and defaults to ``None``.
+- A ``(type, default)`` tuple supplies an explicit default, e.g.
+  ``(list[str], [])``.
+
+Keyword arguments to ``from_dict`` are forwarded to ``msgspec.defstruct``.
+The most commonly used ones are ``name`` (for better error messages) and
+``forbid_unknown_fields``.
+
+.. note::
+   ``Schema.from_dict`` does **not** apply ``rename="kebab"`` automatically,
+   because you can express the kebab-case names directly in the dict keys.
+   Underscores in dict keys stay as underscores and dashes become valid
+   kebab-case field names.
+
+
+Nesting Schemas
+---------------
+
+Both approaches support nesting:
+
+.. code-block:: python
+
+   # Class-based nesting
+   class Inner(Schema):
+       value: str
+
+   class Outer(Schema, forbid_unknown_fields=False, kw_only=True):
+       inner: Optional[Inner] = None
+
+   # Dict-based nesting
+   Outer = Schema.from_dict(
+       {
+           "inner": Schema.from_dict({"value": str}, optional=True),
+       },
+       forbid_unknown_fields=False,
+   )
+
+Pass ``optional=True`` to ``from_dict`` to make the whole nested schema
+optional. This is necessary as function calls are not allowed in type
+annotations, so ``Optional[Schema.from_dict(...)]`` is not valid Python.
+
+
+Mutually Exclusive Fields
+-------------------------
+
+Use the ``exclusive`` keyword to declare groups of fields where at most one
+may be set at a time:
+
+.. code-block:: python
+
+   # Class-based
+   class MySchema(Schema, exclusive=[["field_a", "field_b"]]):
+       field_a: Optional[str] = None
+       field_b: Optional[str] = None
+
+   # Dict-based
+   MySchema = Schema.from_dict(
+       {
+           "field-a": Optional[str],
+           "field-b": Optional[str],
+       },
+       exclusive=[["field_a", "field_b"]],
+   )
+
+``exclusive`` takes a list of groups, where each group is a list of field
+names (Python ``snake_case``). A validation error is raised if more than one
+field in a group is set.
+
+.. note::
+   When using ``exclusive`` with the dict-based approach, refer to fields by
+   their Python attribute names (``snake_case``), not their YAML keys.

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/docs/howto/index.rst

@@ -10,6 +10,7 @@ A collection of how-to guides.
    run-locally
    debugging
    bootstrap-taskgraph
+   define-schemas
    resolve-keyed-by
    use-fetches
    docker

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/docs/reference/migrations.rst

@@ -3,6 +3,13 @@ Migration Guide
 
 This page can help when migrating Taskgraph across major versions.
 
+22.x -> 23.x
+------------
+
+* :func:`taskgraph.parameters.extend_parameters_schema` now requires a msgspec
+  ``Schema`` subclass instead of a voluptuous schema dict. Convert any custom
+  parameter schemas to msgspec.
+
 21.x -> 22.x
 ------------
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/pyproject.toml

@@ -1,7 +1,7 @@
 ### Project
 [project]
 name = "taskcluster-taskgraph"
-version = "22.0.0"
+version = "23.1.0"
 description = "Build taskcluster taskgraphs"
 readme = "README.rst"
 authors = [

taskcluster_taskgraph-23.1.0/renovate.json

@@ -0,0 +1,6 @@
+{
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  "extends": [
+    "github>mozilla-releng/reps:renovate-preset.json5"
+  ]
+}

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/actions/registry.py

@@ -273,12 +273,27 @@ def render_actions_json(parameters, graph_config, decision_task_id):
             artifact.
     """
     assert isinstance(parameters, Parameters), "requires instance of Parameters"
+
+    all_actions = _get_actions(graph_config)
+    disabled = set(graph_config["taskgraph"].get("disabled-actions") or [])
+    known_cb_names = {action.cb_name for action in all_actions}
+    unknown = disabled - known_cb_names
+    if unknown:
+        raise ValueError(
+            "Unknown action(s) in `taskgraph.disabled-actions`: "
+            f"{sorted(unknown)}. Known actions: {sorted(known_cb_names)}"
+        )
+
     actions = []
-    for action in sorted(_get_actions(graph_config), key=lambda action: action.order):
+    for action in sorted(all_actions, key=lambda action: action.order):
+        if action.cb_name in disabled:
+            continue
+
         action = action.action_builder(parameters, graph_config, decision_task_id)
         if action:
             assert is_json(action), "action must be a JSON compatible object"
             actions.append(action)
+
     return {
         "version": 1,
         "variables": {},

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/actions/util.py

@@ -85,7 +85,7 @@ def fetch_graph_and_labels(parameters, graph_config, task_group_id=None):
                 run_label_to_id = get_artifact(task_id, "public/label-to-taskid.json")
                 label_to_taskid.update(run_label_to_id)  # type: ignore
             except HTTPError as e:
-                if e.response.status_code != 404:
+                if e.response is None or e.response.status_code != 404:
                     raise
                 logger.debug(f"No label-to-taskid.json found for {task_id}: {e}")
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/config.py

@@ -57,6 +57,9 @@ class TaskgraphConfig(Schema):
     index_path_regexes: Optional[list[str]] = None
     # Configuration related to the 'run' transforms.
     run: Optional[RunConfig] = None
+    # List of action `cb_name`s to omit from `actions.json`,
+    # e.g. `["retrigger", "retrigger-disabled", "rerun"]`.
+    disabled_actions: Optional[list[str]] = None
 
     def __post_init__(self):
         # Validate repositories has at least 1 entry (was All(..., Length(min=1)))

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/generator.py

@@ -27,6 +27,7 @@ from .task import Task
 from .taskgraph import TaskGraph
 from .transforms.base import TransformConfig, TransformSequence
 from .util.python_path import find_object
+from .util.schema import SchemaValidationError
 from .util.verify import verifications
 from .util.yaml import load_yaml
 
@@ -303,6 +304,9 @@ class TaskGraphGenerator:
                     },
                     self._write_artifacts,
                 )
+            except SchemaValidationError as exc:
+                logger.error(f"Error loading tasks for kind {kind_name}:\n{exc}")
+                raise
             except Exception:
                 logger.exception(f"Error loading tasks for kind {kind_name}:")
                 raise
@@ -358,6 +362,16 @@ class TaskGraphGenerator:
                 for future in done:
                     if exc := future.exception():
                         executor.shutdown(wait=False, cancel_futures=True)
+                        kind_name = futures_to_kind.get(future, "?")
+                        if isinstance(exc, SchemaValidationError):
+                            logger.error(
+                                f"Error loading tasks for kind {kind_name}:\n{exc}"
+                            )
+                        else:
+                            logger.error(
+                                f"Error loading tasks for kind {kind_name}:",
+                                exc_info=(type(exc), exc, exc.__traceback__),
+                            )
                         raise exc
                     kind = futures_to_kind.pop(future)
                     futures.remove(future)

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/main.py

@@ -22,6 +22,8 @@ from urllib.parse import urlparse
 import appdirs
 import yaml
 
+from taskgraph.util.schema import SchemaValidationError
+
 Command = namedtuple("Command", ["func", "args", "kwargs", "defaults"])
 commands = {}
 
@@ -1207,6 +1209,9 @@ def main(args=sys.argv[1:]):
     args = parser.parse_args(args)
     try:
         return args.command(vars(args))
+    except SchemaValidationError:
+        # Message was already logged; skip the traceback for user input errors.
+        sys.exit(1)
     except Exception:
         traceback.print_exc()
         sys.exit(1)

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/parameters.py

@@ -10,16 +10,17 @@ from datetime import datetime
 from io import BytesIO
 from pprint import pformat
 from subprocess import CalledProcessError
+from typing import Optional, Union
 from unittest.mock import Mock
 from urllib.parse import urlparse
 from urllib.request import urlopen
 
 import mozilla_repo_urls
-from voluptuous import ALLOW_EXTRA, Any, Optional, Required, Schema
+import msgspec
 
 from taskgraph.util import json, yaml
 from taskgraph.util.readonlydict import ReadOnlyDict
-from taskgraph.util.schema import validate_schema
+from taskgraph.util.schema import Schema, validate_schema
 from taskgraph.util.taskcluster import find_task_id, get_artifact_url
 from taskgraph.util.vcs import get_repository
 
@@ -30,43 +31,50 @@ class ParameterMismatch(Exception):
 
 #: Schema for base parameters.
 #: Please keep this list sorted and in sync with docs/reference/parameters.rst
-base_schema = Schema(
+base_schema = Schema.from_dict(
     {
-        Required("base_repository"): str,
-        Optional("base_ref"): str,
-        Required("base_rev"): str,
-        Required("build_date"): int,
-        Required("build_number"): int,
-        Required("do_not_optimize"): [str],
-        Required("enable_always_target"): Any(bool, [str]),
-        Required("existing_tasks"): {str: str},
-        Required("files_changed"): [str],
-        Required("filters"): [str],
-        Required("head_ref"): str,
-        Required("head_repository"): str,
-        Required("head_rev"): str,
-        Required("head_tag"): str,
-        Required("level"): str,
-        Required("moz_build_date"): str,
-        Required("next_version"): Any(str, None),
-        Required("optimize_strategies"): Any(str, None),
-        Required("optimize_target_tasks"): bool,
-        Required("owner"): str,
-        Required("project"): str,
-        Required("pushdate"): int,
-        Required("pushlog_id"): str,
-        Required("repository_type"): str,
+        "base_repository": str,
+        "base_ref": Optional[str],
+        "base_rev": str,
+        "build_date": int,
+        "build_number": int,
+        "do_not_optimize": list[str],
+        "enable_always_target": Union[bool, list[str]],
+        "existing_tasks": dict[str, str],
+        "files_changed": list[str],
+        "filters": list[str],
+        "head_ref": str,
+        "head_repository": str,
+        "head_rev": str,
+        "head_tag": str,
+        "level": str,
+        "moz_build_date": str,
+        "next_version": Optional[str],
+        "optimize_strategies": Optional[str],
+        "optimize_target_tasks": bool,
+        "owner": str,
+        "project": str,
+        "pushdate": int,
+        "pushlog_id": str,
+        "repository_type": str,
         # target-kinds is not included, since it should never be
         # used at run-time
-        Required("target_tasks_method"): str,
-        Required("tasks_for"): str,
-        Required("version"): Any(str, None),
-        Optional("code-review"): {
-            Required("phabricator-build-target"): str,
-        },
-    }
+        "target_tasks_method": str,
+        "tasks_for": str,
+        "version": Optional[str],
+        "code-review": Schema.from_dict(
+            {"phabricator-build-target": str},
+            name="CodeReviewConfig",
+            optional=True,
+        ),
+    },
+    name="BaseParametersSchema",
+    forbid_unknown_fields=False,
+    kw_only=True,
 )
 
+_parameter_extensions: list = []
+
 
 def get_contents(path):
     with open(path) as fh:
@@ -83,11 +91,21 @@ def _get_defaults(repo_root=None):
     repo_path = repo_root or os.getcwd()
     try:
         repo = get_repository(repo_path)
-    except RuntimeError:
-        # Use fake values if no repo is detected.
-        repo = Mock(branch="", head_rev="", tool="git")
+        # Resolve git-backed attributes eagerly so any subprocess failures
+        # (e.g. Windows "dubious ownership" when safe.directory isn't honored)
+        # are caught by the except below instead of escaping later.
+        branch = repo.branch
+        head_rev = repo.head_rev
+        tool = repo.tool
+        files_changed = repo.get_changed_files("AM")
+    except (RuntimeError, CalledProcessError):
+        # Use fake values if no repo is detected or git refuses to operate.
+        repo = Mock()
         repo.get_url.return_value = ""
-        repo.get_changed_files.return_value = []
+        branch = ""
+        head_rev = ""
+        tool = "git"
+        files_changed = []
 
     try:
         repo_url = repo.get_url()
@@ -110,11 +128,11 @@ def _get_defaults(repo_root=None):
         "do_not_optimize": [],
         "enable_always_target": True,
         "existing_tasks": {},
-        "files_changed": lambda: repo.get_changed_files("AM"),
+        "files_changed": files_changed,
         "filters": ["target_tasks_method"],
-        "head_ref": repo.branch or repo.head_rev,
+        "head_ref": branch or head_rev,
         "head_repository": repo_url,
-        "head_rev": repo.head_rev,
+        "head_rev": head_rev,
         "head_tag": "",
         "level": "3",
         "moz_build_date": datetime.now().strftime("%Y%m%d%H%M%S"),
@@ -125,7 +143,7 @@ def _get_defaults(repo_root=None):
         "project": project,
         "pushdate": int(time.time()),
         "pushlog_id": "0",
-        "repository_type": repo.tool,
+        "repository_type": tool,
         "target_tasks_method": "default",
         "tasks_for": "",
         "version": get_version(repo_path),
@@ -143,19 +161,27 @@ def extend_parameters_schema(schema, defaults_fn=None):
     graph-configuration.
 
     Args:
-        schema (Schema): The voluptuous.Schema object used to describe extended
-            parameters.
+        schema: A msgspec ``Schema`` subclass describing extended parameters.
         defaults_fn (function): A function which takes no arguments and returns a
             dict mapping parameter name to default value in the
             event strict=False (optional).
     """
-    global base_schema
     global defaults_functions
-    base_schema = base_schema.extend(schema)
+    if not (isinstance(schema, type) and issubclass(schema, msgspec.Struct)):
+        raise TypeError(
+            "extend_parameters_schema requires a msgspec Schema subclass; "
+            f"got {type(schema).__name__}"
+        )
+    _parameter_extensions.append(schema)
     if defaults_fn:
         defaults_functions.append(defaults_fn)
 
 
+def _schema_key_names(schema) -> set:
+    """Return the data-level field names declared by a parameters schema."""
+    return {f.encode_name for f in msgspec.structs.fields(schema)}
+
+
 class Parameters(ReadOnlyDict):
     """An immutable dictionary with nicer KeyError messages on failure"""
 
@@ -214,11 +240,30 @@ class Parameters(ReadOnlyDict):
         return kwargs
 
     def check(self):
-        schema = (
-            base_schema if self.strict else base_schema.extend({}, extra=ALLOW_EXTRA)
-        )
+        data = dict(self.copy())
         try:
-            validate_schema(schema, self.copy(), "Invalid parameters:")
+            # Validate core fields against just the subset of data owned by the
+            # base schema. Extension keys are validated separately below, and a
+            # strict-mode check rejects anything unknown to either.
+            base_keys = _schema_key_names(base_schema)
+            base_data = {k: v for k, v in data.items() if k in base_keys}
+            validate_schema(base_schema, base_data, "Invalid parameters:")
+
+            # Validate each registered extension against the keys it declares.
+            allowed = set(base_keys)
+            for ext in _parameter_extensions:
+                ext_keys = _schema_key_names(ext)
+                allowed |= ext_keys
+                ext_data = {k: data[k] for k in ext_keys if k in data}
+                validate_schema(ext, ext_data, "Invalid parameters:")
+
+            # Strict mode: reject any data key not covered by base or extensions.
+            if self.strict:
+                unknown = sorted(set(data) - allowed)
+                if unknown:
+                    raise Exception(
+                        "Invalid parameters:\nunknown keys: " + ", ".join(unknown)
+                    )
         except Exception as e:
             raise ParameterMismatch(str(e))
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/run-task/fetch-content

@@ -626,7 +626,7 @@ def fetch_and_extract(url, dest_dir, extract=True, sha256=None, size=None):
     the destination directory.
     """
 
-    basename = urllib.parse.urlparse(url).path.split("/")[-1]
+    basename = urllib.parse.unquote(urllib.parse.urlparse(url).path.split("/")[-1])
     dest_path = dest_dir / basename
 
     download_to_path(url, dest_path, sha256=sha256, size=size)
@@ -920,16 +920,17 @@ def command_task_artifacts(args):
         sha256 = None
         if fetch.get("verify-hash"):
             sha256 = get_hash(fetch, root_url)
+        encoded_artifact = urllib.parse.quote(fetch["artifact"])
         if fetch["artifact"].startswith("public/"):
             path = "task/{task}/artifacts/{artifact}".format(
-                task=fetch["task"], artifact=fetch["artifact"]
+                task=fetch["task"], artifact=encoded_artifact
             )
             url = api(root_url, "queue", "v1", path)
         else:
             url = ("{proxy_url}/api/queue/v1/task/{task}/artifacts/{artifact}").format(
                 proxy_url=os.environ["TASKCLUSTER_PROXY_URL"],
                 task=fetch["task"],
-                artifact=fetch["artifact"],
+                artifact=encoded_artifact,
             )
         downloads.append((url, extdir, fetch["extract"], sha256))
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.1.0}/src/taskgraph/taskgraph.py

@@ -44,13 +44,10 @@ class TaskGraph:
 
     def to_json(self):
         "Return a JSON-able object representing the task graph, as documented"
-        named_links_dict = self.graph.named_links_dict()
         # this dictionary may be keyed by label or by taskid, so let's just call it 'key'
         tasks = {}
         for key in self.graph.visit_postorder():
             tasks[key] = self.tasks[key].to_json()
-            # overwrite dependencies with the information in the taskgraph's edges.
-            tasks[key]["dependencies"] = named_links_dict.get(key, {})
         return tasks
 
     @classmethod
@@ -66,6 +63,8 @@ class TaskGraph:
             if "task_id" in value:
                 tasks[key].task_id = value["task_id"]
             for depname, dep in value["dependencies"].items():
-                edges.add((key, dep, depname))
+                # Task filtering can cause dependencies to be removed from the graph.
+                if dep in tasks_dict:
+                    edges.add((key, dep, depname))
         task_graph = cls(tasks, Graph(frozenset(tasks), edges))
         return tasks, task_graph