taskcluster-taskgraph 14.2.0__tar.gz → 14.3.0__tar.gz

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/.pre-commit-config.yaml

@@ -13,13 +13,13 @@ repos:
         exclude: template
       - id: check-added-large-files
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.4
+    rev: v0.11.7
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.6.12
+    rev: 0.7.2
     hooks:
       - id: uv-lock
   - repo: https://github.com/adrienverge/yamllint.git
@@ -37,7 +37,7 @@ repos:
                 test/test_util_path.py
             )$
   - repo: https://github.com/compilerla/conventional-pre-commit
-    rev: v4.0.0
+    rev: v4.1.0
     hooks:
       - id: conventional-pre-commit
         stages: [commit-msg]

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/CHANGELOG.md

@@ -1,35 +1,53 @@
 # Change Log
 
-## [14.2.0] - 2025-04-10
+## [14.3.0] - 2025-05-28
 
-## Fixed
+### Fixed
+
+- MAX_ROUTES bumped up to actual limit of 64
+
+### Added
+
+- `load-task` command now accepts a `--user` flag to run in container as
+- optionally use `orjson` for faster json operations if available
+
+## [14.2.1] - 2025-05-13
+
+### Fixed
+
+- Revert "generic-worker will now verify run-task/fetch-content integrity" from
+  14.2.0 for being broken
+
+## [14.2.0] - 2025-05-12
+
+### Fixed
 
 - fix actions scope check on mercurial (#686)
 
-## Changed
+### Changed
 
 - generic-worker will now verify run-task/fetch-content integrity (#666)
 
-## Added
+### Added
 
 - new 'load-task' command to debug tasks locally (#680)
 
 ## [14.1.1] - 2025-04-10
 
-## Fixed
+### Fixed
 
 - ensure GraphConfig.root_dir is an absolute path (#673)
 
 ## [14.1.0] - 2025-04-07
 
-## Added
+### Added
 
 - allow task-priority to be keyed by level (#653)
 - implement `taskgraph action` command to generate actions.json on the command line (#645)
 - pass push.base_revision to in-tree-action hook (#656)
 - support for `volume` artifact type (#667)
 
-## Fixed
+### Fixed
 
 - bug deleting long path names with forward slashes on Windows (#661)
 - replace deprecated datetime.utcnow() with timezone-aware alternative (#652)

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: taskcluster-taskgraph
-Version: 14.2.0
+Version: 14.3.0
 Summary: Build taskcluster taskgraphs
 Project-URL: Repository, https://github.com/taskcluster/taskgraph
 Project-URL: Issues, https://github.com/taskcluster/taskgraph/issues
@@ -29,6 +29,8 @@ Requires-Dist: taskcluster-urls>=11.0
 Requires-Dist: voluptuous>=0.12.1
 Provides-Extra: load-image
 Requires-Dist: zstandard; extra == 'load-image'
+Provides-Extra: orjson
+Requires-Dist: orjson; extra == 'orjson'
 Description-Content-Type: text/x-rst
 
 

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/docs/tutorials/getting-started.rst

@@ -13,91 +13,86 @@ with your repo in as few steps as possible.
 
 There are three steps to getting running tasks:
 
-1. Install Taskcluster Github
+1. Define Tasks
 2. Request Permissions
-3. Define Tasks
+3. Install Taskcluster Github
 
 Let's dive into each!
 
-Install Taskcluster Github
---------------------------
+Create Tasks
+------------
 
-.. note::
+The first step is to define your tasks. You'll need:
 
-   If you are not using Github, skip to the next section.
+1. A :term:`Decision Task` defined in ``.taskcluster.yml``
+2. A Taskgraph setup in the ``taskcluster`` directory
 
-Each Taskcluster deployment has an associated Github app that needs to be added
-to your repo. Some apps for known Taskcluster deployments are as follows:
+Luckily Taskgraph provides an ``init`` command that can generate both of these
+things automatically! Run:
 
-.. list-table:: Taskcluster Github Apps
-   :widths: 20 60
-   :header-rows: 1
+.. code-block:: shell
+
+   $ pip install taskcluster-taskgraph
+   $ taskgraph init
 
-   * - Instance
-     - Github App
-   * - `Firefox-CI <https://firefox-ci-tc.services.mozilla.com>`_
-     - https://github.com/apps/firefoxci-taskcluster
-   * - `Community <https://community-tc.services.mozilla.com>`_
-     - https://github.com/apps/community-tc-integration
+This will automatically generate all the necessary files. Commit what was
+generated:
 
-Follow the link to the Github app for your deployment and click ``Configure``
-near the top right. Then select the repository you'd like to enable. If you are
-not an admin of the organization you are enabling the repo for, this will send
-a request to your Github administrators.
+.. code-block:: shell
 
-.. note::
+   $ git add .
+   $ git commit -m "Add Taskcluster files"
 
-   If you do not see a ``Configure`` button, contact your Github administrators
-   for assistance.
+Land your changes on the default branch (typically ``main``). Nothing will
+happen yet, but we need that ``.taskcluster.yml`` file to exist at the root of
+the repository for the next step.
 
 Request Permissions
 -------------------
 
-Your repository is now connected to Taskcluster, but it won't have permission
-to run anything. In the Taskcluster world, permissions are also known as
-:term:`scopes <Scope>`. So the next step is to contact your Taskcluster
-administrators and request scopes for your project so that it has access to the
-resources it needs to run tasks.
+Your repository now has some tasks defined, but it won't have permission to run
+anything. In the Taskcluster world, permissions are also known as :term:`scopes
+<Scope>`. So the next step is to contact your Taskcluster administrators and
+request scopes for your project so that it has access to the resources it needs
+to run tasks.
 
-Each Taskcluster instance can be managed differently, but here are the processes
-to request scopes for some known Taskcluster deployments:
+.. note::
 
-.. list-table:: Taskcluster Github Apps
-   :widths: 20 60
-   :header-rows: 1
+   For Mozilla's `Firefox-CI <https://firefox-ci-tc.services.mozilla.com>`_ instance,
+   file a bug under `Release Engineering :: Firefox-CI Administration
+   <https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Firefox-CI%20Administration>`_
 
-   * - Instance
-     - Process
-   * - `Firefox-CI <https://firefox-ci-tc.services.mozilla.com>`_
-     - `File a Bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Firefox-CI%20Administration>`_
-   * - `Community <https://community-tc.services.mozilla.com>`_
-     - `File an Issue <https://github.com/taskcluster/community-tc-config/issues/new>`_
+   Use a title such as: *Please setup `org/my-repo` with Taskcluster*.
 
-Use a title such as: *Please setup `org/my-repo` with Taskcluster*.
+Install Taskcluster Github
+--------------------------
 
-Create Tasks
-------------
+.. note::
 
-The last step is to define your tasks. You'll need:
+   If you are not using Github, skip this section and reach out to your
+   Taskcluster administrators, there may be other steps to integrate your
+   repository.
 
-1. A :term:`Decision Task` defined in ``.taskcluster.yml``
-2. A Taskgraph setup in the ``taskcluster`` directory
+Each Taskcluster deployment has an associated Github app that needs to be added
+to your repo. If you are unsure which app your deployment uses, ask your
+Taskcluster administrators.
 
-Luckily Taskgraph provides an ``init`` command that can generate both of these
-things automatically! Run:
+.. note::
 
-.. code-block:: shell
+   The Github app for Mozilla's `Firefox-CI
+   <https://firefox-ci-tc.services.mozilla.com>`_ instance can be found here:
 
-   $ pip install taskcluster-taskgraph
-   $ taskgraph init
+   https://github.com/apps/firefoxci-taskcluster
 
-This will automatically generate all the necessary files. Commit what was
-generated:
+Open the page of the Github app for your deployment and click ``Configure``
+near the top right. Then select the repository you'd like to enable. If you are
+not an admin of the organization you are enabling the repo for, this will send
+a request to your Github administrators.
 
-.. code-block:: shell
+.. note::
 
-   $ git add .
-   $ git commit -m "Add Taskcluster files"
+   If you do not see a ``Configure`` button, contact your Github administrators
+   for assistance.
 
 Now when you push, you should have a working Decision task that generates a
 ``hello-world`` task as well as a ``docker-image`` task that it depends on!

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/pyproject.toml

@@ -34,9 +34,8 @@ dependencies = [
 dynamic = ["version"]  # set in src/taskgraph/__init__.py
 
 [project.optional-dependencies]
-load-image = [
-  "zstandard"
-]
+load-image = ["zstandard"]
+orjson = ["orjson"]
 
 [project.scripts]
 taskgraph = "taskgraph.main:main"

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/__init__.py

@@ -2,7 +2,7 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
-__version__ = "14.2.0"
+__version__ = "14.3.0"
 
 # Maximum number of dependencies a single task can have
 # https://docs.taskcluster.net/docs/reference/platform/queue/api#createTask

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/actions/registry.py

@@ -4,7 +4,6 @@
 
 
 import functools
-import json
 from collections import namedtuple
 from types import FunctionType
 
@@ -13,7 +12,7 @@ from mozilla_repo_urls import parse
 from taskgraph import create
 from taskgraph.config import load_graph_config
 from taskgraph.parameters import Parameters
-from taskgraph.util import hash, taskcluster, yaml
+from taskgraph.util import hash, json, taskcluster, yaml
 from taskgraph.util.python_path import import_sibling_modules
 
 actions = []

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/create.py

@@ -3,7 +3,6 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-import json
 import logging
 import os
 import sys
@@ -11,6 +10,7 @@ from concurrent import futures
 
 from slugid import nice as slugid
 
+from taskgraph.util import json
 from taskgraph.util.parameterization import resolve_timestamps
 from taskgraph.util.taskcluster import CONCURRENCY, get_session
 from taskgraph.util.time import current_json_time
@@ -116,8 +116,7 @@ def create_task(session, task_id, label, task_def):
             [task_id, task_def],
             sys.stdout,
             sort_keys=True,
-            indent=4,
-            separators=(",", ": "),
+            indent=2,
         )
         # add a newline
         print("")

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/decision.py

@@ -3,7 +3,6 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-import json
 import logging
 import os
 import pathlib
@@ -19,6 +18,7 @@ from taskgraph.create import create_tasks
 from taskgraph.generator import TaskGraphGenerator
 from taskgraph.parameters import Parameters, get_version
 from taskgraph.taskgraph import TaskGraph
+from taskgraph.util import json
 from taskgraph.util.python_path import find_object
 from taskgraph.util.schema import Schema, validate_schema
 from taskgraph.util.vcs import Repository, get_repository
@@ -374,7 +374,7 @@ def write_artifact(filename, data):
             yaml.safe_dump(data, f, allow_unicode=True, default_flow_style=False)
     elif filename.endswith(".json"):
         with open(path, "w") as f:
-            json.dump(data, f, sort_keys=True, indent=2, separators=(",", ": "))
+            json.dump(data, f, sort_keys=True, indent=2)
     elif filename.endswith(".gz"):
         import gzip
 

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/docker.py

@@ -3,7 +3,6 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-import json
 import os
 import shlex
 import subprocess
@@ -18,7 +17,7 @@ try:
 except ImportError as e:
     zstd = e
 
-from taskgraph.util import docker
+from taskgraph.util import docker, json
 from taskgraph.util.taskcluster import (
     get_artifact_url,
     get_root_url,
@@ -251,7 +250,8 @@ def _index(l: List, s: str) -> Optional[int]:
         pass
 
 
-def load_task(task_id, remove=True):
+def load_task(task_id, remove=True, user=None):
+    user = user or "worker"
     task_def = get_task_definition(task_id)
 
     if (
@@ -316,7 +316,7 @@ def load_task(task_id, remove=True):
             image_tag,
             "bash",
             "-c",
-            f"{shlex.join(command)} && cd $TASK_WORKDIR && su -p worker",
+            f"{shlex.join(command)} && cd $TASK_WORKDIR && su -p {user}",
         ]
 
         if remove:

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/main.py

@@ -4,7 +4,6 @@
 
 import argparse
 import atexit
-import json
 import logging
 import os
 import re
@@ -56,9 +55,9 @@ def format_taskgraph_labels(taskgraph):
 
 
 def format_taskgraph_json(taskgraph):
-    return json.dumps(
-        taskgraph.to_json(), sort_keys=True, indent=2, separators=(",", ": ")
-    )
+    from taskgraph.util import json
+
+    return json.dumps(taskgraph.to_json(), sort_keys=True, indent=2)
 
 
 def format_taskgraph_yaml(taskgraph):
@@ -667,11 +666,12 @@ def image_digest(args):
     default=True,
     help="Keep the docker container after exiting.",
 )
+@argument("--user", default=None, help="Container user to start shell with.")
 def load_task(args):
     from taskgraph.docker import load_task
 
     validate_docker()
-    return load_task(args["task_id"], remove=args["remove"])
+    return load_task(args["task_id"], remove=args["remove"], user=args["user"])
 
 
 @command("decision", help="Run the decision task")
@@ -756,6 +756,7 @@ def actions(args):
     from taskgraph.actions import render_actions_json
     from taskgraph.generator import TaskGraphGenerator
     from taskgraph.parameters import parameters_loader
+    from taskgraph.util import json
 
     if args.pop("verbose", False):
         logging.root.setLevel(logging.DEBUG)
@@ -765,7 +766,7 @@ def actions(args):
         tgg = TaskGraphGenerator(root_dir=args.get("root"), parameters=parameters)
 
         actions = render_actions_json(tgg.parameters, tgg.graph_config, "DECISION-TASK")
-        print(json.dumps(actions, sort_keys=True, indent=2, separators=(",", ": ")))
+        print(json.dumps(actions, sort_keys=True, indent=2))
     except Exception:
         traceback.print_exc()
         sys.exit(1)
@@ -783,6 +784,7 @@ def actions(args):
 def action_callback(options):
     from taskgraph.actions import trigger_action_callback
     from taskgraph.actions.util import get_parameters
+    from taskgraph.util import json
 
     try:
         # the target task for this action (or null if it's a group action)
@@ -832,7 +834,7 @@ def test_action_callback(options):
     import taskgraph.actions
     import taskgraph.parameters
     from taskgraph.config import load_graph_config
-    from taskgraph.util import yaml
+    from taskgraph.util import json, yaml
 
     def load_data(filename):
         with open(filename) as f:

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/morph.py

@@ -30,7 +30,7 @@ from .util.workertypes import get_worker_type
 
 here = os.path.abspath(os.path.dirname(__file__))
 logger = logging.getLogger(__name__)
-MAX_ROUTES = 10
+MAX_ROUTES = 64
 
 registered_morphs = []
 
@@ -155,10 +155,9 @@ def make_index_task(parent_task, taskgraph, label_to_taskid, parameters, graph_c
 @register_morph
 def add_index_tasks(taskgraph, label_to_taskid, parameters, graph_config):
     """
-    The TaskCluster queue only allows 10 routes on a task, but we have tasks
-    with many more routes, for purposes of indexing. This graph morph adds
-    "index tasks" that depend on such tasks and do the index insertions
-    directly, avoiding the limits on task.routes.
+    The TaskCluster queue only allows 64 routes on a task. In the event a task
+    exceeds this limit, this graph morph adds "index tasks" that depend on it
+    and do the index insertions directly, avoiding the limit on task.routes.
     """
     logger.debug("Morphing: adding index tasks")
 

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/parameters.py

@@ -4,7 +4,6 @@
 
 import gzip
 import hashlib
-import json
 import os
 import time
 from datetime import datetime
@@ -18,7 +17,7 @@ from urllib.request import urlopen
 import mozilla_repo_urls
 from voluptuous import ALLOW_EXTRA, Any, Optional, Required, Schema
 
-from taskgraph.util import yaml
+from taskgraph.util import json, yaml
 from taskgraph.util.readonlydict import ReadOnlyDict
 from taskgraph.util.schema import validate_schema
 from taskgraph.util.taskcluster import find_task_id, get_artifact_url

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/transforms/docker_image.py

@@ -2,8 +2,6 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
-
-import json
 import logging
 import os
 import re
@@ -12,6 +10,7 @@ from voluptuous import Optional, Required
 
 import taskgraph
 from taskgraph.transforms.base import TransformSequence
+from taskgraph.util import json
 from taskgraph.util.docker import create_context_tar, generate_context_hash
 from taskgraph.util.schema import Schema
 

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/transforms/run/__init__.py

@@ -10,7 +10,6 @@ run-using handlers in `taskcluster/taskgraph/transforms/run`.
 """
 
 import copy
-import json
 import logging
 
 from voluptuous import Exclusive, Extra, Optional, Required
@@ -18,6 +17,7 @@ from voluptuous import Exclusive, Extra, Optional, Required
 from taskgraph.transforms.base import TransformSequence
 from taskgraph.transforms.cached_tasks import order_tasks
 from taskgraph.transforms.task import task_description_schema
+from taskgraph.util import json
 from taskgraph.util import path as mozpath
 from taskgraph.util.python_path import import_sibling_modules
 from taskgraph.util.schema import Schema, validate_schema

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/transforms/run/common.py

@@ -7,11 +7,10 @@ worker implementation they operate on, and take the same three parameters, for
 consistency.
 """
 
-import json
 from typing import Any, Dict, List, Union
 
 from taskgraph.transforms.base import TransformConfig
-from taskgraph.util import path
+from taskgraph.util import json, path
 from taskgraph.util.caches import CACHES, get_checkout_dir
 from taskgraph.util.taskcluster import get_artifact_prefix
 

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/src/taskgraph/transforms/run/run_task.py

@@ -6,9 +6,7 @@ Support for running tasks that are invoked via the `run-task` script.
 """
 
 import dataclasses
-import hashlib
 import os
-from pathlib import Path
 
 from voluptuous import Any, Optional, Required
 
@@ -27,9 +25,6 @@ EXEC_COMMANDS = {
     "powershell": ["powershell.exe", "-ExecutionPolicy", "Bypass"],
 }
 
-RUN_TASK_PATH = Path(__file__).parent.parent.parent / "run-task" / "run-task"
-FETCH_CONTENT_PATH = Path(__file__).parent.parent.parent / "run-task" / "fetch-content"
-
 run_task_schema = Schema(
     {
         Required("using"): "run-task",
@@ -174,14 +169,10 @@ def generic_worker_run_task(config, task, taskdesc):
     common_setup(config, task, taskdesc, command)
 
     worker.setdefault("mounts", [])
-    run_task_sha256 = hashlib.sha256(RUN_TASK_PATH.read_bytes()).hexdigest()
-    fetch_content_sha256 = hashlib.sha256(FETCH_CONTENT_PATH.read_bytes()).hexdigest()
     worker["mounts"].append(
         {
             "content": {
-                "taskId": {"task-reference": "<decision>"},
-                "artifact": "public/run-task",
-                "sha256": run_task_sha256,
+                "url": script_url(config, "run-task"),
             },
             "file": "./run-task",
         }
@@ -190,9 +181,7 @@ def generic_worker_run_task(config, task, taskdesc):
         worker["mounts"].append(
             {
                 "content": {
-                    "taskId": {"task-reference": "<decision>"},
-                    "artifact": "public/fetch-content",
-                    "sha256": fetch_content_sha256,
+                    "url": script_url(config, "fetch-content"),
                 },
                 "file": "./fetch-content",
             }

taskcluster_taskgraph-14.3.0/src/taskgraph/util/json.py

@@ -0,0 +1,73 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""
+This module is a compatibility shim between stdlib json and orjson.
+It uses orjson if available, and falls back to stdlib json.
+
+It only supports features and parameters that both modules support,
+and attempts to produce identical output regardless of which module
+is used.
+
+The interface is a subset of what's supported by stdlib json.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, Callable
+
+if TYPE_CHECKING:
+    from _typeshed import SupportsRead, SupportsWrite
+
+try:
+    import orjson
+except ImportError:
+    orjson = None
+
+
+def loads(s: str | bytes | bytearray) -> Any:
+    if orjson:
+        return orjson.loads(s)
+
+    return json.loads(s)
+
+
+def load(fh: SupportsRead[str | bytes]) -> Any:
+    if orjson:
+        return loads(fh.read())
+    return json.load(fh)
+
+
+def dumps(
+    obj: Any,
+    default: Callable[[Any], Any] | None = None,
+    indent: int | None = None,
+    sort_keys: bool = False,
+) -> str:
+    if indent and indent != 2:
+        raise ValueError("An indent other than 2 is not supported!")
+
+    if orjson:
+        option = 0
+        if indent:
+            option |= orjson.OPT_INDENT_2
+
+        if sort_keys:
+            option |= orjson.OPT_SORT_KEYS
+
+        return orjson.dumps(obj, default=default, option=option).decode("utf-8")
+
+    separators = (",", ": ") if indent else (",", ":")
+    return json.dumps(
+        obj,
+        default=default,
+        indent=indent,
+        sort_keys=sort_keys,
+        separators=separators,
+    )
+
+
+def dump(obj: Any, fh: SupportsWrite[str], **kwargs) -> None:
+    fh.write(dumps(obj, **kwargs))

{taskcluster_taskgraph-14.2.0 → taskcluster_taskgraph-14.3.0}/taskcluster/docker/decision/system-setup.sh

@@ -9,7 +9,7 @@ apt-get install -y --force-yes --no-install-recommends \
     python3-pip
 
 pushd /setup/taskgraph
-uv export --no-cache --no-dev > /setup/requirements.txt
+uv export --no-cache --extra orjson --no-dev > /setup/requirements.txt
 uv pip install --no-cache --system --break-system-packages -r /setup/requirements.txt
 uv pip install --no-cache --system --break-system-packages --no-deps .
 popd