req-compile 1.1.2__tar.gz → 1.2.0__tar.gz

{req_compile-1.1.2 → req_compile-1.2.0}/CHANGELOG.rst

@@ -1,3 +1,16 @@
+New in version 1.2.0:
+
+* Cross-platform compilation. ``--platform``, ``--python-version`` and ``--implementation`` select
+  wheels and evaluate environment markers for an environment other than the one running ``req-compile``.
+  ``--platform`` takes a wheel platform tag such as ``manylinux_2_28_x86_64`` or ``macosx_11_0_arm64``,
+  which is recorded in the header of lock files compiled by Bazel.
+* The ``py_reqs_compiler`` Bazel rule has new ``platform`` and ``python_version`` attributes, so the
+  solutions for every platform a project supports can be compiled and tested from a single machine.
+  Solutions record their platform and Python version in their header, and a solution compiled for
+  another target is not recycled when recompiling without ``--upgrade``.
+* Removed the ``py_reqs_remote_compiler`` Bazel rule. Use the ``platform`` attribute of ``py_reqs_compiler``
+  instead.
+
 New in version 1.1.0:
 
 * Python 3.13 and 3.14 support.

{req_compile-1.1.2/req_compile.egg-info → req_compile-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: req-compile
-Version: 1.1.2
+Version: 1.2.0
 Summary: Python requirements compiler
 Home-page: https://github.com/periareon/req-compile
 Author: Spencer Putt
@@ -38,6 +38,19 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
+New in version 1.2.0:
+
+* Cross-platform compilation. ``--platform``, ``--python-version`` and ``--implementation`` select
+  wheels and evaluate environment markers for an environment other than the one running ``req-compile``.
+  ``--platform`` takes a wheel platform tag such as ``manylinux_2_28_x86_64`` or ``macosx_11_0_arm64``,
+  which is recorded in the header of lock files compiled by Bazel.
+* The ``py_reqs_compiler`` Bazel rule has new ``platform`` and ``python_version`` attributes, so the
+  solutions for every platform a project supports can be compiled and tested from a single machine.
+  Solutions record their platform and Python version in their header, and a solution compiled for
+  another target is not recycled when recompiling without ``--upgrade``.
+* Removed the ``py_reqs_remote_compiler`` Bazel rule. Use the ``platform`` attribute of ``py_reqs_compiler``
+  instead.
+
 New in version 1.1.0:
 
 * Python 3.13 and 3.14 support.
@@ -112,6 +125,9 @@ To get started with Bazel, add the following to your ``MODULE.bazel`` file:
     )
     use_repo(requirements, "my_pip_deps")
 
+The solutions for every platform can be compiled from a single machine with the ``platform`` attribute of
+``py_reqs_compiler``. See the `Bazel rules documentation <docs/docs.md>`_ for details.
+
 
 Why use it?
 -----------
@@ -229,6 +245,33 @@ was passed on the commandline.
 By default, PyPI (https://pypi.org/) or the default pip index is added as a default repository. It can be removed by passing
 ``--no-index`` on the commandline or passing a different index via ``--index-url``.
 
+Compiling for another platform
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+By default a solution is compiled for the interpreter running Req-Compile: wheels are selected and
+environment markers are evaluated for its platform, Python version and implementation. A solution for
+a different environment can be compiled from any machine with the target options::
+
+    > req-compile requirements.txt --platform macosx_11_0_arm64 --python-version 3.11 --only-binary :all:
+
+* ``--platform``
+
+  The platform to compile for as a wheel platform tag, the same values ``pip --platform`` accepts:
+  ``manylinux_2_28_x86_64``, ``manylinux2014_aarch64``, ``macosx_11_0_arm64``, ``macosx_10_15_x86_64``,
+  ``win_amd64`` and so on. A manylinux tag accepts wheels built for that glibc or older, a macosx tag
+  wheels built for that release or older, and plain ``linux_x86_64`` accepts no manylinux wheels at all,
+  as with pip.
+* ``--python-version``
+
+  The Python version to compile for, e.g. ``3.11`` or ``3.11.4``. This can be used on its own to
+  compile for a different version of Python on the current platform.
+* ``--implementation``
+
+  The interpreter tag of the implementation to compile for, ``cp`` for CPython or ``pp`` for PyPy.
+
+Source distributions have their metadata extracted by running them on the machine performing the
+compilation, which may not match the target. Prefer ``--only-binary :all:`` when compiling for another
+platform. ``req-candidates`` accepts the same options to show the candidates available for a target.
+
 Identifying source of constraints
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Why did I just get version 1.11.0 of ``six``? Find out by examining the output::

{req_compile-1.1.2 → req_compile-1.2.0}/README.rst

@@ -47,6 +47,9 @@ To get started with Bazel, add the following to your ``MODULE.bazel`` file:
     )
     use_repo(requirements, "my_pip_deps")
 
+The solutions for every platform can be compiled from a single machine with the ``platform`` attribute of
+``py_reqs_compiler``. See the `Bazel rules documentation <docs/docs.md>`_ for details.
+
 
 Why use it?
 -----------
@@ -164,6 +167,33 @@ was passed on the commandline.
 By default, PyPI (https://pypi.org/) or the default pip index is added as a default repository. It can be removed by passing
 ``--no-index`` on the commandline or passing a different index via ``--index-url``.
 
+Compiling for another platform
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+By default a solution is compiled for the interpreter running Req-Compile: wheels are selected and
+environment markers are evaluated for its platform, Python version and implementation. A solution for
+a different environment can be compiled from any machine with the target options::
+
+    > req-compile requirements.txt --platform macosx_11_0_arm64 --python-version 3.11 --only-binary :all:
+
+* ``--platform``
+
+  The platform to compile for as a wheel platform tag, the same values ``pip --platform`` accepts:
+  ``manylinux_2_28_x86_64``, ``manylinux2014_aarch64``, ``macosx_11_0_arm64``, ``macosx_10_15_x86_64``,
+  ``win_amd64`` and so on. A manylinux tag accepts wheels built for that glibc or older, a macosx tag
+  wheels built for that release or older, and plain ``linux_x86_64`` accepts no manylinux wheels at all,
+  as with pip.
+* ``--python-version``
+
+  The Python version to compile for, e.g. ``3.11`` or ``3.11.4``. This can be used on its own to
+  compile for a different version of Python on the current platform.
+* ``--implementation``
+
+  The interpreter tag of the implementation to compile for, ``cp`` for CPython or ``pp`` for PyPy.
+
+Source distributions have their metadata extracted by running them on the machine performing the
+compilation, which may not match the target. Prefer ``--only-binary :all:`` when compiling for another
+platform. ``req-candidates`` accepts the same options to show the candidates available for a target.
+
 Identifying source of constraints
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Why did I just get version 1.11.0 of ``six``? Find out by examining the output::

{req_compile-1.1.2 → req_compile-1.2.0}/req_compile/candidates.py

@@ -14,7 +14,9 @@ from req_compile.cmdline import (
     SplitProjectsFilter,
     add_logging_args,
     add_repo_args,
+    add_target_args,
     build_repo,
+    build_target,
 )
 from req_compile.repos.pypi import PyPIRepository
 from req_compile.repos.repository import (
@@ -71,6 +73,7 @@ def candidates_main() -> None:
     )
     add_logging_args(parser)
     add_repo_args(parser)
+    add_target_args(parser)
     args = parser.parse_args()
 
     logger = logging.getLogger("req_compile")
@@ -81,6 +84,12 @@ def candidates_main() -> None:
         logging.basicConfig(level=logging.CRITICAL, stream=sys.stderr)
         logging.getLogger().setLevel(level=logging.CRITICAL)
 
+    try:
+        target = build_target(args)
+    except ValueError as ex:
+        print(f"ERROR: {ex}", file=sys.stderr)
+        sys.exit(1)
+
     start = time.monotonic()
     wheeldir = tempfile.mkdtemp(suffix="-wheeldir")
     repo = build_repo(
@@ -94,10 +103,13 @@ def candidates_main() -> None:
         no_index=args.no_index,
         wheeldir=wheeldir,
         allow_prerelease=args.allow_prerelease,
+        target=target,
     )
 
     if isinstance(repo, PyPIRepository) and args.project_name is None:
-        repo = SourceRepository(".", excluded_paths=args.excluded_sources)
+        repo = SourceRepository(
+            ".", excluded_paths=args.excluded_sources, target=target
+        )
 
     total_candidates = 0
     try:
@@ -112,10 +124,13 @@ def candidates_main() -> None:
         candidates = repo.get_candidates(req)
         if not args.all:
             candidates = filter_candidates(
-                req, candidates, allow_prereleases=args.allow_prerelease
+                req,
+                candidates,
+                allow_prereleases=args.allow_prerelease,
+                target=repo.target,
             )
 
-        for candidate in sort_candidates(candidates):
+        for candidate in sort_candidates(candidates, target=repo.target):
             if (
                 candidate.type == DistributionType.SDIST
                 and candidate.name in args.only_binary

{req_compile-1.1.2 → req_compile-1.2.0}/req_compile/cmdline.py

@@ -45,6 +45,7 @@ from req_compile.repos.solution import (
     SolutionRepository,
 )
 from req_compile.repos.source import SourceRepository
+from req_compile.target import TargetEnvironment, resolve
 from req_compile.utils import (
     NormName,
     normalize_project_name,
@@ -56,6 +57,8 @@ from req_compile.versions import is_possible
 # Blacklist of requirements that will be filtered out of the output
 BLACKLIST: Iterable[str] = []
 
+LOG = logging.getLogger("req_compile.cmdline")
+
 
 def _cantusereason_to_text(
     reason: CantUseReason,
@@ -147,7 +150,7 @@ def _generate_no_candidate_display(
                     pinned_req = parse_requirement(f"{walkback_project}=={version}")
                     print(f"  {pinned_req}", file=sys.stderr, end="")
                     try:
-                        collection = DistributionCollection()
+                        collection = DistributionCollection(target=repo.target)
                         collection.add_dist(
                             repo.get_dist(pinned_req)[0], None, pinned_req
                         )
@@ -263,7 +266,9 @@ def _dump_repo_candidates(
         print("  {}:".format(repo), file=sys.stderr)
         if candidates:
             attempted_versions = set()
-            for num, candidate in enumerate(sort_candidates(candidates)):
+            for num, candidate in enumerate(
+                sort_candidates(candidates, target=repo.target)
+            ):
                 if not req.specifier.contains(candidate.version):
                     continue
                 attempted_versions.add(candidate.version)
@@ -682,7 +687,13 @@ def build_repo(
     extra_index_urls: Optional[Iterable[str]] = None,
     no_index: bool = False,
     allow_prerelease: bool = False,
+    target: Optional[TargetEnvironment] = None,
 ) -> Repository:
+    """Build the repository to compile against from command line style options.
+
+    Args:
+        target: The environment to compile for. Defaults to the host.
+    """
     pooled_repos: List[Repository] = []
     if find_links:
         pooled_repos.extend(
@@ -692,6 +703,7 @@ def build_repo(
                 relative_to=(
                     find_links[find_link] if isinstance(find_links, Mapping) else None
                 ),
+                target=target,
             )
             for find_link in find_links
         )
@@ -704,6 +716,7 @@ def build_repo(
                     wheeldir,
                     allow_prerelease=allow_prerelease,
                     index_type=IndexType.DEFAULT,
+                    target=target,
                 )
             )
         else:
@@ -713,6 +726,7 @@ def build_repo(
                     wheeldir,
                     allow_prerelease=allow_prerelease,
                     index_type=IndexType.INDEX_URL,
+                    target=target,
                 )
                 for index_url in index_urls
             )
@@ -723,38 +737,93 @@ def build_repo(
                     wheeldir,
                     allow_prerelease=allow_prerelease,
                     index_type=IndexType.EXTRA_INDEX_URL,
+                    target=target,
                 )
                 for index_url in extra_index_urls
             )
 
     repos: List[Repository] = []
-    if solutions:
-        repos.extend(
-            SolutionRepository(solution, excluded_packages=upgrade_packages)
-            for solution in solutions
-        )
+    ignored_solutions: List[SolutionRepository] = []
+    for solution_repo in load_solutions(solutions, upgrade_packages, target):
+        if solution_repo.matches_target(target):
+            repos.append(solution_repo)
+        else:
+            ignored_solutions.append(solution_repo)
     if sources:
         repos.extend(
-            SourceRepository(source, excluded_paths=excluded_sources)
+            SourceRepository(source, excluded_paths=excluded_sources, target=target)
             for source in sources
         )
 
     if len(pooled_repos) > 1:
-        repos.append(PooledCandidateMultiRepository(*pooled_repos))
+        repos.append(PooledCandidateMultiRepository(*pooled_repos, target=target))
     elif len(pooled_repos) == 1:
         repos.append(pooled_repos[0])
 
     if not repos:
+        if ignored_solutions:
+            raise RepositoryInitializationError(
+                SolutionRepository,
+                "The solution {} was compiled for {}, not for {}. Recompile it for the new target.".format(
+                    ", ".join(repr(repo) for repo in ignored_solutions),
+                    _recorded_target(ignored_solutions[0]),
+                    resolve(target),
+                ),
+            )
         raise ValueError(
             "At least one source of Python distributions must be provided."
         )
     if len(repos) > 1:
-        repo: Repository = MultiRepository(*repos)
+        repo: Repository = MultiRepository(*repos, target=target)
     else:
         repo = repos[0]
     return repo
 
 
+def _recorded_target(solution: SolutionRepository) -> str:
+    return "{} ({})".format(
+        solution.recorded_platform or "an unknown platform",
+        solution.recorded_python or "unknown Python version",
+    )
+
+
+def load_solutions(
+    solutions: Iterable[str],
+    upgrade_packages: Iterable[str],
+    target: Optional[TargetEnvironment] = None,
+) -> List[SolutionRepository]:
+    """Load previous solutions, warning about any compiled for another target.
+
+    A solution compiled for another platform or Python version pins artifacts and
+    already evaluated dependencies that do not apply to this compilation. Such
+    solutions are returned, so callers can tell the user, but must not be used.
+    See `SolutionRepository.matches_target`.
+
+    Args:
+        solutions: Paths to solution files.
+        upgrade_packages: Packages to omit from the solutions.
+        target: The environment being compiled for. Defaults to the host.
+
+    Returns:
+        A repository per solution file.
+    """
+    repos = []
+    for solution in solutions:
+        repo = SolutionRepository(
+            solution, excluded_packages=upgrade_packages, target=target
+        )
+        if not repo.matches_target(target):
+            LOG.warning(
+                "Ignoring the solution %s, it was compiled for %s but this compilation "
+                "is for %s. Requirements will be resolved again.",
+                solution,
+                _recorded_target(repo),
+                resolve(target),
+            )
+        repos.append(repo)
+    return repos
+
+
 class IndentFilter(logging.Filter):
     """A filter to indent to a level specified by the depth attribute of a logging record"""
 
@@ -914,6 +983,7 @@ def compile_main(raw_args: Optional[Sequence[str]] = None) -> None:
     )
     add_logging_args(parser)
     add_repo_args(parser)
+    add_target_args(parser)
 
     args = parser.parse_args(args=raw_args)
     logger = logging.getLogger("req_compile")
@@ -926,6 +996,12 @@ def compile_main(raw_args: Optional[Sequence[str]] = None) -> None:
     else:
         logging.basicConfig(level=logging.CRITICAL, stream=sys.stderr)
 
+    try:
+        target = build_target(args)
+    except ValueError as ex:
+        print(f"ERROR: {ex}", file=sys.stderr)
+        sys.exit(1)
+
     wheeldir = args.wheel_dir
     if wheeldir:
         try:
@@ -1018,6 +1094,7 @@ def compile_main(raw_args: Optional[Sequence[str]] = None) -> None:
         extra_index_urls=args.extra_index_urls,
         no_index=args.no_index,
         allow_prerelease=args.allow_prerelease,
+        target=target,
     )
     try:
         results, roots = perform_compile(
@@ -1027,6 +1104,7 @@ def compile_main(raw_args: Optional[Sequence[str]] = None) -> None:
             constraint_reqs=constraint_reqs,
             remove_constraints=args.remove_constraints,
             only_binary=args.only_binary,
+            target=target,
         )
     except RepositoryInitializationError as ex:
         logger.exception("Error initialization repository")
@@ -1113,6 +1191,72 @@ def norm_index_url(index_url: str) -> str:
     return index_url.rstrip("/")
 
 
+def add_target_args(parser: argparse.ArgumentParser) -> None:
+    """Add arguments describing the environment to compile for."""
+    group = parser.add_argument_group(
+        "target environment",
+        description=(
+            "Compile for an environment other than the one running req-compile. "
+            "Wheels are selected and environment markers are evaluated as if "
+            "installing on the target. Source distributions still have their "
+            "metadata extracted on this machine, so prefer --only-binary when "
+            "compiling for another platform."
+        ),
+    )
+    group.add_argument(
+        "--platform",
+        dest="target_platform",
+        default=None,
+        metavar="platform",
+        help=(
+            "Platform to compile for as a wheel platform tag, as accepted by pip, e.g. "
+            "manylinux_2_28_x86_64, manylinux2014_aarch64, macosx_11_0_arm64 or win_amd64. "
+            "A manylinux tag accepts wheels for that glibc or older, a macosx tag wheels for "
+            "that release or older. Defaults to the current platform."
+        ),
+    )
+    group.add_argument(
+        "--python-version",
+        dest="target_python_version",
+        default=None,
+        metavar="X.Y[.Z]",
+        help="Python version to compile for, e.g. 3.11. Defaults to the running interpreter.",
+    )
+    group.add_argument(
+        "--implementation",
+        dest="target_implementation",
+        default=None,
+        metavar="tag",
+        help=(
+            "Python implementation to compile for as a wheel interpreter tag, e.g. cp "
+            "for CPython or pp for PyPy. Defaults to cp when --platform or "
+            "--python-version is given, otherwise the running interpreter."
+        ),
+    )
+
+
+def build_target(args: argparse.Namespace) -> Optional[TargetEnvironment]:
+    """Build the target environment from parsed `add_target_args` arguments.
+
+    Returns:
+        The target, or None when compiling for the host.
+
+    Raises:
+        ValueError: If the arguments do not describe a supported target.
+    """
+    if (
+        args.target_platform is None
+        and args.target_python_version is None
+        and args.target_implementation is None
+    ):
+        return None
+    return TargetEnvironment.from_spec(
+        platform_spec=args.target_platform,
+        python_version=args.target_python_version,
+        implementation=args.target_implementation,
+    )
+
+
 def add_repo_args(parser: argparse.ArgumentParser) -> None:
     """Add arguments related to adding repositories to the command line"""
     group = parser.add_argument_group("repositories")

{req_compile-1.1.2 → req_compile-1.2.0}/req_compile/compile.py

@@ -17,6 +17,7 @@ from req_compile.dists import DependencyNode, DistributionCollection
 from req_compile.errors import MetadataError, NoCandidateException
 from req_compile.repos.repository import Repository
 from req_compile.repos.source import SourceRepository
+from req_compile.target import TargetEnvironment
 from req_compile.utils import (
     NormName,
     is_pinned_requirement,
@@ -310,6 +311,7 @@ def perform_compile(
     allow_circular_dependencies: bool = True,
     only_binary: Optional[Set[NormName]] = None,
     max_downgrade: Optional[int] = None,
+    target: Optional[TargetEnvironment] = None,
 ) -> Tuple[DistributionCollection, Set[DependencyNode]]:
     """Perform a compilation using the given inputs and constraints.
 
@@ -327,11 +329,15 @@ def perform_compile(
         allow_circular_dependencies: Whether to allow circular dependencies
         only_binary: Set of projects that should only consider binary distributions.
         max_downgrade: The maximum number of version downgrades that will be allowed for conflicts.
+        target: The environment to compile for. Defaults to the target of `repo`, which
+            itself defaults to the host.
 
     Returns:
         the solution and root nodes used to generate it
     """
-    results = req_compile.dists.DistributionCollection()
+    if target is None:
+        target = repo.target
+    results = req_compile.dists.DistributionCollection(target=target)
 
     constraint_nodes = set()
     nodes = set()

{req_compile-1.1.2 → req_compile-1.2.0}/req_compile/containers.py

@@ -1,32 +1,42 @@
 import itertools
 import os
 from pathlib import Path
-from typing import Any, Iterable, Iterator, List, Optional, Tuple, Union
+from typing import Any, Iterable, Iterator, List, Optional, Set, Tuple, Union
 
 import packaging.requirements
 import packaging.version
 from packaging.requirements import InvalidRequirement
 
-from req_compile.utils import reduce_requirements, req_iter_from_file
+from req_compile.target import TargetEnvironment, resolve
+from req_compile.utils import (
+    NormName,
+    normalize_project_name,
+    reduce_requirements,
+    req_iter_from_file,
+)
 
 
 def req_uses_extra(
-    req: packaging.requirements.Requirement, extra: Optional[str]
+    req: packaging.requirements.Requirement,
+    extra: Optional[str],
+    target: Optional[TargetEnvironment] = None,
 ) -> bool:
     """Determine if this requirement would be used with the given extra.
 
     If a distribution is requested with one of its extras, this filter will determine
     if the given requirement in its install requirements is provided. All base
     requirements that don't require the extra will also be included.
+
+    Environment markers other than ``extra`` are evaluated against ``target``,
+    which defaults to the host.
     """
     if extra and not req.marker:
         return False
     keep_req = True
     if req.marker:
-        extras = {"extra": ""}
-        if extra:
-            extras = {"extra": extra}
-        keep_req = req.marker.evaluate(extras)
+        keep_req = req.marker.evaluate(
+            resolve(target).marker_environment_with_extra(extra)
+        )
     return keep_req
 
 
@@ -48,17 +58,36 @@ class RequirementContainer:
         self.version: Optional[packaging.version.Version] = None
         self.hash: Optional[str] = None
         self.candidate: Any = None
+        # The extras that `reqs` is known to fully describe. `None` means the
+        # metadata is complete, so every extra the distribution provides is
+        # described. Solutions only record the extras that were active when they
+        # were compiled, so they populate this to mark the rest as unknown.
+        self.known_extras: Optional[Set[NormName]] = None
 
     def __iter__(self) -> Iterator[packaging.requirements.Requirement]:
         return iter(self.reqs)
 
     def requires(
-        self, extra: Optional[str] = None
+        self, extra: Optional[str] = None, target: Optional[TargetEnvironment] = None
     ) -> Iterable[packaging.requirements.Requirement]:
+        """The requirements that apply for the given extra on the given target."""
+        target = resolve(target)
         return reduce_requirements(
-            req for req in self.reqs if req_uses_extra(req, extra)
+            req for req in self.reqs if req_uses_extra(req, extra, target)
         )
 
+    def describes_extras(self, extras: Iterable[str]) -> bool:
+        """Whether this container describes the requirements of all the given extras.
+
+        A container built from incomplete metadata, such as a solution file, only
+        knows about the extras that were recorded in it. Asking it for the
+        requirements of any other extra silently yields nothing, so callers need
+        to check first and go find complete metadata instead.
+        """
+        if self.known_extras is None:
+            return True
+        return {normalize_project_name(extra) for extra in extras} <= self.known_extras
+
     def to_definition(
         self, extras: Optional[Iterable[str]]
     ) -> Tuple[str, Optional[packaging.version.Version]]: