req-compile 1.6.2__tar.gz → 1.6.4__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. {req_compile-1.6.2 → req_compile-1.6.4}/CHANGELOG.rst +34 -0
  2. {req_compile-1.6.2/req_compile.egg-info → req_compile-1.6.4}/PKG-INFO +35 -1
  3. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/compile.py +6 -0
  4. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/wasm/entry.py +151 -153
  5. req_compile-1.6.4/req_compile/wasm/errors.py +84 -0
  6. req_compile-1.6.4/req_compile/wasm/prior.py +203 -0
  7. {req_compile-1.6.2 → req_compile-1.6.4/req_compile.egg-info}/PKG-INFO +35 -1
  8. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile.egg-info/SOURCES.txt +3 -1
  9. {req_compile-1.6.2 → req_compile-1.6.4}/version.bzl +1 -1
  10. {req_compile-1.6.2 → req_compile-1.6.4}/LICENSE.txt +0 -0
  11. {req_compile-1.6.2 → req_compile-1.6.4}/MANIFEST.in +0 -0
  12. {req_compile-1.6.2 → req_compile-1.6.4}/README.rst +0 -0
  13. {req_compile-1.6.2 → req_compile-1.6.4}/pyproject.toml +0 -0
  14. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/__init__.py +0 -0
  15. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/__main__.py +0 -0
  16. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/candidates.py +0 -0
  17. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/cmdline.py +0 -0
  18. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/config.py +0 -0
  19. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/containers.py +0 -0
  20. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/dists.py +0 -0
  21. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/errors.py +0 -0
  22. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/filename.py +0 -0
  23. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/__init__.py +0 -0
  24. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/dist_info.py +0 -0
  25. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/extractor.py +0 -0
  26. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/metadata.py +0 -0
  27. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/patch.py +0 -0
  28. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/pyproject.py +0 -0
  29. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/metadata/source.py +0 -0
  30. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/py.typed +0 -0
  31. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/__init__.py +0 -0
  32. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/findlinks.py +0 -0
  33. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/multi.py +0 -0
  34. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/pypi.py +0 -0
  35. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/repository.py +0 -0
  36. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/solution.py +0 -0
  37. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/repos/source.py +0 -0
  38. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/target.py +0 -0
  39. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/utils.py +0 -0
  40. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/versions.py +0 -0
  41. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/wasm/__init__.py +0 -0
  42. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/wasm/files.py +0 -0
  43. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile/wasm/ingest.py +0 -0
  44. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile.egg-info/dependency_links.txt +0 -0
  45. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile.egg-info/entry_points.txt +0 -0
  46. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile.egg-info/requires.txt +0 -0
  47. {req_compile-1.6.2 → req_compile-1.6.4}/req_compile.egg-info/top_level.txt +0 -0
  48. {req_compile-1.6.2 → req_compile-1.6.4}/requirements.in +0 -0
  49. {req_compile-1.6.2 → req_compile-1.6.4}/setup.cfg +0 -0
  50. {req_compile-1.6.2 → req_compile-1.6.4}/setup.py +0 -0
@@ -1,3 +1,37 @@
1
+ New in version 1.6.4:
2
+
3
+ * A bumped pin in ``requirements_txt`` whose hash does not match no longer resolves quietly to the previous
4
+ version. The solver cannot tell, while it resolves, a pin a requirement excludes from one it left for lack
5
+ of a usable, hash-matching artifact, so the finished solution is checked: a pinned version that every
6
+ requirement on the package and every constraint file still allows must be the one used, with one of its
7
+ hashes, or the resolution fails naming the package, the platform with the least to offer and the artifacts
8
+ the index has for that version. This covers a wrong hash (also beside a matching sdist hash, which vouches
9
+ for nothing when sdists are not allowed), a release without a wheel for one platform, and an older release
10
+ on another index. A pin a requirement excludes still moves by itself.
11
+
12
+ New in version 1.6.3:
13
+
14
+ * A hub's ``requirements_txt`` is to the resolution what an existing lock is to pip-compile: each package
15
+ keeps the version the file pins, with the file's hashes, while ``requirements_in`` and the other packages
16
+ allow it, and is resolved again when they do not. Raising a bound in ``requirements_in`` therefore moves the
17
+ pin by itself; before, the file's pins were hard and the bump failed until ``--upgrade`` lifted them. The
18
+ file's hashes vouch for the pinned version only, so a bump with a wrong hash still fails naming the package,
19
+ while a version the file does not pin needs no hash and gets one when ``requirements.update`` writes it.
20
+ A mismatch on the pinned version stays an error even when another index or an older release would let the
21
+ solver resolve around it. A pin without a local segment covers every build of its release (``1.0`` covers
22
+ ``1.0+cpu``) and its hashes say which build is meant. With ``enforce_hashes = False`` the file's hashes are
23
+ not read at all.
24
+ * A soft pin, the recorded solution's version or the pinned file's, yields before the solver walks back a
25
+ reverse dependency to keep it: raising ``a>=2`` when the new ``a`` needs ``b>=2`` moves the pinned ``b``
26
+ instead of failing on ``a``. This also applies to ``py_reqs_compiler`` recycling a solution.
27
+ * The lock records whether a distribution's metadata came from the index's separately served file; once the
28
+ pinned file vouches for the artifact, its metadata is read from the wheel itself, as it always is for a
29
+ hash-pinned artifact, so an index cannot describe a newly pinned wheel through a stale sidecar.
30
+ * Every place that tells how to move a pin on purpose says the same thing: ``bazel run
31
+ @<hub>//:requirements.update -- --upgrade <package>`` (``-U`` with no name for every package); the variable
32
+ ``REQ_COMPILE_UPGRADE=<package>[,<package>]`` on that command when the hub itself cannot be resolved, or with
33
+ ``bazel mod deps`` for a hub that has no ``requirements_txt``.
34
+
1
35
  New in version 1.6.2:
2
36
 
3
37
  * ``@<hub>//:requirements.test`` compares the pinned file by meaning: it passes when the file pins exactly the
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: req-compile
3
- Version: 1.6.2
3
+ Version: 1.6.4
4
4
  Summary: Python requirements compiler
5
5
  Home-page: https://github.com/periareon/req-compile
6
6
  Author: Spencer Putt
@@ -38,6 +38,40 @@ Dynamic: requires-dist
38
38
  Dynamic: requires-python
39
39
  Dynamic: summary
40
40
 
41
+ New in version 1.6.4:
42
+
43
+ * A bumped pin in ``requirements_txt`` whose hash does not match no longer resolves quietly to the previous
44
+ version. The solver cannot tell, while it resolves, a pin a requirement excludes from one it left for lack
45
+ of a usable, hash-matching artifact, so the finished solution is checked: a pinned version that every
46
+ requirement on the package and every constraint file still allows must be the one used, with one of its
47
+ hashes, or the resolution fails naming the package, the platform with the least to offer and the artifacts
48
+ the index has for that version. This covers a wrong hash (also beside a matching sdist hash, which vouches
49
+ for nothing when sdists are not allowed), a release without a wheel for one platform, and an older release
50
+ on another index. A pin a requirement excludes still moves by itself.
51
+
52
+ New in version 1.6.3:
53
+
54
+ * A hub's ``requirements_txt`` is to the resolution what an existing lock is to pip-compile: each package
55
+ keeps the version the file pins, with the file's hashes, while ``requirements_in`` and the other packages
56
+ allow it, and is resolved again when they do not. Raising a bound in ``requirements_in`` therefore moves the
57
+ pin by itself; before, the file's pins were hard and the bump failed until ``--upgrade`` lifted them. The
58
+ file's hashes vouch for the pinned version only, so a bump with a wrong hash still fails naming the package,
59
+ while a version the file does not pin needs no hash and gets one when ``requirements.update`` writes it.
60
+ A mismatch on the pinned version stays an error even when another index or an older release would let the
61
+ solver resolve around it. A pin without a local segment covers every build of its release (``1.0`` covers
62
+ ``1.0+cpu``) and its hashes say which build is meant. With ``enforce_hashes = False`` the file's hashes are
63
+ not read at all.
64
+ * A soft pin, the recorded solution's version or the pinned file's, yields before the solver walks back a
65
+ reverse dependency to keep it: raising ``a>=2`` when the new ``a`` needs ``b>=2`` moves the pinned ``b``
66
+ instead of failing on ``a``. This also applies to ``py_reqs_compiler`` recycling a solution.
67
+ * The lock records whether a distribution's metadata came from the index's separately served file; once the
68
+ pinned file vouches for the artifact, its metadata is read from the wheel itself, as it always is for a
69
+ hash-pinned artifact, so an index cannot describe a newly pinned wheel through a stale sidecar.
70
+ * Every place that tells how to move a pin on purpose says the same thing: ``bazel run
71
+ @<hub>//:requirements.update -- --upgrade <package>`` (``-U`` with no name for every package); the variable
72
+ ``REQ_COMPILE_UPGRADE=<package>[,<package>]`` on that command when the hub itself cannot be resolved, or with
73
+ ``bazel mod deps`` for a hub that has no ``requirements_txt``.
74
+
41
75
  New in version 1.6.2:
42
76
 
43
77
  * ``@<hub>//:requirements.test`` compares the pinned file by meaning: it passes when the file pins exactly the
@@ -198,6 +198,12 @@ def compile_roots(
198
198
 
199
199
  if not is_possible(spec_req):
200
200
  logger.info("Requirement conflict for %s (%s)", node, spec_req)
201
+ if soft_pin is not None and is_possible(unpinned_req):
202
+ # The soft pin alone is in the way: it is a preference, so it
203
+ # yields before any reverse dependency is walked back to keep
204
+ # it (a lock's old `b==1.0` must not downgrade the `a` that now
205
+ # needs `b>=2`).
206
+ raise _no_candidate(no_candidate=False)
201
207
  # Try walking back. Some reverse dependency contributed a requirement
202
208
  # expression that lead to this node being unsolveable. Instead of attempting
203
209
  # to search the full space, find the one that conflicts with the most
@@ -56,6 +56,7 @@ from req_compile.repos.repository import (
56
56
  )
57
57
  from req_compile.target import TargetEnvironment
58
58
  from req_compile.utils import NormName
59
+ from req_compile.wasm.errors import HashMismatch, PinNotUsed, UnpinnedPackage
59
60
  from req_compile.wasm.files import (
60
61
  FactFiles,
61
62
  NeedFacts,
@@ -65,57 +66,22 @@ from req_compile.wasm.files import (
65
66
  file_constraints,
66
67
  pinned_hashes,
67
68
  )
68
- from req_compile.wasm.ingest import ingest_index, ingest_metadata, metadata_fact
69
+ from req_compile.wasm.ingest import ingest_index, ingest_metadata
70
+ from req_compile.wasm.prior import (
71
+ check_file_pins,
72
+ exact_version,
73
+ link_recorded_metadata,
74
+ prior_locks,
75
+ prior_pins,
76
+ same_release,
77
+ upgraded,
78
+ )
69
79
 
70
80
  ACCEPT_INDEX = "application/vnd.pypi.simple.v1+json, text/html;q=0.1"
71
81
 
72
82
  LOG = logging.getLogger("req_compile.wasm")
73
83
 
74
84
 
75
- class HashMismatch(Exception):
76
- """No artifact of a project matches the hashes its requirement file pins."""
77
-
78
- def __init__(self, name: str, pinned: Set[str], found: Dict[str, str]) -> None:
79
- """`found` maps the sha256 of each artifact the indexes offer to its file name."""
80
- super().__init__(name)
81
- self.name = name
82
- self.pinned = pinned
83
- self.found = found
84
-
85
- def __str__(self) -> str:
86
- available = ", ".join(
87
- "{} ({})".format(filename, sha256[:12])
88
- for sha256, filename in sorted(
89
- self.found.items(), key=lambda item: (item[1], item[0])
90
- )
91
- )
92
- return (
93
- "No artifact of `{}` on the index matches the --hash values pinned for it "
94
- "({}). The index offers: {}. Update the hashes along with the version, or "
95
- "set `enforce_hashes = False`."
96
- ).format(
97
- self.name,
98
- ", ".join(sorted(value[:12] for value in self.pinned)),
99
- available or "nothing",
100
- )
101
-
102
-
103
- class UnpinnedPackage(Exception):
104
- """A requirement file pins hashes, but none for this project."""
105
-
106
- def __init__(self, name: str) -> None:
107
- super().__init__(name)
108
- self.name = name
109
-
110
- def __str__(self) -> str:
111
- return (
112
- "`{}` is needed, but the requirement files pin no --hash for it. When a file "
113
- "pins hashes every package must be pinned, as with `pip --require-hashes`; "
114
- "otherwise an index could add a dependency the files never vouched for. Add "
115
- "it with its hash, or set `enforce_hashes = False`."
116
- ).format(self.name)
117
-
118
-
119
85
  class PageCache:
120
86
  """Candidates built from index pages, shared by the repositories of all targets.
121
87
 
@@ -187,13 +153,16 @@ class FactRepository(Repository):
187
153
  self._python_ok: Dict[str, bool] = {}
188
154
  self._tags_ok: Dict[Tuple[Any, ...], bool] = {}
189
155
  self._candidates: Dict[str, List[Candidate]] = {}
190
- # The `--hash` values pinned per project; set by `solve` once the files are read.
156
+ # The `--hash` values pinned per project by the requirement and constraint
157
+ # files; set by `solve` once the files are read. They bind every version.
191
158
  self.hashes: Dict[NormName, Set[str]] = {}
192
- # Whether a project without a pinned hash is an error. True when the
193
- # requirement files themselves pin hashes (`pip --require-hashes`); false
194
- # when only the hub's pinned requirements file does, which pins what it
195
- # lists and leaves new packages to be resolved and then recorded in it.
159
+ # Whether a project without a pinned hash is an error: true when the
160
+ # requirement files themselves pin hashes (`pip --require-hashes`).
196
161
  self.require_pins = False
162
+ # The hub's own pinned requirements file, per project: the version it
163
+ # pins and the hashes it gives for it. The version is a soft pin (see
164
+ # `_file_pinned`); the hashes bind that version only.
165
+ self.file_pins: Dict[NormName, Tuple[packaging.version.Version, Set[str]]] = {}
197
166
  self._entries: Dict[Tuple[str, str], Dict[str, Any]] = {}
198
167
 
199
168
  def __repr__(self) -> str:
@@ -311,14 +280,14 @@ class FactRepository(Repository):
311
280
  its hash is the usual cause, and naming the new version's artifacts is
312
281
  what makes the message actionable.
313
282
  """
314
- if not self.hashes or not candidates:
283
+ if not candidates:
315
284
  return candidates
316
285
  key = utils.normalize_project_name(name)
317
286
  pinned = self.hashes.get(key)
318
287
  if not pinned:
319
- if self.require_pins:
288
+ if self.hashes and self.require_pins:
320
289
  raise UnpinnedPackage(name)
321
- return candidates
290
+ return self._file_pinned(key, name, candidates, req)
322
291
  matching = [
323
292
  candidate
324
293
  for candidate in candidates
@@ -337,12 +306,79 @@ class FactRepository(Repository):
337
306
  self.entry(candidate)["sha256"]: candidate.filename or ""
338
307
  for candidate in relevant
339
308
  }
309
+ self._note_mismatch(key, name, pinned, found)
310
+ return []
311
+
312
+ def _note_mismatch(
313
+ self, key: NormName, name: str, pinned: Set[str], found: Dict[str, str]
314
+ ) -> None:
340
315
  mismatch = self.mismatches.get(key)
341
316
  if mismatch is None:
342
317
  self.mismatches[key] = HashMismatch(name, pinned, found)
343
318
  else:
344
319
  mismatch.found.update(found)
345
- return []
320
+
321
+ def _file_pinned(
322
+ self,
323
+ key: NormName,
324
+ name: str,
325
+ candidates: List[Candidate],
326
+ req: Optional[packaging.requirements.Requirement],
327
+ ) -> List[Candidate]:
328
+ """Apply the hub's pinned requirements file to a project's candidates.
329
+
330
+ The file's version is a soft pin: `solve` hands it to the solver, which
331
+ prefers it while the requirements allow it and moves on when they do not,
332
+ as pip-compile does with an existing lock. The file's hashes vouch for
333
+ that version only: an artifact of the pinned version with another hash is
334
+ left out, and when none of its artifacts match while the requirement still
335
+ allows the version, that is an error, so that a pin bumped without its hash
336
+ cannot resolve quietly to something else. Other versions are not pinned
337
+ by hash; `requirements.update` writes theirs.
338
+ """
339
+ pin = self.file_pins.get(key)
340
+ if pin is None or not pin[1]:
341
+ return candidates
342
+ version, hashes = pin
343
+ # A local build (`1.0+cpu`) is an artifact of the pin `1.0`: the hashes
344
+ # say which build the file means, so the solver cannot drift onto one
345
+ # the soft pin `==1.0` would also accept.
346
+ of_version = [c for c in candidates if same_release(c.version, version)]
347
+ # Only artifacts the solve may install count: a matching hash on an sdist
348
+ # the hub does not allow vouches for nothing the solver can pick.
349
+ usable = [
350
+ c
351
+ for c in of_version
352
+ if self.allow_sdists or c.type == DistributionType.WHEEL
353
+ ]
354
+ if not usable:
355
+ # Nothing of the pinned version for this target; the pin yields here
356
+ # and `_check_file_pins` decides whether that was legitimate.
357
+ return candidates
358
+ matching = [c for c in usable if self.entry(c)["sha256"] in hashes]
359
+ if not matching and (
360
+ req is None or req.specifier.contains(version, prereleases=True)
361
+ ):
362
+ found = {self.entry(c)["sha256"]: c.filename or "" for c in usable}
363
+ self._note_mismatch(key, name, hashes, found)
364
+ return []
365
+ return [c for c in candidates if c not in of_version or c in matching]
366
+
367
+ def artifacts_of(self, name: str, version: packaging.version.Version) -> List[str]:
368
+ """The artifacts of a release this repository has seen, named with their hash."""
369
+ page = self.page_url(name)
370
+ return [
371
+ "{} ({})".format(c.filename or "", (self.entry(c)["sha256"] or "")[:12])
372
+ for c in self._candidates.get(page) or []
373
+ if same_release(c.version, version)
374
+ ]
375
+
376
+ def file_vouches(self, candidate: Candidate) -> bool:
377
+ """Whether the pinned file pins this candidate's version with a hash."""
378
+ pin = self.file_pins.get(utils.normalize_project_name(candidate.name))
379
+ return (
380
+ pin is not None and bool(pin[1]) and same_release(candidate.version, pin[0])
381
+ )
346
382
 
347
383
  def _recorded(
348
384
  self, req: packaging.requirements.Requirement
@@ -354,6 +390,19 @@ class FactRepository(Repository):
354
390
  return None
355
391
  if self.require_pins and key not in self.hashes:
356
392
  raise UnpinnedPackage(req.name)
393
+ pin = self.file_pins.get(key)
394
+ if pin is not None:
395
+ try:
396
+ recorded_version = packaging.version.Version(entry.get("version") or "")
397
+ except packaging.version.InvalidVersion:
398
+ return None
399
+ if (
400
+ not same_release(recorded_version, pin[0])
401
+ or entry.get("sha256") not in pin[1]
402
+ ):
403
+ # The file pins another version, or another artifact of this one
404
+ # (a Dependabot bump): the page decides, not the recorded choice.
405
+ return None
357
406
  index = (entry.get("index") or "").rstrip("/")
358
407
  if index != self.index_url:
359
408
  # Recorded from another index. Its repository answers while that index
@@ -416,7 +465,10 @@ class FactRepository(Repository):
416
465
  # for the wheel the whole file is fetched too, so that the download's
417
466
  # digest can pin it.
418
467
  use_sidecar = (
419
- isinstance(core, dict) and bool(entry["sha256"]) and not self.hashes
468
+ isinstance(core, dict)
469
+ and bool(entry["sha256"])
470
+ and not self.hashes
471
+ and not self.file_vouches(candidate)
420
472
  )
421
473
  return {
422
474
  "id": self.metadata_key(candidate),
@@ -431,8 +483,13 @@ class FactRepository(Repository):
431
483
  def ensure_metadata(self, candidate: Candidate) -> Optional[Dict[str, Any]]:
432
484
  """The metadata fact of a candidate, requesting it when missing."""
433
485
  fact = self.facts.get(self.metadata_key(candidate))
434
- if fact is None:
486
+ if fact is None or (fact.get("sidecar") and self.file_vouches(candidate)):
487
+ # No metadata yet, or metadata an index served separately for an
488
+ # artifact the pinned file now vouches for: the wheel's own is read
489
+ # instead (`metadata_request` asks for the whole file), so that the
490
+ # index cannot describe a pinned wheel as something it is not.
435
491
  self.requests.add(self.metadata_request(candidate))
492
+ return None
436
493
  return fact
437
494
 
438
495
  def resolve_candidate(
@@ -466,72 +523,6 @@ class FactRepository(Repository):
466
523
  return dist, True
467
524
 
468
525
 
469
- def _prior_locks(
470
- spec: Dict[str, Any], upgrade: Any
471
- ) -> Dict[str, Dict[NormName, Dict[str, Any]]]:
472
- """The recorded distributions per platform, minus the packages being upgraded.
473
-
474
- `prior` is either the recorded solution, `{"locks": {platform: {name: entry}}}`,
475
- or, for compatibility, a plain mapping of name to version, which pins but
476
- cannot stand in for index pages.
477
- """
478
- prior = spec.get("prior")
479
- if not prior or upgrade == "all":
480
- return {}
481
- skip = {utils.normalize_project_name(name) for name in (upgrade or [])}
482
- if "locks" in prior:
483
- locks = prior["locks"]
484
- else:
485
- locks = {
486
- platform: {name: {"version": version} for name, version in prior.items()}
487
- for platform in spec["platforms"]
488
- }
489
- result: Dict[str, Dict[NormName, Dict[str, Any]]] = {}
490
- for platform, packages in locks.items():
491
- entries: Dict[NormName, Dict[str, Any]] = {}
492
- for name, entry in packages.items():
493
- key = utils.normalize_project_name(entry.get("package") or name)
494
- if key in skip:
495
- continue
496
- entries[key] = entry
497
- result[platform] = entries
498
- return result
499
-
500
-
501
- def _link_recorded_metadata(
502
- prior: Dict[str, Dict[NormName, Dict[str, Any]]], facts: Dict[str, Any]
503
- ) -> None:
504
- """Serve the metadata of recorded distributions from the `md:` facts.
505
-
506
- The lockfile keeps metadata once per distinct text, keyed by its digest, and
507
- each recorded distribution names the digest of its own; the solver looks
508
- metadata up by distribution, so the two are joined here.
509
- """
510
- for packages in prior.values():
511
- for entry in packages.values():
512
- digest = entry.get("metadata")
513
- identity = entry.get("sha256") or entry.get("url")
514
- if not digest or not identity or "meta:" + identity in facts:
515
- continue
516
- text = facts.get("md:" + digest)
517
- if isinstance(text, str):
518
- facts["meta:" + identity] = metadata_fact(decode_text(text))
519
-
520
-
521
- def _prior_pins(
522
- locks: Dict[str, Dict[NormName, Dict[str, Any]]],
523
- ) -> Dict[NormName, packaging.version.Version]:
524
- """The versions a prior solution chose, as soft pins."""
525
- pins: Dict[NormName, packaging.version.Version] = {}
526
- for packages in locks.values():
527
- for key, entry in packages.items():
528
- try:
529
- pins.setdefault(key, packaging.version.Version(entry["version"]))
530
- except (KeyError, packaging.version.InvalidVersion):
531
- continue
532
- return pins
533
-
534
-
535
526
  class _Hub:
536
527
  def __init__(
537
528
  self,
@@ -552,9 +543,9 @@ class _Hub:
552
543
  # Whether the pinned hashes have been read; metadata is requested only
553
544
  # after, since they decide how it is fetched.
554
545
  self.hashes_ready = not spec.get("enforce_hashes", True)
555
- prior = _prior_locks(spec, upgrade)
556
- self.pins = _prior_pins(prior)
557
- _link_recorded_metadata(prior, facts)
546
+ prior = prior_locks(spec, upgrade)
547
+ self.pins = prior_pins(prior)
548
+ link_recorded_metadata(prior, facts)
558
549
  self.mismatches: Dict[NormName, HashMismatch] = {}
559
550
  self.indexes: List[List[FactRepository]] = []
560
551
  self.repos: List[Repository] = []
@@ -605,12 +596,15 @@ def _render(
605
596
  requires_python = ""
606
597
  index = ""
607
598
  digest = ""
599
+ sidecar = False
608
600
  if isinstance(meta.origin, FactRepository):
609
601
  entry = meta.origin.entry(meta.candidate)
610
602
  core_metadata = entry.get("core_metadata", False)
611
603
  requires_python = entry.get("requires_python") or ""
612
604
  index = meta.origin.index_url
613
- hash_pinned = node.key in meta.origin.hashes
605
+ hash_pinned = node.key in meta.origin.hashes or meta.origin.file_vouches(
606
+ meta.candidate
607
+ )
614
608
  fact = meta.origin.facts.get(meta.origin.metadata_key(meta.candidate)) or {}
615
609
  if not sha256:
616
610
  # The index gave no hash, so the file itself was fetched and the
@@ -619,6 +613,7 @@ def _render(
619
613
  digest = fact.get("digest") or ""
620
614
  if digest:
621
615
  metadata[digest] = fact["metadata"]
616
+ sidecar = bool(fact.get("sidecar"))
622
617
  packages[node.key] = {
623
618
  "package": meta.name,
624
619
  "version": str(meta.version),
@@ -630,6 +625,7 @@ def _render(
630
625
  "requires_python": requires_python,
631
626
  "index": index,
632
627
  "metadata": digest,
628
+ "sidecar": sidecar,
633
629
  "deps": sorted(dep.key for dep in node.dependencies),
634
630
  "via": sorted(via),
635
631
  }
@@ -706,13 +702,6 @@ def _speculate(hubs: List[_Hub], facts: Dict[str, Any]) -> None:
706
702
  _prefetch(hub, req)
707
703
 
708
704
 
709
- def _upgraded(upgrade: Any) -> Optional[Set[NormName]]:
710
- """The projects `REQ_COMPILE_UPGRADE` names, or None for all of them."""
711
- if upgrade == "all":
712
- return None
713
- return {utils.normalize_project_name(name) for name in (upgrade or [])}
714
-
715
-
716
705
  def _failed_requests(hub: _Hub, ex: Exception) -> List[Dict[str, Any]]:
717
706
  """The requests behind a resolution error whose download failed.
718
707
 
@@ -796,25 +785,31 @@ def solve(payload: Dict[str, Any]) -> Dict[str, Any]:
796
785
  path, reqs.parameters or [], hub.spec["indexes"]
797
786
  )
798
787
  )
799
- # The hub's pinned requirements file: hard pins, with hashes,
800
- # for the packages it lists, except the ones being upgraded,
801
- # which resolve from the index again and are then rewritten.
788
+ # The hub's pinned requirements file, the lock pip-compile
789
+ # would read: the version it pins is preferred while the
790
+ # requirements allow it and dropped when they do not, its
791
+ # hashes vouch for that version, and packages being upgraded
792
+ # are resolved from the index again and then rewritten. Its
793
+ # versions outrank the recorded solution's: a bump in the file
794
+ # is what the next resolution uses.
802
795
  pins = hub.spec.get("pins")
803
- lifted = _upgraded(upgrade)
796
+ lifted = upgraded(upgrade)
797
+ file_pins = {}
804
798
  if pins:
805
799
  pinned = RequirementsFile.from_file(pins)
806
- constraints.append(
807
- RequirementsFile(
808
- pinned.name,
809
- [
810
- req
811
- for req in pinned.reqs
812
- if lifted is not None
813
- and utils.normalize_project_name(req.name)
814
- not in lifted
815
- ],
816
- parameters=pinned.parameters,
817
- )
800
+ file_hashes = (
801
+ pinned_hashes([pins])
802
+ if hub.spec.get("enforce_hashes", True)
803
+ else {}
804
+ )
805
+ for req in pinned.reqs:
806
+ key = utils.normalize_project_name(req.name)
807
+ version = exact_version(req)
808
+ if version is None or lifted is None or key in lifted:
809
+ continue
810
+ file_pins[key] = (version, set(file_hashes.get(key, ())))
811
+ hub.pins.update(
812
+ {key: version for key, (version, _) in file_pins.items()}
818
813
  )
819
814
  constraints.extend(
820
815
  file_constraints(
@@ -827,14 +822,11 @@ def solve(payload: Dict[str, Any]) -> Dict[str, Any]:
827
822
  + list(hub.spec.get("constraints", []))
828
823
  )
829
824
  hashes = {name: set(values) for name, values in strict.items()}
830
- if pins:
831
- for name, values in pinned_hashes([pins]).items():
832
- if lifted is not None and name not in lifted:
833
- hashes.setdefault(name, set()).update(values)
834
825
  for repos in hub.indexes:
835
826
  for repo in repos:
836
827
  repo.hashes = hashes
837
828
  repo.require_pins = bool(strict)
829
+ repo.file_pins = file_pins
838
830
  hub.hashes_ready = True
839
831
  solutions = perform_compile_group(
840
832
  input_reqs,
@@ -844,11 +836,13 @@ def solve(payload: Dict[str, Any]) -> Dict[str, Any]:
844
836
  only_binary=set() if hub.allow_sdists else AllOnlyBinarySet(),
845
837
  pins=hub.pins,
846
838
  )
839
+ check_file_pins(hub, file_pins, constraints, solutions)
847
840
  except (
848
841
  NoCandidateException,
849
842
  NoCommonVersionException,
850
843
  MetadataError,
851
844
  HashMismatch,
845
+ PinNotUsed,
852
846
  UnpinnedPackage,
853
847
  RequirementsFileError,
854
848
  ) as ex:
@@ -897,6 +891,10 @@ def ingest(payload: Dict[str, Any]) -> Dict[str, Any]:
897
891
  fact = ingest_metadata(body)
898
892
  if payload.get("sha256"):
899
893
  fact["sha256"] = payload["sha256"]
894
+ # Whether the text came from the index's separately served file rather
895
+ # than the distribution itself; `ensure_metadata` reads the latter for
896
+ # artifacts the pinned file vouches for.
897
+ fact["sidecar"] = bool(payload.get("sidecar"))
900
898
  return fact
901
899
  raise ValueError("unknown ingest kind {!r}".format(payload["kind"]))
902
900
 
@@ -0,0 +1,84 @@
1
+ """The errors the solver raises about pinned hashes, beyond req-compile's own."""
2
+
3
+ from typing import Dict, List, Set
4
+
5
+ import packaging.version
6
+
7
+
8
+ class HashMismatch(Exception):
9
+ """No artifact of a project matches the hashes its requirement file pins."""
10
+
11
+ def __init__(self, name: str, pinned: Set[str], found: Dict[str, str]) -> None:
12
+ """`found` maps the sha256 of each artifact the indexes offer to its file name."""
13
+ super().__init__(name)
14
+ self.name = name
15
+ self.pinned = pinned
16
+ self.found = found
17
+
18
+ def __str__(self) -> str:
19
+ available = ", ".join(
20
+ "{} ({})".format(filename, sha256[:12])
21
+ for sha256, filename in sorted(
22
+ self.found.items(), key=lambda item: (item[1], item[0])
23
+ )
24
+ )
25
+ return (
26
+ "No artifact of `{}` on the index matches the --hash values pinned for it "
27
+ "({}). The index offers: {}. Update the hashes along with the version, or "
28
+ "set `enforce_hashes = False`."
29
+ ).format(
30
+ self.name,
31
+ ", ".join(sorted(value[:12] for value in self.pinned)),
32
+ available or "nothing",
33
+ )
34
+
35
+
36
+ class PinNotUsed(Exception):
37
+ """The pinned file names a version, with hashes, that the solution does not use.
38
+
39
+ Raised after a solve when the requirements still allow the pinned version, so
40
+ the solver moved away from it only because no artifact of it the platform can
41
+ use carries one of the pinned hashes: a bump with a wrong hash, or one to a
42
+ release without a wheel for the platform. Resolving quietly to another version
43
+ would hide exactly what the pin was meant to say.
44
+ """
45
+
46
+ def __init__( # pylint: disable=too-many-arguments
47
+ self,
48
+ name: str,
49
+ version: packaging.version.Version,
50
+ chosen: packaging.version.Version,
51
+ target: str,
52
+ hashes: Set[str],
53
+ offers: List[str],
54
+ ) -> None:
55
+ self.offers = len(offers)
56
+ super().__init__(
57
+ "`{name}` is pinned to {version} in the hub's requirements file but the hub "
58
+ "resolved {chosen} for {target}: no artifact of {version} the platform can "
59
+ "use matches the --hash values pinned for it ({hashes}). The index offers: "
60
+ "{offers}.".format(
61
+ name=name,
62
+ version=version,
63
+ chosen=chosen,
64
+ target=target,
65
+ hashes=", ".join(sorted(value[:12] for value in hashes)),
66
+ offers=", ".join(offers) or "nothing of that version",
67
+ )
68
+ )
69
+
70
+
71
+ class UnpinnedPackage(Exception):
72
+ """A requirement file pins hashes, but none for this project."""
73
+
74
+ def __init__(self, name: str) -> None:
75
+ super().__init__(name)
76
+ self.name = name
77
+
78
+ def __str__(self) -> str:
79
+ return (
80
+ "`{}` is needed, but the requirement files pin no --hash for it. When a file "
81
+ "pins hashes every package must be pinned, as with `pip --require-hashes`; "
82
+ "otherwise an index could add a dependency the files never vouched for. Add "
83
+ "it with its hash, or set `enforce_hashes = False`."
84
+ ).format(self.name)
@@ -0,0 +1,203 @@
1
+ """What a re-resolution starts from: the recorded solution and the pinned file.
2
+
3
+ A hub may come with the solution a previous evaluation recorded in the lock
4
+ file, and with the pinned requirements file it maintains in the workspace. Both
5
+ feed the solver soft pins, versions it prefers while the requirements allow them
6
+ (see `_file_pinned` in `entry.py` for how the file's hashes bind), and
7
+ `REQ_COMPILE_UPGRADE` names the packages left out of both.
8
+ """
9
+
10
+ from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
11
+
12
+ import packaging.requirements
13
+ import packaging.version
14
+
15
+ from req_compile import utils
16
+ from req_compile.utils import NormName
17
+ from req_compile.wasm.errors import PinNotUsed
18
+ from req_compile.wasm.files import decode_text
19
+ from req_compile.wasm.ingest import metadata_fact
20
+
21
+
22
+ def prior_locks(
23
+ spec: Dict[str, Any], upgrade: Any
24
+ ) -> Dict[str, Dict[NormName, Dict[str, Any]]]:
25
+ """The recorded distributions per platform, minus the packages being upgraded.
26
+
27
+ `prior` is either the recorded solution, `{"locks": {platform: {name: entry}}}`,
28
+ or, for compatibility, a plain mapping of name to version, which pins but
29
+ cannot stand in for index pages.
30
+ """
31
+ prior = spec.get("prior")
32
+ if not prior or upgrade == "all":
33
+ return {}
34
+ skip = {utils.normalize_project_name(name) for name in (upgrade or [])}
35
+ if "locks" in prior:
36
+ locks = prior["locks"]
37
+ else:
38
+ locks = {
39
+ platform: {name: {"version": version} for name, version in prior.items()}
40
+ for platform in spec["platforms"]
41
+ }
42
+ result: Dict[str, Dict[NormName, Dict[str, Any]]] = {}
43
+ for platform, packages in locks.items():
44
+ entries: Dict[NormName, Dict[str, Any]] = {}
45
+ for name, entry in packages.items():
46
+ key = utils.normalize_project_name(entry.get("package") or name)
47
+ if key in skip:
48
+ continue
49
+ entries[key] = entry
50
+ result[platform] = entries
51
+ return result
52
+
53
+
54
+ def link_recorded_metadata(
55
+ prior: Dict[str, Dict[NormName, Dict[str, Any]]], facts: Dict[str, Any]
56
+ ) -> None:
57
+ """Serve the metadata of recorded distributions from the `md:` facts.
58
+
59
+ The lockfile keeps metadata once per distinct text, keyed by its digest, and
60
+ each recorded distribution names the digest of its own; the solver looks
61
+ metadata up by distribution, so the two are joined here.
62
+ """
63
+ for packages in prior.values():
64
+ for entry in packages.values():
65
+ digest = entry.get("metadata")
66
+ identity = entry.get("sha256") or entry.get("url")
67
+ if not digest or not identity or "meta:" + identity in facts:
68
+ continue
69
+ text = facts.get("md:" + digest)
70
+ if isinstance(text, str):
71
+ fact = metadata_fact(decode_text(text))
72
+ # Where the text came from, so that an artifact the pinned file
73
+ # now vouches for gets its metadata read from the wheel itself.
74
+ fact["sidecar"] = bool(entry.get("sidecar"))
75
+ facts["meta:" + identity] = fact
76
+
77
+
78
+ def same_release(
79
+ version: packaging.version.Version, pin: packaging.version.Version
80
+ ) -> bool:
81
+ """Whether `version` is the release the pin `pin` names.
82
+
83
+ A pin without a local segment names every build of that release (`1.0`
84
+ covers `1.0+cpu`), which the pin's hashes then tell apart; a pin with one
85
+ names that build alone.
86
+ """
87
+ if pin.local:
88
+ return version == pin
89
+ return version.public == pin.public
90
+
91
+
92
+ def prior_pins(
93
+ locks: Dict[str, Dict[NormName, Dict[str, Any]]],
94
+ ) -> Dict[NormName, packaging.version.Version]:
95
+ """The versions a prior solution chose, as soft pins."""
96
+ pins: Dict[NormName, packaging.version.Version] = {}
97
+ for packages in locks.values():
98
+ for key, entry in packages.items():
99
+ try:
100
+ pins.setdefault(key, packaging.version.Version(entry["version"]))
101
+ except (KeyError, packaging.version.InvalidVersion):
102
+ continue
103
+ return pins
104
+
105
+
106
+ def exact_version(
107
+ req: packaging.requirements.Requirement,
108
+ ) -> Optional[packaging.version.Version]:
109
+ """The version a `name==version` line pins, or None for any other line."""
110
+ specs = list(req.specifier)
111
+ if (
112
+ len(specs) != 1
113
+ or specs[0].operator not in ("==", "===")
114
+ or specs[0].version.endswith(".*")
115
+ ):
116
+ return None
117
+ try:
118
+ return packaging.version.Version(specs[0].version)
119
+ except packaging.version.InvalidVersion:
120
+ return None
121
+
122
+
123
+ def upgraded(upgrade: Any) -> Optional[Set[NormName]]:
124
+ """The projects `REQ_COMPILE_UPGRADE` names, or None for all of them."""
125
+ if upgrade == "all":
126
+ return None
127
+ return {utils.normalize_project_name(name) for name in (upgrade or [])}
128
+
129
+
130
+ def _allows(
131
+ req: Optional[packaging.requirements.Requirement],
132
+ version: packaging.version.Version,
133
+ ) -> bool:
134
+ return req is None or req.specifier.contains(version, prereleases=True)
135
+
136
+
137
+ def check_file_pins(
138
+ hub: Any,
139
+ file_pins: Dict[NormName, Tuple[packaging.version.Version, Set[str]]],
140
+ constraints: Sequence[Any],
141
+ solutions: Sequence[Tuple[Any, Any]],
142
+ ) -> None:
143
+ """Fail when the solution left a pinned version the requirements still allow.
144
+
145
+ The file's version is a soft pin, so the solver may move away from it; that
146
+ is right when `requirements.in`, a constraint or another package's
147
+ requirement excludes the version, and wrong when the only reason was that
148
+ no artifact of the version the platform can use carried one of the file's
149
+ hashes (a bump with a wrong hash, a release without a wheel for a platform,
150
+ another index offering an older release). The solver cannot tell the two
151
+ apart while it resolves, so the solution is checked afterwards against the
152
+ requirements it ended up with: a pinned version every requirement on the
153
+ package still allows must be the one used, with one of its hashes.
154
+ """
155
+ for key, (version, hashes) in file_pins.items():
156
+ if not hashes:
157
+ continue
158
+ if any(
159
+ utils.normalize_project_name(req.name) == key and not _allows(req, version)
160
+ for container in constraints
161
+ for req in getattr(container, "reqs", [])
162
+ ):
163
+ continue
164
+ # Every platform that left the pin; the one with the least to offer for
165
+ # the pinned version is the one to name, since the platforms agree on a
166
+ # version and the poorest page decides for all of them.
167
+ left: List[PinNotUsed] = []
168
+ for target, repos, (solution, _) in zip(hub.targets, hub.indexes, solutions):
169
+ for node in solution:
170
+ meta = node.metadata
171
+ if (
172
+ node.key != key
173
+ or meta is None
174
+ or meta.meta
175
+ or meta.candidate is None
176
+ or meta.version is None
177
+ ):
178
+ continue
179
+ sha256 = ""
180
+ if hasattr(meta.origin, "entry"): # a FactRepository
181
+ sha256 = meta.origin.entry(meta.candidate).get("sha256") or ""
182
+ if same_release(meta.version, version) and sha256 in hashes:
183
+ continue
184
+ if any(
185
+ not _allows(reverse.dependencies.get(node), version)
186
+ for reverse in node.reverse_deps
187
+ ):
188
+ continue
189
+ mismatch = hub.mismatches.get(key)
190
+ if mismatch is not None:
191
+ raise mismatch
192
+ offers = [
193
+ offer
194
+ for repo in repos
195
+ for offer in repo.artifacts_of(meta.name, version)
196
+ ]
197
+ left.append(
198
+ PinNotUsed(
199
+ meta.name, version, meta.version, target.name, hashes, offers
200
+ )
201
+ )
202
+ if left:
203
+ raise min(left, key=lambda error: error.offers)
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: req-compile
3
- Version: 1.6.2
3
+ Version: 1.6.4
4
4
  Summary: Python requirements compiler
5
5
  Home-page: https://github.com/periareon/req-compile
6
6
  Author: Spencer Putt
@@ -38,6 +38,40 @@ Dynamic: requires-dist
38
38
  Dynamic: requires-python
39
39
  Dynamic: summary
40
40
 
41
+ New in version 1.6.4:
42
+
43
+ * A bumped pin in ``requirements_txt`` whose hash does not match no longer resolves quietly to the previous
44
+ version. The solver cannot tell, while it resolves, a pin a requirement excludes from one it left for lack
45
+ of a usable, hash-matching artifact, so the finished solution is checked: a pinned version that every
46
+ requirement on the package and every constraint file still allows must be the one used, with one of its
47
+ hashes, or the resolution fails naming the package, the platform with the least to offer and the artifacts
48
+ the index has for that version. This covers a wrong hash (also beside a matching sdist hash, which vouches
49
+ for nothing when sdists are not allowed), a release without a wheel for one platform, and an older release
50
+ on another index. A pin a requirement excludes still moves by itself.
51
+
52
+ New in version 1.6.3:
53
+
54
+ * A hub's ``requirements_txt`` is to the resolution what an existing lock is to pip-compile: each package
55
+ keeps the version the file pins, with the file's hashes, while ``requirements_in`` and the other packages
56
+ allow it, and is resolved again when they do not. Raising a bound in ``requirements_in`` therefore moves the
57
+ pin by itself; before, the file's pins were hard and the bump failed until ``--upgrade`` lifted them. The
58
+ file's hashes vouch for the pinned version only, so a bump with a wrong hash still fails naming the package,
59
+ while a version the file does not pin needs no hash and gets one when ``requirements.update`` writes it.
60
+ A mismatch on the pinned version stays an error even when another index or an older release would let the
61
+ solver resolve around it. A pin without a local segment covers every build of its release (``1.0`` covers
62
+ ``1.0+cpu``) and its hashes say which build is meant. With ``enforce_hashes = False`` the file's hashes are
63
+ not read at all.
64
+ * A soft pin, the recorded solution's version or the pinned file's, yields before the solver walks back a
65
+ reverse dependency to keep it: raising ``a>=2`` when the new ``a`` needs ``b>=2`` moves the pinned ``b``
66
+ instead of failing on ``a``. This also applies to ``py_reqs_compiler`` recycling a solution.
67
+ * The lock records whether a distribution's metadata came from the index's separately served file; once the
68
+ pinned file vouches for the artifact, its metadata is read from the wheel itself, as it always is for a
69
+ hash-pinned artifact, so an index cannot describe a newly pinned wheel through a stale sidecar.
70
+ * Every place that tells how to move a pin on purpose says the same thing: ``bazel run
71
+ @<hub>//:requirements.update -- --upgrade <package>`` (``-U`` with no name for every package); the variable
72
+ ``REQ_COMPILE_UPGRADE=<package>[,<package>]`` on that command when the hub itself cannot be resolved, or with
73
+ ``bazel mod deps`` for a hub that has no ``requirements_txt``.
74
+
41
75
  New in version 1.6.2:
42
76
 
43
77
  * ``@<hub>//:requirements.test`` compares the pinned file by meaning: it passes when the file pins exactly the
@@ -43,5 +43,7 @@ req_compile/repos/solution.py
43
43
  req_compile/repos/source.py
44
44
  req_compile/wasm/__init__.py
45
45
  req_compile/wasm/entry.py
46
+ req_compile/wasm/errors.py
46
47
  req_compile/wasm/files.py
47
- req_compile/wasm/ingest.py
48
+ req_compile/wasm/ingest.py
49
+ req_compile/wasm/prior.py
@@ -1,3 +1,3 @@
1
1
  """req-compile version"""
2
2
 
3
- VERSION = "1.6.2"
3
+ VERSION = "1.6.4"
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes