@biffo/cli 0.296.14 → 0.296.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,485 @@
|
|
|
1
|
+
"""No module in this plugin's own package may re-derive an expression one of
|
|
2
|
+
its helpers already holds — for **every** such helper, enumerated from the
|
|
3
|
+
code itself rather than named here one at a time.
|
|
4
|
+
|
|
5
|
+
## The class this closes (biffo-template#1587)
|
|
6
|
+
|
|
7
|
+
A shared helper is introduced to fix a defect, adopted at some call sites and
|
|
8
|
+
not others, and the helper's existence makes everyone believe it is fixed. The
|
|
9
|
+
unconverted sites keep the bug, behind a closed issue, looking done. Measured
|
|
10
|
+
twice in `biffo-plugin-marketing`: `public_base_url_for` was adopted at
|
|
11
|
+
**1 call site of 3** (marketing#72 — the two missed sites 500'd both packs),
|
|
12
|
+
and an artefact-body parser was filed against **2** sites when the tree held
|
|
13
|
+
**7 across 6 files** (marketing#49, swept in marketing#111).
|
|
14
|
+
|
|
15
|
+
`biffo-plugin-marketing` and `biffo-plugin-ideation` each grew a copy of this
|
|
16
|
+
guard by hand, after the fact, once their own instance of the class had
|
|
17
|
+
already shipped. #1587 is that the skeleton — the thing every plugin repo is
|
|
18
|
+
actually born from — carried no such test, so a **third** plugin starts life
|
|
19
|
+
non-adopting by construction and repeats the same class a third time before
|
|
20
|
+
anyone notices. Shipping the sweep here, rather than as a pattern to remember
|
|
21
|
+
to copy, is the fix: it travels with the code it watches from commit one.
|
|
22
|
+
|
|
23
|
+
## Why this is parameterised rather than a copy of the reference
|
|
24
|
+
|
|
25
|
+
`biffo-plugin-marketing`'s copy of this file hardcodes `src / "marketing"` as
|
|
26
|
+
the package it sweeps. That literal cannot be copied into this skeleton: every
|
|
27
|
+
plugin scaffolded from it gets a *different* package name (`biffo plugin
|
|
28
|
+
create` rewrites `example_plugin` throughout the repo — see
|
|
29
|
+
`.scaffold-tokens.json` and `cli/src/lib/plugin-scaffold.ts`'s
|
|
30
|
+
`substitutions()`), so a hardcoded name here would be correct for zero
|
|
31
|
+
scaffolded repos. `_package_dir()` below finds the package by shape — the sole
|
|
32
|
+
directory under `src/` with an `__init__.py` — so it resolves correctly both
|
|
33
|
+
in this template (`example_plugin`) and in every plugin it produces, with
|
|
34
|
+
nothing to rename when the scaffold runs.
|
|
35
|
+
|
|
36
|
+
## What counts as a consolidation helper
|
|
37
|
+
|
|
38
|
+
A function whose entire body is ``return <expression>`` (a docstring is
|
|
39
|
+
allowed). That is the shape of every helper this class has produced in the
|
|
40
|
+
sibling plugins: one expression, held once, called from many places. The
|
|
41
|
+
expression becomes a **template** whose parameters are wildcards; any
|
|
42
|
+
expression elsewhere in the package that structurally unifies with it is a
|
|
43
|
+
call site that bypassed the helper.
|
|
44
|
+
|
|
45
|
+
Three shapes are excluded, each for a reason rather than to quieten the output
|
|
46
|
+
(mirrored from the marketing reference copy, where each was needed for real):
|
|
47
|
+
|
|
48
|
+
- **FastAPI dependency providers** (a parameter defaulting to ``Depends(...)``).
|
|
49
|
+
A plugin adding an `api_ingress` app (ADR-0021) will typically declare these
|
|
50
|
+
per-router on purpose, so `dependency_overrides` can replace them per app —
|
|
51
|
+
consolidating them would break that.
|
|
52
|
+
- **Pure delegations** — a lone call whose every argument is a bare parameter
|
|
53
|
+
(``return await self._load_owned(owner_sub=owner_sub, ...)``). A façade like
|
|
54
|
+
that holds no expression, so every ordinary call to the underlying function
|
|
55
|
+
would be reported as bypassing it.
|
|
56
|
+
- **Expressions below `_MIN_TEMPLATE_NODES`.** ``return value or {}`` matches
|
|
57
|
+
half of any package and means nothing.
|
|
58
|
+
|
|
59
|
+
## What this plugin's package sweeps to today, and why that is correct
|
|
60
|
+
|
|
61
|
+
At the time this file was added, this skeleton's own `example_plugin` package
|
|
62
|
+
holds **zero** consolidation-helper-shaped functions — `on_install()` and
|
|
63
|
+
`on_uninstall()` (**not invoked** by anything, biffo-template#709) are no-ops
|
|
64
|
+
returning a bare `None`, which is far below `_MIN_TEMPLATE_NODES`. That is the
|
|
65
|
+
right starting state, not a gap: a fresh
|
|
66
|
+
plugin has not yet had the chance to duplicate anything. The guard exists so
|
|
67
|
+
that when the first real helper — and later, the first hand-copy of its
|
|
68
|
+
expression — is written, this file catches it without anyone having to add a
|
|
69
|
+
new test for it. `test_the_swept_package_points_at_real_source` and
|
|
70
|
+
`test_the_guard_can_actually_fail` below exist precisely so a zero here is
|
|
71
|
+
never mistaken for "the collector is broken" (an empty denominator passing
|
|
72
|
+
for the wrong reason is the failure this whole class is about).
|
|
73
|
+
|
|
74
|
+
## Waivers are a ledger, not a mute button
|
|
75
|
+
|
|
76
|
+
A real duplicate that should *not* be consolidated goes in `_ACCEPTED_DUPLICATES`
|
|
77
|
+
with the reason, so it is one review-visible line rather than a silently
|
|
78
|
+
tolerated copy — and `test_no_accepted_duplicate_has_gone_stale` deletes the
|
|
79
|
+
excuse when the code moves on.
|
|
80
|
+
"""
|
|
81
|
+
|
|
82
|
+
from __future__ import annotations
|
|
83
|
+
|
|
84
|
+
import ast
|
|
85
|
+
from pathlib import Path
|
|
86
|
+
|
|
87
|
+
# --------------------------------------------------------------------------
|
|
88
|
+
# Subject: this plugin's own package, located by shape rather than by name —
|
|
89
|
+
# see the module docstring's "Why this is parameterised" section.
|
|
90
|
+
# --------------------------------------------------------------------------
|
|
91
|
+
|
|
92
|
+
_PLUGIN_ROOT = Path(__file__).resolve().parents[1]
|
|
93
|
+
_SRC_ROOT = _PLUGIN_ROOT / "src"
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def _package_dir() -> Path:
|
|
97
|
+
"""The plugin's own package directory, found by shape rather than by name.
|
|
98
|
+
|
|
99
|
+
`src/` holds exactly one package directory in this template and in every
|
|
100
|
+
plugin scaffolded from it (`packages = ["src/<name>"]` in `pyproject.toml`
|
|
101
|
+
is single-valued too), so locating the sole directory with an
|
|
102
|
+
`__init__.py` works everywhere a literal package name would not.
|
|
103
|
+
"""
|
|
104
|
+
candidates = sorted(
|
|
105
|
+
p for p in _SRC_ROOT.iterdir() if p.is_dir() and (p / "__init__.py").is_file()
|
|
106
|
+
)
|
|
107
|
+
assert len(candidates) == 1, (
|
|
108
|
+
f"expected exactly one package under {_SRC_ROOT}, found "
|
|
109
|
+
f"{[c.name for c in candidates]} — the sweep cannot tell which one is "
|
|
110
|
+
"this plugin's own source."
|
|
111
|
+
)
|
|
112
|
+
return candidates[0]
|
|
113
|
+
|
|
114
|
+
|
|
115
|
+
_SRC = _package_dir()
|
|
116
|
+
|
|
117
|
+
#: Below this many AST nodes an expression is too common to mean anything.
|
|
118
|
+
#: Matches the marketing reference copy's threshold: it admits real duplicated
|
|
119
|
+
#: expressions (a ternary, an `HTTPException(...)` call) while still rejecting
|
|
120
|
+
#: `raw or {}` (4 nodes) — see `test_a_near_miss_is_not_reported` below for a
|
|
121
|
+
#: fixture at exactly that shape.
|
|
122
|
+
_MIN_TEMPLATE_NODES = 8
|
|
123
|
+
|
|
124
|
+
#: (helper name, file that re-derives it) -> why that copy is correct.
|
|
125
|
+
#: Empty today: this skeleton ships with no known-correct duplicate. A real
|
|
126
|
+
#: plugin built from it adds entries here as it earns them, same as the
|
|
127
|
+
#: marketing and ideation copies did.
|
|
128
|
+
_ACCEPTED_DUPLICATES: dict[tuple[str, str], str] = {}
|
|
129
|
+
|
|
130
|
+
|
|
131
|
+
# --------------------------------------------------------------------------
|
|
132
|
+
# Finding the helpers
|
|
133
|
+
# --------------------------------------------------------------------------
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
def _parameter_names(fn: ast.FunctionDef | ast.AsyncFunctionDef) -> set[str]:
|
|
137
|
+
"""Every name bound by the signature — the template's wildcards."""
|
|
138
|
+
args = fn.args
|
|
139
|
+
names = {a.arg for a in [*args.posonlyargs, *args.args, *args.kwonlyargs]}
|
|
140
|
+
if args.vararg:
|
|
141
|
+
names.add(args.vararg.arg)
|
|
142
|
+
if args.kwarg:
|
|
143
|
+
names.add(args.kwarg.arg)
|
|
144
|
+
return names
|
|
145
|
+
|
|
146
|
+
|
|
147
|
+
def _sole_returned_expression(fn: ast.FunctionDef | ast.AsyncFunctionDef) -> ast.expr | None:
|
|
148
|
+
"""The expression of a body that is exactly ``return <expr>``, or `None`.
|
|
149
|
+
|
|
150
|
+
A leading docstring is skipped, because helpers in this plugin's style
|
|
151
|
+
tend to have one and requiring a bare body would exclude them.
|
|
152
|
+
"""
|
|
153
|
+
body = list(fn.body)
|
|
154
|
+
if (
|
|
155
|
+
body
|
|
156
|
+
and isinstance(body[0], ast.Expr)
|
|
157
|
+
and isinstance(body[0].value, ast.Constant)
|
|
158
|
+
and isinstance(body[0].value.value, str)
|
|
159
|
+
):
|
|
160
|
+
body = body[1:]
|
|
161
|
+
if len(body) != 1:
|
|
162
|
+
return None
|
|
163
|
+
only = body[0]
|
|
164
|
+
return only.value if isinstance(only, ast.Return) and only.value is not None else None
|
|
165
|
+
|
|
166
|
+
|
|
167
|
+
def _node_count(node: ast.AST) -> int:
|
|
168
|
+
return sum(1 for _ in ast.walk(node))
|
|
169
|
+
|
|
170
|
+
|
|
171
|
+
def _is_dependency_provider(fn: ast.FunctionDef | ast.AsyncFunctionDef) -> bool:
|
|
172
|
+
"""True for a FastAPI dependency (a parameter defaulting to `Depends(...)`).
|
|
173
|
+
|
|
174
|
+
Declared per-router on purpose where they exist: `dependency_overrides` is
|
|
175
|
+
keyed by the function object, so a shared one could not be overridden for
|
|
176
|
+
a single app under test.
|
|
177
|
+
"""
|
|
178
|
+
defaults = [d for d in [*fn.args.defaults, *fn.args.kw_defaults] if d is not None]
|
|
179
|
+
return any(
|
|
180
|
+
isinstance(d, ast.Call)
|
|
181
|
+
and (getattr(d.func, "id", None) == "Depends" or getattr(d.func, "attr", None) == "Depends")
|
|
182
|
+
for d in defaults
|
|
183
|
+
)
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
def _is_pure_delegation(expr: ast.expr, params: set[str]) -> bool:
|
|
187
|
+
"""True for ``return f(a, b=b)`` — a call forwarding parameters unchanged.
|
|
188
|
+
|
|
189
|
+
Such a function is a façade over another function, not a held expression:
|
|
190
|
+
treating it as a template would report every ordinary call to the callee as
|
|
191
|
+
a bypass of the façade, which is the opposite of what this guard means.
|
|
192
|
+
"""
|
|
193
|
+
call = expr.value if isinstance(expr, ast.Await) else expr
|
|
194
|
+
if not isinstance(call, ast.Call):
|
|
195
|
+
return False
|
|
196
|
+
arguments = [*call.args, *[kw.value for kw in call.keywords]]
|
|
197
|
+
return all(isinstance(a, ast.Name) and a.id in params for a in arguments)
|
|
198
|
+
|
|
199
|
+
|
|
200
|
+
#: Node types that make an expression a *composition* rather than a lookup.
|
|
201
|
+
#: `JoinedStr`/`BinOp` are included because URL and key building is the
|
|
202
|
+
#: sibling plugins' most expensive drift shape (marketing#72 was a base URL),
|
|
203
|
+
#: and an f-string helper contains no call at all.
|
|
204
|
+
_COMPOSING_NODES = (
|
|
205
|
+
ast.Call,
|
|
206
|
+
ast.IfExp,
|
|
207
|
+
ast.BoolOp,
|
|
208
|
+
ast.Compare,
|
|
209
|
+
ast.JoinedStr,
|
|
210
|
+
ast.BinOp,
|
|
211
|
+
ast.ListComp,
|
|
212
|
+
ast.DictComp,
|
|
213
|
+
ast.SetComp,
|
|
214
|
+
ast.GeneratorExp,
|
|
215
|
+
)
|
|
216
|
+
|
|
217
|
+
|
|
218
|
+
def _has_structure(expr: ast.expr) -> bool:
|
|
219
|
+
"""True if the expression actually composes something. Pure attribute
|
|
220
|
+
chains, subscripts and literals are shared by too much code to be evidence
|
|
221
|
+
of anything."""
|
|
222
|
+
return any(isinstance(n, _COMPOSING_NODES) for n in ast.walk(expr))
|
|
223
|
+
|
|
224
|
+
|
|
225
|
+
class _Helper:
|
|
226
|
+
"""A consolidation helper and the expression it holds."""
|
|
227
|
+
|
|
228
|
+
def __init__(
|
|
229
|
+
self, module: str, fn: ast.FunctionDef | ast.AsyncFunctionDef, expr: ast.expr
|
|
230
|
+
) -> None:
|
|
231
|
+
self.module = module
|
|
232
|
+
self.name = fn.name
|
|
233
|
+
self.lineno = fn.lineno
|
|
234
|
+
self.template = expr
|
|
235
|
+
self.params = _parameter_names(fn)
|
|
236
|
+
#: Node identities of the helper's own definition — the one place its
|
|
237
|
+
#: expression is allowed to appear. Identity, not line numbers, so
|
|
238
|
+
#: moving the function neither breaks the guard nor widens it.
|
|
239
|
+
self.own_nodes = {id(n) for n in ast.walk(fn)}
|
|
240
|
+
|
|
241
|
+
def __repr__(self) -> str: # pragma: no cover - diagnostics only
|
|
242
|
+
return f"{self.module}:{self.lineno} {self.name}()"
|
|
243
|
+
|
|
244
|
+
|
|
245
|
+
def consolidation_helpers(trees: dict[str, ast.AST]) -> list[_Helper]:
|
|
246
|
+
"""Every single-expression helper in the swept package, discovered from
|
|
247
|
+
the tree.
|
|
248
|
+
|
|
249
|
+
This is the sweep's **denominator**, and the reason the guard does not
|
|
250
|
+
need a hand-maintained list of helper names: a helper added tomorrow lands
|
|
251
|
+
here on its own.
|
|
252
|
+
"""
|
|
253
|
+
found: list[_Helper] = []
|
|
254
|
+
for module, tree in trees.items():
|
|
255
|
+
for node in ast.walk(tree):
|
|
256
|
+
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
|
257
|
+
continue
|
|
258
|
+
expr = _sole_returned_expression(node)
|
|
259
|
+
if expr is None or _is_dependency_provider(node):
|
|
260
|
+
continue
|
|
261
|
+
if _node_count(expr) < _MIN_TEMPLATE_NODES or not _has_structure(expr):
|
|
262
|
+
continue
|
|
263
|
+
if _is_pure_delegation(expr, _parameter_names(node)):
|
|
264
|
+
continue
|
|
265
|
+
found.append(_Helper(module, node, expr))
|
|
266
|
+
return found
|
|
267
|
+
|
|
268
|
+
|
|
269
|
+
# --------------------------------------------------------------------------
|
|
270
|
+
# Matching a call site against a helper's expression
|
|
271
|
+
# --------------------------------------------------------------------------
|
|
272
|
+
|
|
273
|
+
#: Fields that carry no meaning for this comparison. `ctx` is Load/Store/Del,
|
|
274
|
+
#: which differs between an expression that is read and one that is assigned to
|
|
275
|
+
#: without the expressions themselves differing at all.
|
|
276
|
+
_IGNORED_FIELDS = {"ctx"}
|
|
277
|
+
|
|
278
|
+
|
|
279
|
+
def _unifies(
|
|
280
|
+
template: ast.AST, candidate: ast.AST, params: set[str], binding: dict[str, str]
|
|
281
|
+
) -> bool:
|
|
282
|
+
"""True if `candidate` is `template` with its parameters filled in.
|
|
283
|
+
|
|
284
|
+
A parameter is a wildcard that matches any sub-expression, but must match
|
|
285
|
+
the *same* sub-expression everywhere it appears — so
|
|
286
|
+
``json.loads(raw) if isinstance(raw, str) else raw`` matches a call site
|
|
287
|
+
using one variable throughout and not one that mixes two.
|
|
288
|
+
"""
|
|
289
|
+
if isinstance(template, ast.Name) and template.id in params:
|
|
290
|
+
dumped = ast.dump(candidate)
|
|
291
|
+
already = binding.get(template.id)
|
|
292
|
+
if already is None:
|
|
293
|
+
binding[template.id] = dumped
|
|
294
|
+
return True
|
|
295
|
+
return already == dumped
|
|
296
|
+
|
|
297
|
+
if type(template) is not type(candidate):
|
|
298
|
+
return False
|
|
299
|
+
if isinstance(template, ast.Name):
|
|
300
|
+
return template.id == candidate.id # type: ignore[attr-defined]
|
|
301
|
+
if isinstance(template, ast.Constant):
|
|
302
|
+
other = candidate.value # type: ignore[attr-defined]
|
|
303
|
+
return type(template.value) is type(other) and template.value == other
|
|
304
|
+
|
|
305
|
+
candidate_fields = dict(ast.iter_fields(candidate))
|
|
306
|
+
for field, expected in ast.iter_fields(template):
|
|
307
|
+
if field in _IGNORED_FIELDS:
|
|
308
|
+
continue
|
|
309
|
+
actual = candidate_fields.get(field)
|
|
310
|
+
if isinstance(expected, list):
|
|
311
|
+
if not isinstance(actual, list) or len(expected) != len(actual):
|
|
312
|
+
return False
|
|
313
|
+
for want, got in zip(expected, actual, strict=True):
|
|
314
|
+
if isinstance(want, ast.AST):
|
|
315
|
+
if not isinstance(got, ast.AST) or not _unifies(want, got, params, binding):
|
|
316
|
+
return False
|
|
317
|
+
elif want != got:
|
|
318
|
+
return False
|
|
319
|
+
elif isinstance(expected, ast.AST):
|
|
320
|
+
if not isinstance(actual, ast.AST) or not _unifies(expected, actual, params, binding):
|
|
321
|
+
return False
|
|
322
|
+
elif expected != actual:
|
|
323
|
+
return False
|
|
324
|
+
return True
|
|
325
|
+
|
|
326
|
+
|
|
327
|
+
def bypassing_call_sites(trees: dict[str, ast.AST]) -> list[tuple[str, str, int]]:
|
|
328
|
+
"""`(helper name, module, line)` for every expression that re-derives a
|
|
329
|
+
helper's body outside that helper."""
|
|
330
|
+
helpers = consolidation_helpers(trees)
|
|
331
|
+
hits: set[tuple[str, str, int]] = set()
|
|
332
|
+
for helper in helpers:
|
|
333
|
+
for module, tree in trees.items():
|
|
334
|
+
for node in ast.walk(tree):
|
|
335
|
+
if not isinstance(node, ast.expr) or id(node) in helper.own_nodes:
|
|
336
|
+
continue
|
|
337
|
+
if type(node) is not type(helper.template):
|
|
338
|
+
continue
|
|
339
|
+
if _unifies(helper.template, node, helper.params, {}):
|
|
340
|
+
hits.add((helper.name, module, node.lineno))
|
|
341
|
+
return sorted(hits)
|
|
342
|
+
|
|
343
|
+
|
|
344
|
+
def _parse_package() -> dict[str, ast.AST]:
|
|
345
|
+
return {p.name: ast.parse(p.read_text(), filename=str(p)) for p in sorted(_SRC.glob("*.py"))}
|
|
346
|
+
|
|
347
|
+
|
|
348
|
+
# --------------------------------------------------------------------------
|
|
349
|
+
# The guard
|
|
350
|
+
# --------------------------------------------------------------------------
|
|
351
|
+
|
|
352
|
+
|
|
353
|
+
def test_the_swept_package_points_at_real_source() -> None:
|
|
354
|
+
"""A collector that silently resolves to nothing passes for the wrong
|
|
355
|
+
reason. #1587 exists because the previous state of this skeleton had no
|
|
356
|
+
sweep at all; this pins that `_package_dir()` actually finds this plugin's
|
|
357
|
+
real, non-empty source tree rather than an empty or missing directory."""
|
|
358
|
+
assert _SRC.is_dir(), f"{_SRC} does not exist — the sweep is watching nothing"
|
|
359
|
+
py_files = list(_SRC.glob("*.py"))
|
|
360
|
+
assert py_files, f"{_SRC} has no source files — the sweep is watching nothing"
|
|
361
|
+
|
|
362
|
+
|
|
363
|
+
def test_no_module_re_derives_a_shared_helpers_expression() -> None:
|
|
364
|
+
offenders = [
|
|
365
|
+
f"{module}:{line} re-derives {helper}()"
|
|
366
|
+
for helper, module, line in bypassing_call_sites(_parse_package())
|
|
367
|
+
if (helper, module) not in _ACCEPTED_DUPLICATES
|
|
368
|
+
]
|
|
369
|
+
assert not offenders, (
|
|
370
|
+
"These write out an expression a helper in this plugin already holds, "
|
|
371
|
+
"instead of calling it (biffo-template#1587). A hand-written copy "
|
|
372
|
+
"stops tracking the helper the moment the helper is corrected, and "
|
|
373
|
+
"review cannot see the difference — an unconverted call site looks "
|
|
374
|
+
"exactly like code that was never meant to use the helper. Call the "
|
|
375
|
+
"helper, or record why this copy is correct in "
|
|
376
|
+
"`_ACCEPTED_DUPLICATES`:\n " + "\n ".join(offenders)
|
|
377
|
+
)
|
|
378
|
+
|
|
379
|
+
|
|
380
|
+
def test_the_guard_can_actually_fail() -> None:
|
|
381
|
+
"""The detector must detect, on source that is never imported.
|
|
382
|
+
|
|
383
|
+
This plugin's real package holds zero consolidation-helper-shaped
|
|
384
|
+
functions today (see the module docstring), so proving the guard against
|
|
385
|
+
it would only prove it stays green over an empty denominator. This plants
|
|
386
|
+
a fresh helper and a site that bypasses it instead — the same synthetic
|
|
387
|
+
check the marketing reference copy uses, for the same reason.
|
|
388
|
+
"""
|
|
389
|
+
tree = ast.parse(
|
|
390
|
+
"def _normalise(raw):\n"
|
|
391
|
+
' """Hold this once."""\n'
|
|
392
|
+
" return json.loads(raw) if isinstance(raw, str) else (raw or {})\n"
|
|
393
|
+
"\n"
|
|
394
|
+
"def converted(row):\n"
|
|
395
|
+
" return _normalise(row.get('body'))\n"
|
|
396
|
+
"\n"
|
|
397
|
+
"def bypassing(row):\n"
|
|
398
|
+
" raw = row.get('body')\n"
|
|
399
|
+
" return json.loads(raw) if isinstance(raw, str) else (raw or {})\n"
|
|
400
|
+
)
|
|
401
|
+
assert bypassing_call_sites({"planted.py": tree}) == [("_normalise", "planted.py", 10)]
|
|
402
|
+
|
|
403
|
+
|
|
404
|
+
def test_a_near_miss_is_not_reported() -> None:
|
|
405
|
+
"""Different code must not be called drift.
|
|
406
|
+
|
|
407
|
+
A guard that fires on anything adjacent gets waived wholesale, so the
|
|
408
|
+
near-misses are pinned: a different test, a different call, and the same
|
|
409
|
+
shape over two different variables are all legitimate code.
|
|
410
|
+
|
|
411
|
+
(Each near-miss is written with a second statement so it is a call site
|
|
412
|
+
rather than a helper of its own. A one-line function IS a template, and a
|
|
413
|
+
template whose parameters are wildcards legitimately matches another
|
|
414
|
+
helper's body — that is a duplicated *helper*, which this guard reports on
|
|
415
|
+
purpose.)
|
|
416
|
+
"""
|
|
417
|
+
tree = ast.parse(
|
|
418
|
+
"def _normalise(raw):\n"
|
|
419
|
+
" return json.loads(raw) if isinstance(raw, str) else (raw or {})\n"
|
|
420
|
+
"\n"
|
|
421
|
+
"def different_test(row):\n"
|
|
422
|
+
" raw = row.get('body')\n"
|
|
423
|
+
" return json.loads(raw) if raw.startswith('{') else (raw or {})\n"
|
|
424
|
+
"\n"
|
|
425
|
+
"def different_call(row):\n"
|
|
426
|
+
" raw = row.get('body')\n"
|
|
427
|
+
" return int(raw) if isinstance(raw, str) else (raw or {})\n"
|
|
428
|
+
"\n"
|
|
429
|
+
"def two_variables(row, other):\n"
|
|
430
|
+
" raw = row.get('body')\n"
|
|
431
|
+
" return json.loads(raw) if isinstance(other, str) else (other or {})\n"
|
|
432
|
+
)
|
|
433
|
+
assert bypassing_call_sites({"planted.py": tree}) == []
|
|
434
|
+
|
|
435
|
+
|
|
436
|
+
def test_a_dependency_provider_is_not_treated_as_a_helper() -> None:
|
|
437
|
+
"""A router-style `Depends(...)` default must not be swept as a helper.
|
|
438
|
+
|
|
439
|
+
If this exclusion were dropped, a plugin declaring the same FastAPI
|
|
440
|
+
dependency provider in several routers on purpose (so
|
|
441
|
+
`dependency_overrides` can replace it per app) would report every
|
|
442
|
+
duplicate as an offender, and the only correct resolution would be a
|
|
443
|
+
waiver — a guard that starts life with a wall of waivers is one nobody
|
|
444
|
+
reads.
|
|
445
|
+
"""
|
|
446
|
+
tree = ast.parse(
|
|
447
|
+
"def get_campaign_client(admin = Depends(require_admin)):\n"
|
|
448
|
+
" return principal_client.PrincipalCoreClient(admin.token, verify=True)\n"
|
|
449
|
+
"\n"
|
|
450
|
+
"def other_router_provider(admin = Depends(require_admin)):\n"
|
|
451
|
+
" return principal_client.PrincipalCoreClient(admin.token, verify=True)\n"
|
|
452
|
+
)
|
|
453
|
+
assert consolidation_helpers({"planted.py": tree}) == []
|
|
454
|
+
assert bypassing_call_sites({"planted.py": tree}) == []
|
|
455
|
+
|
|
456
|
+
|
|
457
|
+
def test_a_pure_delegation_is_not_treated_as_a_helper() -> None:
|
|
458
|
+
"""`return await self._load_owned(owner_sub=..., session_id=...)` holds no
|
|
459
|
+
expression — every ordinary call to `_load_owned` would otherwise be
|
|
460
|
+
reported as bypassing the façade in front of it."""
|
|
461
|
+
tree = ast.parse(
|
|
462
|
+
"class S:\n"
|
|
463
|
+
" async def get_session(self, *, owner_sub, session_id):\n"
|
|
464
|
+
" return await self._load_owned(owner_sub=owner_sub, session_id=session_id)\n"
|
|
465
|
+
"\n"
|
|
466
|
+
" async def chat_turn(self, *, owner_sub, session_id):\n"
|
|
467
|
+
" session = await self._load_owned(owner_sub=owner_sub, session_id=session_id)\n"
|
|
468
|
+
" return session\n"
|
|
469
|
+
)
|
|
470
|
+
assert consolidation_helpers({"planted.py": tree}) == []
|
|
471
|
+
|
|
472
|
+
|
|
473
|
+
def test_no_accepted_duplicate_has_gone_stale() -> None:
|
|
474
|
+
"""A waiver outlives the code it excuses unless something deletes it.
|
|
475
|
+
|
|
476
|
+
Each entry must still name a duplicate the sweep really finds. When the
|
|
477
|
+
copy is consolidated or the file is renamed, this fails and the excuse goes
|
|
478
|
+
with it, rather than sitting there granting a permission nobody re-examined.
|
|
479
|
+
"""
|
|
480
|
+
live = {(helper, module) for helper, module, _ in bypassing_call_sites(_parse_package())}
|
|
481
|
+
stale = sorted(pair for pair in _ACCEPTED_DUPLICATES if pair not in live)
|
|
482
|
+
assert not stale, (
|
|
483
|
+
"These waivers no longer excuse anything — the duplicate is gone or has "
|
|
484
|
+
f"moved. Delete them: {stale}"
|
|
485
|
+
)
|
package/package.json
CHANGED
package/scripts/pg-test-db.sh
CHANGED
|
@@ -43,6 +43,7 @@
|
|
|
43
43
|
# eval "$(sh scripts/pg-test-db.sh --export)" # export BIFFO_TEST_PG_DSN and TABSII_TEST_PG_DSN
|
|
44
44
|
# sh scripts/pg-test-db.sh # print the DSN on stdout
|
|
45
45
|
# sh scripts/pg-test-db.sh --recreate # force a rebuild
|
|
46
|
+
# sh scripts/pg-test-db.sh --reap # housekeeping only: reap and exit
|
|
46
47
|
#
|
|
47
48
|
# Only the DSN reaches stdout, so it is safe to capture; progress goes to stderr.
|
|
48
49
|
#
|
|
@@ -164,10 +165,12 @@ CONTAINER="${BIFFO_PG_CONTAINER:-biffo-pg-test-$_checkout_suffix}"
|
|
|
164
165
|
|
|
165
166
|
RECREATE=0
|
|
166
167
|
EXPORT=0
|
|
168
|
+
REAP_ONLY=0
|
|
167
169
|
for arg in "$@"; do
|
|
168
170
|
case "$arg" in
|
|
169
171
|
--recreate) RECREATE=1 ;;
|
|
170
172
|
--export) EXPORT=1 ;;
|
|
173
|
+
--reap) REAP_ONLY=1 ;;
|
|
171
174
|
-h | --help)
|
|
172
175
|
sed -n '2,72p' "$0" | sed 's/^#\{1,2\} \{0,1\}//'
|
|
173
176
|
exit 0
|
|
@@ -250,6 +253,7 @@ if [ "$BIFFO_PG_REAP_HOURS" -gt 0 ] 2>/dev/null && command -v docker >/dev/null
|
|
|
250
253
|
say "cannot compute a reap cutoff on this date(1); skipping container reaping"
|
|
251
254
|
else
|
|
252
255
|
_reaped=0
|
|
256
|
+
_reaped_gone=0
|
|
253
257
|
_considered=0
|
|
254
258
|
# TWO filters, not one literal name (#1383). The label is what containers
|
|
255
259
|
# created from here now carry; the name prefix keeps covering every one
|
|
@@ -272,14 +276,65 @@ if [ "$BIFFO_PG_REAP_HOURS" -gt 0 ] 2>/dev/null && command -v docker >/dev/null
|
|
|
272
276
|
_made=$(docker inspect -f '{{.Created}}' "$_c" 2>/dev/null | cut -c1-19)
|
|
273
277
|
[ -z "$_made" ] && continue
|
|
274
278
|
_considered=$((_considered + 1))
|
|
279
|
+
# ── Ownership beats age, where ownership is knowable ─────────────────
|
|
280
|
+
#
|
|
281
|
+
# Age was only ever a PROXY. The container is keyed to a checkout
|
|
282
|
+
# (see `CONTAINER` above), so the honest question is not "is this old?"
|
|
283
|
+
# but "does the checkout that owns it still exist?" -- and once the
|
|
284
|
+
# worktree is deleted the answer is a fact, not an estimate. A container
|
|
285
|
+
# whose checkout is gone can never be reused by anything, so there is no
|
|
286
|
+
# 4-second-rebuild trade to weigh: it is pure garbage the moment the
|
|
287
|
+
# directory disappears.
|
|
288
|
+
#
|
|
289
|
+
# Measured 2026-08-22 on one workstation: 42 live containers, of which 19
|
|
290
|
+
# belonged to checkouts that no longer existed. Under the age rule alone
|
|
291
|
+
# those 19 each held a running Postgres and ~500MB for up to 24 more
|
|
292
|
+
# hours -- and #703's real complaint was never disk, it was that these
|
|
293
|
+
# compete for the same page cache and I/O as the lane being timed.
|
|
294
|
+
#
|
|
295
|
+
# The path is read from a LABEL SET AT CREATION, never derived from the
|
|
296
|
+
# container's name. Deriving it would mean hashing candidate paths to see
|
|
297
|
+
# which produces this suffix, and the `biffo-pg-test-` prefix is shared by
|
|
298
|
+
# every repo in the estate -- so a run in one repo, finding no matching
|
|
299
|
+
# worktree of its OWN, would confidently reap a container another repo's
|
|
300
|
+
# test lane was mid-run against. The label makes the claim self-describing
|
|
301
|
+
# and repo-independent.
|
|
302
|
+
#
|
|
303
|
+
# Containers created before this label existed report an empty value and
|
|
304
|
+
# fall through to the age rule below, exactly as `biffo.ephemeral=1`
|
|
305
|
+
# migrated in (#1383). Nothing is stranded; they simply age out once.
|
|
306
|
+
_owner=$(docker inspect -f '{{index .Config.Labels "biffo.checkout"}}' "$_c" 2>/dev/null)
|
|
307
|
+
if [ -n "$_owner" ] && [ "$_owner" != "<no value>" ] && [ ! -d "$_owner" ]; then
|
|
308
|
+
if docker rm -f -v "$_c" >/dev/null 2>&1; then
|
|
309
|
+
_reaped=$((_reaped + 1))
|
|
310
|
+
_reaped_gone=$((_reaped_gone + 1))
|
|
311
|
+
fi
|
|
312
|
+
continue
|
|
313
|
+
fi
|
|
275
314
|
# Both are UTC ISO-8601 to the second, so a string compare IS a time
|
|
276
315
|
# compare -- no epoch conversion, and portable across date(1) flavours.
|
|
277
316
|
if awk -v a="$_made" -v b="$_reap_cutoff" 'BEGIN { exit !(a < b) }'; then
|
|
278
|
-
|
|
317
|
+
# `-v` REMOVES THE CONTAINER'S ANONYMOUS VOLUME WITH IT.
|
|
318
|
+
#
|
|
319
|
+
# Without it every reap orphans a full Postgres data directory. Measured on one
|
|
320
|
+
# workstation 2026-08-20: 413 dangling volumes holding 104.8GB -- 95% of all local
|
|
321
|
+
# volume space -- against 11 live containers totalling 2.5MB. The containers were
|
|
322
|
+
# tidied and their data was not, so the leak grew by roughly a database per reap
|
|
323
|
+
# and nothing pointed at it.
|
|
324
|
+
#
|
|
325
|
+
# It is invisible by construction: `docker ps` looks clean, the reaper reports how
|
|
326
|
+
# many it removed, and the space is only findable with `docker volume ls -qf
|
|
327
|
+
# dangling=true`. The first symptom is a full disk somewhere unrelated.
|
|
328
|
+
docker rm -f -v "$_c" >/dev/null 2>&1 && _reaped=$((_reaped + 1))
|
|
279
329
|
fi
|
|
280
330
|
done
|
|
281
|
-
|
|
282
|
-
|
|
331
|
+
# Two reasons, counted apart. A single total would let the cheap, certain
|
|
332
|
+
# rule and the age guess read as one number, and the whole point of the
|
|
333
|
+
# ownership rule is that it is NOT a guess -- if it ever reaps something
|
|
334
|
+
# still wanted, that total must say so on its own.
|
|
335
|
+
if [ "$_reaped" -gt 0 ]; then
|
|
336
|
+
say "reaped $_reaped of $_considered container(s): $_reaped_gone whose checkout no longer exists, $((_reaped - _reaped_gone)) unused for over ${BIFFO_PG_REAP_HOURS}h (set BIFFO_PG_REAP_HOURS=0 to disable)"
|
|
337
|
+
fi
|
|
283
338
|
|
|
284
339
|
# ── What the reaper can SEE but must not touch ──────────────────────────
|
|
285
340
|
#
|
|
@@ -330,10 +385,31 @@ if [ "$BIFFO_PG_REAP_HOURS" -gt 0 ] 2>/dev/null && command -v docker >/dev/null
|
|
|
330
385
|
say "NOT reaped -- Postgres containers over ${BIFFO_PG_REAP_HOURS}h old that this script did not create:"
|
|
331
386
|
for _u in $_unclaimed; do say " $_u"; done
|
|
332
387
|
say " A stale one costs test failures that belong to nobody (#1383). Remove by hand"
|
|
333
|
-
say " (docker rm -f <name>
|
|
388
|
+
say " (docker rm -f -v <name> -- the -v matters, or its data volume is orphaned),"
|
|
389
|
+
say " or start it with --label biffo.ephemeral=1 to have it reaped."
|
|
334
390
|
fi
|
|
335
391
|
}
|
|
336
392
|
fi
|
|
393
|
+
|
|
394
|
+
# `--reap` is housekeeping ONLY: reap, report, and stop before starting or
|
|
395
|
+
# touching a server.
|
|
396
|
+
#
|
|
397
|
+
# The reaper is otherwise LAZY -- it runs only when something else runs the
|
|
398
|
+
# lane, so the moment the fleet goes quiet nothing reclaims anything and the
|
|
399
|
+
# mess sits until the next test. That is the opposite of what is wanted: idle
|
|
400
|
+
# is exactly when reclaiming is free. This flag is the callable form, so a
|
|
401
|
+
# worktree teardown or a periodic sweep can collect without standing up a
|
|
402
|
+
# Postgres nobody asked for.
|
|
403
|
+
if [ "$REAP_ONLY" -eq 1 ]; then
|
|
404
|
+
exit 0
|
|
405
|
+
fi
|
|
406
|
+
fi
|
|
407
|
+
|
|
408
|
+
# Guard the case above: with reaping disabled there is nothing for `--reap` to
|
|
409
|
+
# do, and it must still not fall through into starting a server.
|
|
410
|
+
if [ "$REAP_ONLY" -eq 1 ]; then
|
|
411
|
+
say "reaping is disabled (BIFFO_PG_REAP_HOURS=0); nothing to do"
|
|
412
|
+
exit 0
|
|
337
413
|
fi
|
|
338
414
|
|
|
339
415
|
if ! psql_admin -c 'SELECT 1' >/dev/null 2>&1; then
|
|
@@ -352,7 +428,11 @@ if ! psql_admin -c 'SELECT 1' >/dev/null 2>&1; then
|
|
|
352
428
|
# is still scanned, but it only ever described containers this script named;
|
|
353
429
|
# anything started under another name was outside the reaper's denominator
|
|
354
430
|
# entirely. A label travels with the container whatever it is called.
|
|
431
|
+
# `biffo.checkout` is what makes the container's owner knowable after the
|
|
432
|
+
# fact. The reaper above uses it to remove a container the moment its
|
|
433
|
+
# checkout is deleted, rather than waiting out a 24-hour proxy.
|
|
355
434
|
docker run -d --name "$CONTAINER" --label biffo.ephemeral=1 \
|
|
435
|
+
--label "biffo.checkout=$REPO_ROOT" \
|
|
356
436
|
-e POSTGRES_PASSWORD="$PASS" -p "$PORT:5432" "$IMAGE" >/dev/null
|
|
357
437
|
fi
|
|
358
438
|
# Polled, not slept: a cold image pull and a warm restart differ by an order of
|