workproof 0.3.1 → 0.4.1

package/README.md

@@ -175,6 +175,48 @@ so it never merges them for you: pass each address to `--author`, or write a `.m
 The addresses themselves stay out of the document unless you pass `--emails`, the same rule the
 rest of the report follows.
 
+## Names that are not ASCII
+
+The identity fold decides which commits are yours, so a gap in it is a missing figure rather than
+an ugly one. Two gaps were closed against Unicode 17.0, and both changed counts:
+
+- `Weiß` and `WEISS` are one person. JavaScript's `toLowerCase` is Unicode's simple case fold and
+  leaves ß alone, so the two spellings never met and half of one person's commits fell outside
+  every figure. The full fold maps ß to ss
+  ([CaseFolding.txt](https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt), status `F`),
+  which is what the tool does now.
+- A name typed in fullwidth Latin letters, which is what a Japanese or Korean keyboard produces
+  without switching modes, is the same name typed in ASCII. Matching normalises to NFKC, so the
+  two forms meet, along with the fi ligature and the compatibility ideographs.
+
+Turkish dotted and dotless i and decomposed accents were already folded. The fold is for matching
+only. The report always prints the name git holds.
+
+Display is separate. An Arabic or Hebrew name is the first strong character of the plain-language
+paragraph, and without an isolate it turns the whole paragraph right to left: the English after it
+reverses and the figures land in the wrong order. Names and repository names are wrapped in U+2068
+and U+2069 in the Markdown report, the isolates [UAX #9](https://www.unicode.org/reports/tr9/)
+defines for this. They are invisible, they are added only where a strong right-to-left character
+appears, and they are never written into the JSON, so the hash over a report is unchanged and a
+report written before this still verifies.
+
+Three things here are deliberate:
+
+- Numbers are printed in the `en-US` format wherever a person reads them, so `12,345` is twelve
+  thousand for every reader. The hash is taken over the RFC 8785 canonical JSON, where numbers
+  are not formatted at all, so this cannot affect `workproof verify`. It is about the document:
+  two reports are meant to be read side by side, and a thousands separator that follows the
+  reader's machine makes that harder for no gain.
+- Weeks are ISO weeks, starting on Monday. That is not the local week for most of the world: of
+  the twenty most populous countries, CLDR gives Monday to seven, Sunday to eleven and Saturday
+  to two, which `Intl.Locale.prototype.getWeekInfo` reports in Node 24. Cadence counts weeks a
+  person was active, and that figure is only worth reading next to another report, so it uses one
+  definition everywhere rather than the reader's. The days inside a week are already the author's
+  own local dates, which is what decides whether a Sunday commit belongs to this week or the next.
+- A `.mailmap` name is matched by git, not by this tool. git compares those names ignoring case
+  for ASCII letters only, verified on git 2.50.1: an entry written `josÉ Álvarez` matches a commit
+  by `JOSÉ ÁLVAREZ`, and one written `josé álvarez` does not. Map by address wherever you can.
+
 ## In plain language
 
 Each repository section of the Markdown report opens with a paragraph a non-engineer can
@@ -229,6 +271,16 @@ Three commands, in increasing order of what the reader needs to have.
   compares HEAD, then recomputes every figure and prints what differs. A report from another
   repository stops at the fingerprint. If HEAD moved since the report, it says so and shows
   which figures changed.
+The hash is reproducible **across operating systems, git versions, locales and directory
+names**, for the same repository at the same commit with the same command and the same
+`--fingerprint-key`. It is not reproducible across a rewritten history, a different sample
+or a different seed, and it is not meant to be: those change the measurement. Two fields
+in the report are deliberately outside the hash, because a verifier on another machine
+cannot reproduce them: the local directory name, and the git version. They stay in the
+document, where they explain why two runs might legitimately differ. Before 0.4.0 both
+were inside it, so the same repository in a folder with a different name hashed
+differently; a test now pins all of this, and CI checks it on Linux, macOS and Windows.
+
 - `workproof attest report.json` writes `report.intoto.json`, an
   [in-toto](https://in-toto.io) v1 Statement whose subject is the report hash and whose
   predicate carries the tool version, parameters, HEAD, keyed fingerprint, git version and
@@ -483,6 +535,28 @@ with an attestation if the reader cannot reach the repository.
 It measures survivorship and activity, not quality, review, design or mentoring. It does
 not rank people. It does not replace references. It is not a legal document.
 
+## What the research says about numbers like these
+
+Repository-derived figures have a literature, and most of it is a warning. Kalliamvakou,
+Gousios and Blincoe catalogued the traps in mining a forge, including that most repositories
+are personal and inactive and that a large share of merged pull requests are not recorded as
+merged ([MSR 2014](https://doi.org/10.1145/2597073.2597074), extended in [EMSE
+2015](https://doi.org/10.1007/s10664-015-9393-5)). Bird and colleagues related ownership
+concentration to defects rather than to credit, which is a different question from this one
+and is not evidence that a high share means good work. On survival, Spinellis, Louridas and
+Kechagia put the median line lifespan at about 2.4 years across 89 repositories ([PeerJ CS
+7:e372, 2021](https://doi.org/10.7717/peerj-cs.372)), and Gurov found over half of all lines
+are never deleted in 120 TypeScript repositories ([arXiv
+2606.04993](https://arxiv.org/abs/2606.04993)).
+
+None of that work measures a person's share of the surviving code, which is what the
+surviving-lines figure here reports. A first measurement of how far that share diverges from
+commit share, over twelve repositories, is in
+[surviving-lines/research](https://github.com/Bubblegunn/surviving-lines/tree/main/research):
+the same person tops both in 25 of 36 runs, and among substantial contributors the two
+differ by a median of 7.5 percentage points. That is the honest size of the effect this
+report exists to expose, measured rather than asserted.
+
 ## Where it comes from
 
 The method is written up in

package/README.tr.md

@@ -115,6 +115,49 @@ sayılmadan paydadan çıktı; bu, aynı kişinin eski sürümde %24,1 görünen
 adlandırırdı. Bu boşluk, iki yönde de, bir raporun birinin işi hakkında söyleyebileceği en
 dürüst şeydir.
 
+## ASCII olmayan isimler
+
+Kimlik katlaması hangi commit'lerin size ait olduğuna karar verir; oradaki bir boşluk çirkin bir
+görüntü değil, eksik bir figür demektir. Unicode 17.0'a bakılarak iki boşluk kapatıldı ve ikisi de
+sayıları değiştirdi:
+
+- `Weiß` ile `WEISS` tek kişidir. JavaScript'in `toLowerCase`'i Unicode'un basit harf katlamasıdır
+  ve ß'ye dokunmaz; iki yazım hiç buluşmuyordu ve bir kişinin commit'lerinin yarısı bütün
+  figürlerin dışında kalıyordu. Tam katlama ß'yi ss'ye eşler
+  ([CaseFolding.txt](https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt), durum `F`)
+  ve araç artık bunu yapıyor.
+- Tam genişlikli Latin harfleriyle yazılan bir isim, ki Japonca ya da Korece bir klavyede mod
+  değiştirmeden yazılan budur, ASCII ile yazılan aynı isimdir. Eşleştirme NFKC'ye normalleştirir;
+  fi bağlaması ve uyumluluk ideografları da bununla birlikte katlanır.
+
+Türkçe noktalı ve noktasız i ile ayrışık aksanlar zaten katlanıyordu. Katlama yalnız eşleştirme
+içindir; rapor her zaman git'in tuttuğu ismi yazar.
+
+Görüntü ayrı bir konu. Arapça ya da İbranice bir isim, sade dil paragrafının ilk güçlü
+karakteridir ve yalıtıcı olmadan bütün paragrafı sağdan sola çevirir: ardındaki İngilizce ters
+döner, sayılar yanlış sırada görünür. Markdown raporunda isimler ve depo adları U+2068 ile U+2069
+arasına alınıyor; [UAX #9](https://www.unicode.org/reports/tr9/) bu yalıtıcıları tam olarak bunun
+için tanımlar. Görünmezler, yalnız güçlü sağdan sola karakter geçtiğinde eklenirler ve JSON'a hiç
+yazılmazlar; yani rapor özeti değişmez ve bu değişiklikten önce yazılmış bir rapor hâlâ doğrulanır.
+
+Üç şey bilerek böyle:
+
+- Sayılar, insanın okuduğu her yerde `en-US` biçiminde yazılır; `12,345` her okuyucu için on iki
+  bindir. Özet, sayıların hiç biçimlendirilmediği RFC 8785 kanonik JSON üzerinden alınır, yani bu
+  seçim `workproof verify`'ı etkileyemez. Mesele belgedir: iki rapor yan yana okunmak için yazılır
+  ve okuyucunun makinesine göre değişen bir binlik ayıracı bunu boşuna zorlaştırır.
+- Haftalar ISO haftasıdır, pazartesi başlar. Dünyanın çoğu için yerel hafta bu değildir: en
+  kalabalık yirmi ülkeden CLDR'ye göre yedisinde pazartesi, on birinde pazar, ikisinde cumartesi
+  başlar; Node 24'te bunu `Intl.Locale.prototype.getWeekInfo` söyler. Cadence, bir kişinin etkin
+  olduğu haftaları sayar ve bu figür ancak başka bir raporun yanında anlamlıdır, o yüzden
+  okuyucunun tanımını değil her yerde aynı tanımı kullanır. Haftanın içindeki günler zaten yazarın
+  kendi yerel tarihleridir; bir pazar commit'inin bu haftaya mı yoksa sonrakine mi düştüğüne o
+  karar verir.
+- `.mailmap` isimlerini bu araç değil git eşleştirir. git bu isimleri yalnız ASCII harfleri için
+  büyük küçük harf farkını yok sayarak karşılaştırır; git 2.50.1 üzerinde doğrulandı:
+  `josÉ Álvarez` yazılmış bir satır `JOSÉ ÁLVAREZ` imzalı bir commit ile eşleşiyor,
+  `josé álvarez` yazılmış olan eşleşmiyor. Mümkün olan her yerde adresle eşleyin.
+
 ## Ne ölçer
 
 Her figür git'ten gelir, başka hiçbir şeyden değil. Bot commit'leri ile üretilmiş, vendored,

package/dist/src/figures/identity.d.ts

@@ -13,6 +13,19 @@ import type { Identity } from "./types.js";
  * turn English "I" into "ı", so the four i forms collapse to one instead. The cost is that two
  * names differing only in dotted and dotless i match each other; the benefit is that a Turkish
  * name matches itself in any case, and the error message prints the folded form that was tried.
+ *
+ * The German sharp s. `"WEISS".toLowerCase()` is "weiss" and `"Weiß".toLowerCase()` is "weiß",
+ * because JavaScript's `toLowerCase` is Unicode's simple case fold. The full fold maps ß to ss
+ * (CaseFolding.txt, status F, https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt), so
+ * ß and capital ẞ are mapped before it runs. Without this, one person who signs both ways was two
+ * people, and half their commits were left out of every figure.
+ *
+ * Compatibility spellings. A name typed in fullwidth Latin letters, which is what a Japanese or
+ * Korean keyboard produces without switching modes, is the same name typed in ASCII; NFKC is what
+ * says so. It folds the fi ligature and the compatibility ideographs with it.
+ *
+ * This fold decides which commits are counted, so a gap in it is a wrong figure rather than a
+ * cosmetic problem. It is never a display form: the report prints the name git holds.
  */
 export declare function foldIdentity(s: string): string;
 /**

package/dist/src/figures/identity.js

@@ -12,14 +12,28 @@ import { configuredEmail, configuredName } from "../git.js";
  * turn English "I" into "ı", so the four i forms collapse to one instead. The cost is that two
  * names differing only in dotted and dotless i match each other; the benefit is that a Turkish
  * name matches itself in any case, and the error message prints the folded form that was tried.
+ *
+ * The German sharp s. `"WEISS".toLowerCase()` is "weiss" and `"Weiß".toLowerCase()` is "weiß",
+ * because JavaScript's `toLowerCase` is Unicode's simple case fold. The full fold maps ß to ss
+ * (CaseFolding.txt, status F, https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt), so
+ * ß and capital ẞ are mapped before it runs. Without this, one person who signs both ways was two
+ * people, and half their commits were left out of every figure.
+ *
+ * Compatibility spellings. A name typed in fullwidth Latin letters, which is what a Japanese or
+ * Korean keyboard produces without switching modes, is the same name typed in ASCII; NFKC is what
+ * says so. It folds the fi ligature and the compatibility ideographs with it.
+ *
+ * This fold decides which commits are counted, so a gap in it is a wrong figure rather than a
+ * cosmetic problem. It is never a display form: the report prints the name git holds.
  */
 export function foldIdentity(s) {
     return s
-        .normalize("NFC")
+        .normalize("NFKC")
         .replace(/[\u0130\u0131Ii]/g, "i")
         .replace(/\u0307/g, "")
+        .replace(/[\u00df\u1e9e]/g, "ss")
         .toLowerCase()
-        .normalize("NFC")
+        .normalize("NFKC")
         .trim();
 }
 /**

package/dist/src/report.d.ts

@@ -10,7 +10,6 @@ export interface Report {
     /** sha256 of the RFC 8785 canonical JSON of { params, repositories }. */
     hash: string;
 }
-/** The hash a report with these parameters and repositories must carry. */
 export declare const hashOf: (params: unknown, repositories: unknown) => string;
 export declare function buildReport(repositories: RepoReport[], params: Params, meta: {
     version: string;

package/dist/src/report.js

@@ -2,8 +2,36 @@ import { createHash } from "node:crypto";
 import { canonicalize } from "./canonical.js";
 import { publicEmail } from "./analyse.js";
 import { plainSummary } from "./summary.js";
+import { isolate } from "./text.js";
 /** The hash a report with these parameters and repositories must carry. */
-export const hashOf = (params, repositories) => createHash("sha256").update(canonicalize({ params, repositories })).digest("hex");
+/**
+ * What of a repository the hash covers: everything a stranger with the same repository at
+ * the same commit can reproduce, and nothing else.
+ *
+ * Two fields are recorded in the report and deliberately left out. `name` is the local
+ * directory's basename, so the same repository cloned to a different folder hashed
+ * differently. `environment.git` is the local git version string, so two people with the
+ * same repository and different git hashed differently. Both were reproduced with real
+ * runs before 0.4.0. They stay in the document because they explain why two runs might
+ * legitimately differ; they are not something a verifier can be asked to reproduce.
+ *
+ * The blame flags, the ignore-revs file and the seed stay in, because those are the
+ * question the caller asked, not the machine they asked it on.
+ */
+function forHash(repositories) {
+    if (!Array.isArray(repositories))
+        return repositories;
+    return repositories.map((repo) => {
+        if (repo === null || typeof repo !== "object")
+            return repo;
+        const { name: _name, environment, ...rest } = repo;
+        if (environment === null || typeof environment !== "object")
+            return rest;
+        const { git: _git, ...restEnvironment } = environment;
+        return { ...rest, environment: restEnvironment };
+    });
+}
+export const hashOf = (params, repositories) => createHash("sha256").update(canonicalize({ params, repositories: forHash(repositories) })).digest("hex");
 /**
  * What of the parameters the report may carry: never the fingerprint key, never a GitHub
  * noreply login, and without --emails no author given as an email address.
@@ -72,9 +100,9 @@ export function renderMarkdown(report, narrative) {
         ``,
     ];
     for (const repo of report.repositories) {
-        out.push(`## ${repo.name}`, ``, `HEAD \`${repo.head.slice(0, 12)}\` · fingerprint \`${repo.fingerprint.slice(0, 16)}\` · identities: ${repo.identity.names.join(", ")}${repo.identity.emails.length ? ` (${repo.identity.emails.join(", ")})` : ""}`, ``, ...(repo.identity.possiblySplit
+        out.push(`## ${isolate(repo.name)}`, ``, `HEAD \`${repo.head.slice(0, 12)}\` · fingerprint \`${repo.fingerprint.slice(0, 16)}\` · identities: ${repo.identity.names.map(isolate).join(", ")}${repo.identity.emails.length ? ` (${repo.identity.emails.join(", ")})` : ""}`, ``, ...(repo.identity.possiblySplit
             ? [
-                `**${plural(repo.identity.possiblySplit.names.length, "other identity", "other identities")} in this repository look${repo.identity.possiblySplit.names.length === 1 ? "s" : ""} like the same person and ${repo.identity.possiblySplit.names.length === 1 ? "was" : "were"} not counted here**: ${repo.identity.possiblySplit.names.join(", ")}${repo.identity.possiblySplit.emails?.length ? ` (${repo.identity.possiblySplit.emails.join(", ")})` : ""}. ${repo.identity.possiblySplit.reasons.map((r, i) => (i === 0 ? r.charAt(0).toUpperCase() + r.slice(1) : r)).join("; ")}. Every figure below therefore describes only the work under the identities named above. If they are the same person, re-run with each address passed to \`--author\`, or merge them in a \`.mailmap\`.`,
+                `**${plural(repo.identity.possiblySplit.names.length, "other identity", "other identities")} in this repository look${repo.identity.possiblySplit.names.length === 1 ? "s" : ""} like the same person and ${repo.identity.possiblySplit.names.length === 1 ? "was" : "were"} not counted here**: ${repo.identity.possiblySplit.names.map(isolate).join(", ")}${repo.identity.possiblySplit.emails?.length ? ` (${repo.identity.possiblySplit.emails.join(", ")})` : ""}. ${repo.identity.possiblySplit.reasons.map((r, i) => (i === 0 ? r.charAt(0).toUpperCase() + r.slice(1) : r)).join("; ")}. Every figure below therefore describes only the work under the identities named above. If they are the same person, re-run with each address passed to \`--author\`, or merge them in a \`.mailmap\`.`,
                 ``,
             ]
             : []), repo.excluded.enabled

package/dist/src/summary.js

@@ -1,3 +1,4 @@
+import { isolate } from "./text.js";
 const pct = (x) => `${Math.round(x * 100)}%`;
 const n = (x) => x.toLocaleString("en-US");
 const value = (repo, id) => repo.figures.find((f) => f.id === id)?.value;
@@ -10,7 +11,9 @@ function longDate(iso) {
     return `${d} ${MONTHS[m - 1]} ${y}`;
 }
 export function plainSummary(repo) {
-    const who = repo.identity.names[0] ?? "This author";
+    // Isolated so a right-to-left name at the head of the paragraph cannot reverse the sentence
+    // around it. Presentation only: the data, and the hash over it, keep the name git holds.
+    const who = isolate(repo.identity.names[0] ?? "This author");
     const tenure = value(repo, "tenure");
     const commits = value(repo, "commitShare");
     const surviving = value(repo, "survivingLines");
@@ -19,7 +22,7 @@ export function plainSummary(repo) {
     const absence = value(repo, "absenceFactor");
     const sentences = [];
     if (tenure) {
-        sentences.push(`${who} worked in ${repo.name} from ${longDate(tenure.first)} to ${longDate(tenure.last)}, a span of ${n(tenure.days)} days.`);
+        sentences.push(`${who} worked in ${isolate(repo.name)} from ${longDate(tenure.first)} to ${longDate(tenure.last)}, a span of ${n(tenure.days)} days.`);
     }
     if (commits && surviving) {
         sentences.push(`They made ${n(commits.author)} of the ${n(commits.total)} changes recorded in that period (${pct(commits.share)}), and ${n(surviving.lines)} of the ${n(surviving.linesAttributed)} lines of code still in the project today are theirs (${pct(surviving.share)}).`);

package/dist/src/text.d.ts

@@ -0,0 +1,20 @@
+/**
+ * Text that has to survive being embedded in someone else's sentence.
+ */
+/**
+ * Wrap a name, a repository name or any other borrowed string in the first strong isolate, so it
+ * cannot reorder the sentence it is dropped into.
+ *
+ * The plain-language paragraph puts the author's name at the start of a sentence and the
+ * repository's name in the middle of it. An Arabic or Hebrew name there is the first strong
+ * character of the paragraph, which under the bidirectional algorithm makes the whole paragraph
+ * right to left: the English words after it reverse, and the numbers land in the wrong order.
+ * U+2068 and U+2069 are what UAX #9 provides for embedding text of unknown direction
+ * (https://www.unicode.org/reports/tr9/, section 2.7). They are default-ignorable, so they take
+ * no width, and they are added only to strings that carry strong right-to-left characters, so
+ * every other report is byte for byte what it was.
+ *
+ * This is presentation only. The report's data, and therefore its hash, keeps the exact string
+ * git holds: `workproof verify` on a report written before this change still passes.
+ */
+export declare function isolate(s: string): string;

package/dist/src/text.js

@@ -0,0 +1,26 @@
+/**
+ * Text that has to survive being embedded in someone else's sentence.
+ */
+/** Scripts written right to left, whose first strong character sets a paragraph's direction. */
+const RTL = /[\p{sc=Arabic}\p{sc=Hebrew}\p{sc=Syriac}\p{sc=Thaana}\p{sc=Nko}\p{sc=Samaritan}\p{sc=Mandaic}\p{sc=Adlam}]/u;
+/**
+ * Wrap a name, a repository name or any other borrowed string in the first strong isolate, so it
+ * cannot reorder the sentence it is dropped into.
+ *
+ * The plain-language paragraph puts the author's name at the start of a sentence and the
+ * repository's name in the middle of it. An Arabic or Hebrew name there is the first strong
+ * character of the paragraph, which under the bidirectional algorithm makes the whole paragraph
+ * right to left: the English words after it reverse, and the numbers land in the wrong order.
+ * U+2068 and U+2069 are what UAX #9 provides for embedding text of unknown direction
+ * (https://www.unicode.org/reports/tr9/, section 2.7). They are default-ignorable, so they take
+ * no width, and they are added only to strings that carry strong right-to-left characters, so
+ * every other report is byte for byte what it was.
+ *
+ * This is presentation only. The report's data, and therefore its hash, keeps the exact string
+ * git holds: `workproof verify` on a report written before this change still passes.
+ */
+export function isolate(s) {
+    if (!RTL.test(s) || /\p{Bidi_Control}/u.test(s))
+        return s;
+    return `⁨${s}⁩`;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "workproof",
-  "version": "0.3.1",
+  "version": "0.4.1",
   "description": "Turn a private git repository into a verifiable engineering report for one author, without showing any code: thirteen figures from git, a hash anyone can recompute offline, verify, and an in-toto attestation.",
   "type": "module",
   "main": "./dist/src/index.js",