@hrg/gitignore 0.3.1 → 0.3.3

package/README.md

@@ -260,23 +260,23 @@ The three JS columns are **the same vectors, the same port, three matching strat
   which is a pattern with `**` where the regex happened to be fast. That row is the open tuning question, not a settled
   result.
 
-The Java port still runs the full DP (4397 ms total, `backtrack` 3205 ms, `anchored` and `doublestar` regressed against its
-regex version). The same hybrid applies to it and is the next step; the numbers above are why it should be measured per
-workload rather than by the total.
-
+Both ports now run the same segment DP as the reference, with the ancestor memoisation that took them from 17x and 11x behind
+Zig to about 1.6x and 2.2x at the time it was measured - and the reference has since become roughly 10x faster than it started,
+so the relative gap moved again. The current numbers, the method, and every improvement (including what was measured and
+refuted) are in [doc/performance/](doc/performance/README.md) and [doc/measurement.md](doc/measurement.md).
 ### How these numbers are produced
 
 Cold-process runs, with start-up measured separately and subtracted. That is the honest number for a CLI and it is
-also noisy: the same Java build has reported `backtrack` at 3205 ms and 5129 ms. [+doc/measurement.md](doc/measurement.md)](doc/measurement.md) records the two measurement modes, the steady-state table (12 rounds inside one process) and every
+also noisy: the same Java build has reported `backtrack` at 3205 ms and 5129 ms. [doc/measurement.md](doc/measurement.md) records the two measurement modes, the steady-state table (12 rounds inside one process) and every
 strategy tried on the Java port with what it cost. The Java column above predates the literal-segment fast path.
 
 ### Two caveats
 
 - The runner measures a process; the start-up column is a subtraction, and on a loaded machine it is a noisy one. The
   large rows are far outside that noise; the small ones are not.
-- The ports are young. Their remaining distance to Zig is a design difference, not tuning: Java has not adopted the DP, and
-  neither port has the Zig implementation's per-chain indexing.
-
+- The ports are no longer "young" in the sense that mattered: the Java port adopted the segment DP, then literal segments, then
+  the ancestor memoisation, and the JS port did the same. What remains is the reference's own head start - it has three indexes
+  the ports do not have, and [doc/performance/shared.md](doc/performance/shared.md) is the list a new port should work through.
 ## Running the three
 
 ```text
@@ -379,3 +379,14 @@ Requirements, learned from a project of this author's that publishes this way (`
 
 Afterwards, `fs-meta` resolves `hr.hrg:gitignore` from the repository like any other artifact; it no longer needs the local
 `mvn install`.
+## Checks
+
+```text
+bun scripts/check-verdicts.mjs         # the three implementations agree, byte for byte
+bun scripts/check-verdicts.mjs --json  # one NDJSON object per workload
+```
+
+The 34 conformance vectors say what the matcher must decide; this says that the three implementations decide the *same thing*
+on the six bench workloads — 10,000 verdicts, compared byte for byte, with the first differing line named on failure. A missing
+binary is exit 2 with the build command, never a skip. It exists because twice in this project's history "the vectors pass" was
+mistaken for "the ports agree", and the comparison had been run by hand in scratch files that were never committed.

package/js/gitignore.mjs

@@ -120,6 +120,9 @@ function translateCharacterClass(glob, open) {
   return { regex: `${translated}]`, close };
 }
 
+/** A pattern segment with none of these bytes is a plain name, so a directory-only rule can be indexed by it. */
+const hasWildcardByte = /[*?[\\]/;
+
 /** One parsed pattern. */
 function parseRule(rawLine) {
   let line = stripTrailingUnescapedSpaces(rawLine);
@@ -148,6 +151,11 @@ function parseRule(rawLine) {
   return {
     negated,
     directoryOnly,
+    anchored,
+    // A plain name (no wildcard, no escape) is indexed rather than scanned: the directory-only ones are what a
+    // per-path prefix walk asks about most, and a name is the whole question for the unanchored form.
+    literal: hasWildcardByte.test(line) ? null : line,
+    indexed: false,
     // Both anchors are required: Java's `Matcher.matches()` anchors the whole pattern, while JavaScript's
     // `RegExp.test` searches — and a missing `^` lets an anchored pattern match a path *suffix*. That one
     // difference is what the ignore vector `anchored-pattern` caught between the two ports.
@@ -197,6 +205,42 @@ export function parseGitignore(relativeDirectory, text) {
       rules.push(rule);
     }
   }
+  // The directory-only literal rules are indexed rather than scanned, because they are the class a per-path
+  // decision asks about most and the one whose scan costs a prefix walk per rule. Two forms, because the
+  // question differs: an anchored literal is a whole path prefix (`module0/src/main/java/`), so it is keyed
+  // by that path and looked up once per depth the path has; an unanchored one is a single name (`build/`),
+  // so it is keyed by the name and looked up once per segment. A literal can only match a prefix of exactly
+  // its own depth, which is what the per-depth flag preserves: `java/fory-0/...` is rejected against a
+  // `java/fory-1/...` rule by the lookup itself, the case a first-segment prune cannot see. The flags are a
+  // plain array here rather than the reference's `u256`, and a pattern deeper than the array stays on the
+  // scanned path, where the answer is still the correct one.
+  const anchoredLiterals = new Map();
+  const unanchoredLiterals = new Map();
+  const anchoredDepths = [];
+  let deepestAnchoredDepth = 0;
+  for (let index = 0; index < rules.length; index++) {
+    const rule = rules[index];
+    if (!rule.directoryOnly || rule.literal === null) {
+      continue;
+    }
+    if (rule.anchored) {
+      let depth = 1;
+      for (let at = rule.literal.indexOf('/'); at >= 0; at = rule.literal.indexOf('/', at + 1)) {
+        depth += 1;
+      }
+      if (depth > 256) {
+        continue;
+      }
+      anchoredLiterals.set(rule.literal, index);
+      anchoredDepths[depth] = true;
+      if (depth > deepestAnchoredDepth) {
+        deepestAnchoredDepth = depth;
+      }
+    } else {
+      unanchoredLiterals.set(rule.literal, index);
+    }
+    rule.indexed = true;
+  }
   return {
     relativeDirectory: directory,
     ruleCount: rules.length,
@@ -216,11 +260,58 @@ export function parseGitignore(relativeDirectory, text) {
         }
         scoped = relativePath.slice(prefix.length);
       }
+
+      // Ask the index first. `best` is the highest rule position any indexed rule can reach for this path, so
+      // only rules after it can override the answer — usually none. That is also why a hit needs no split:
+      // a path the index decides is answered before the matcher is reached.
+      let best = -1;
+      if (unanchoredLiterals.size > 0) {
+        // An unanchored directory rule names one segment, and for a file the last segment is the file itself.
+        let start = 0;
+        for (;;) {
+          const slash = scoped.indexOf('/', start);
+          const last = slash < 0;
+          if (!last || isDirectory) {
+            const name = last ? scoped.slice(start) : scoped.slice(start, slash);
+            const at = unanchoredLiterals.get(name);
+            if (at !== undefined && at > best) {
+              best = at;
+            }
+          }
+          if (last) break;
+          start = slash + 1;
+        }
+      }
+      if (deepestAnchoredDepth > 0) {
+        // One lookup per depth that actually holds a rule, not one per rule: a path shorter than a rule's depth
+        // costs nothing at all.
+        let depth = 0;
+        let start = 0;
+        for (;;) {
+          const slash = scoped.indexOf('/', start);
+          depth += 1;
+          const last = slash < 0;
+          if (last && !isDirectory) break;
+          if (anchoredDepths[depth] === true) {
+            const prefix = last ? scoped : scoped.slice(0, slash);
+            const at = anchoredLiterals.get(prefix);
+            if (at !== undefined && at > best) {
+              best = at;
+            }
+          }
+          if (last) break;
+          start = slash + 1;
+        }
+      }
+
       // Split once for every `**` rule of this file, and only if one of them is reached: a file whose patterns all
       // compile to a regex pays no split at all (the `*.log`-shaped rule sets of the small workloads).
       let segments = null;
-      for (let index = rules.length - 1; index >= 0; index--) {
+      for (let index = rules.length - 1; index > best; index--) {
         const rule = rules[index];
+        if (rule.indexed) {
+          continue;
+        }
         if (rule.segments !== null && segments === null) {
           segments = scoped.split('/');
         }
@@ -228,6 +319,9 @@ export function parseGitignore(relativeDirectory, text) {
           return !rule.negated;
         }
       }
+      if (best >= 0) {
+        return !rules[best].negated;
+      }
       return undefined;
     },
   };
@@ -293,19 +387,29 @@ export function readGitignore(directoryPath, relativeDirectory) {
  */
 export class GitIgnoreRules {
   #stack = [];
-  // A directory's answer, keyed by its path relative to the walk root. A path list visits the same directories over
+  // A directory's state, keyed by its path relative to the walk root. A path list visits the same directories over
   // and over (`backtrack` asks about 84,000 ancestor directories that are 42 distinct ones), and the rules are a
-  // snapshot, so the answer cannot change while the stack is unchanged. `push`/`pop` clear it: they are the only
-  // things that change which files govern a directory, and a cached "excluded" must never outlive them.
+  // snapshot, so the state cannot change while the stack is unchanged. `push`/`pop` clear it: they are the only
+  // things that change which files govern a directory, and a cached answer must never outlive them.
+  //
+  // The state is the two facts a decision needs about the path's directory: whether it (or an ancestor) is excluded,
+  // and — implicit here — the chain of files that governs it. The chain half is already free in this port because the
+  // walk hands the stack to the decider instead of making it walk up the file system, so the cache pays for the
+  // ancestor half: one map lookup and one `dirname` per decision instead of a lookup and a substring per level.
+  #states = new Map();
+  // The historical walk, kept for a path outside the contract (a leading or trailing separator), where the index key
+  // would be a different one than the walk would ask about. Same shape as the memo it replaced.
   #excluded = new Map();
 
   push(file) {
     this.#stack.push(file);
+    this.#states.clear();
     this.#excluded.clear();
   }
 
   pop() {
     this.#stack.pop();
+    this.#states.clear();
     this.#excluded.clear();
   }
 
@@ -316,12 +420,45 @@ export class GitIgnoreRules {
   decide(relativePath, isDirectory) {
     const direct = this.decideDirect(relativePath, isDirectory);
     if (direct === true) {
+      // A directly excluded path needs no ancestor fact at all — the common case on a rule set with a literal hit.
+      return true;
+    }
+    const slash = relativePath.lastIndexOf('/');
+    // The index keys on the directory, and `dirOf` is the substring before the last separator for every path in the
+    // contract (`/`-separated, no leading or trailing slash). A path outside it keeps the historical answer rather
+    // than being answered from an index built on a different key.
+    if (slash === 0 || slash === relativePath.length - 1) {
+      return this.#walkAncestors(relativePath, direct);
+    }
+    if (this.#stateFor(slash < 0 ? '' : relativePath.slice(0, slash)).ancestorExcluded) {
       return true;
     }
-    // An excluded directory is never entered, so a rule for a file inside it cannot re-include that file: ask the
-    // ancestors, deepest first. A parent that a later rule re-includes answers false and is skipped. The ancestor is
-    // found by walking the separators backwards, so no ancestor costs a split and a join — one substring, which is
-    // also the memo's key.
+    return direct;
+  }
+
+  /**
+   * The ancestor-excluded fact for a directory, inherited: a directory is excluded when its parent is, or when its
+   * own decision through the stack excludes it — exactly the prefix set the per-path walk asked, computed once per
+   * directory instead of once per path per level.
+   */
+  #stateFor(directory) {
+    let state = this.#states.get(directory);
+    if (state !== undefined) {
+      return state;
+    }
+    let ancestorExcluded = false;
+    if (directory !== '') {
+      const slash = directory.lastIndexOf('/');
+      const parent = this.#stateFor(slash < 0 ? '' : directory.slice(0, slash));
+      ancestorExcluded = parent.ancestorExcluded || this.decideDirect(directory, true) === true;
+    }
+    state = { ancestorExcluded };
+    this.#states.set(directory, state);
+    return state;
+  }
+
+  /** The pre-index ancestor walk, used only for a path the index does not key on. */
+  #walkAncestors(relativePath, direct) {
     for (let slash = relativePath.lastIndexOf('/'); slash >= 0; slash = relativePath.lastIndexOf('/', slash - 1)) {
       const ancestor = relativePath.slice(0, slash);
       let excluded = this.#excluded.get(ancestor);

package/js/glob.mjs

@@ -50,6 +50,23 @@ function rows(size) {
  * @param {number} [count] how many segments to consider (default: all of them)
  */
 export function segmentsMatch(segments, text, count = text.length) {
+  // The pattern's last segment, when it is not `**`, has to match the last segment the call considers — otherwise the
+  // whole DP is spent on a match that cannot exist. `count` is the first `count` segments, so the last considered text
+  // segment is `text[count - 1]` (the path's own last segment only when the whole path is being matched); a
+  // directory-only ancestor question passes its own last considered segment this way instead of the path's.
+  if (count > 0) {
+    const tail = segments[segments.length - 1];
+    if (tail.doubleStar !== true) {
+      const textTail = text[count - 1];
+      if (tail.literal !== undefined) {
+        if (textTail !== tail.literal) {
+          return false;
+        }
+      } else if (!tail.regex.test(textTail)) {
+        return false;
+      }
+    }
+  }
   const [first, second] = rows(count + 1);
   let reach = first;
   let next = second;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hrg/gitignore",
-  "version": "0.3.1",
+  "version": "0.3.3",
   "description": "git's .gitignore (wildmatch) semantics: one matcher, shared by the Zig, JavaScript and Java ports.",
   "license": "LicenseRef-GPL-3.0-with-Commons-Clause",
   "author": "Davor Hrg",