ceedling 1.2.0 → 1.2.2

data/lib/ceedling/defaults.rb

@@ -7,7 +7,7 @@
 
 require 'ceedling/constants'
 require 'ceedling/system_wrapper'
-require 'ceedling/file_path_utils'
+require 'ceedling/filesystem/file_path_utils'
 
 # Assign a default value for system testing where CEEDLING_APPCFG may not be present
 # TODO: Create code config & test structure that does not internalize a test path like this
@@ -58,14 +58,14 @@ DEFAULT_TEST_LINKER_TOOL = {
   }
 
 DEFAULT_TEST_FIXTURE_TOOL = {
-  :executable => '${1}'.freeze, # Unity test runner executable
+  :executable => '"${1}"'.freeze, # Unity test runner executable, quoted as a path
   :name => 'default_test_fixture'.freeze,
   :optional => false.freeze,
   :arguments => [].freeze
   }
 
 DEFAULT_TEST_FIXTURE_SIMPLE_BACKTRACE_TOOL = {
-  :executable => '${1}'.freeze, # Unity test runner executable
+  :executable => '"${1}"'.freeze, # Unity test runner executable, quoted as a path
   :name => 'default_test_fixture_simple_backtrace'.freeze,
   :optional => false.freeze,
   :arguments => [
@@ -204,7 +204,7 @@ DEFAULT_TEST_BACKTRACE_GDB_TOOL = {
     '--eval-command run'.freeze,
     "--command \"${1}\"".freeze, # Debug script file to run
     '--args'.freeze,
-    '${2}'.freeze,               # Test executable
+    '"${2}"'.freeze,             # Test executable, quoted as a path
     # Test case matching flag and (quoted) value, fully composed by the caller as
     # either `-n "<exact name>"` or `-f "<base name>"` -- see
     # GeneratorTestResultsBacktrace#unity_filter_arg.

data/lib/ceedling/dependencies/README.md

@@ -2,29 +2,29 @@
 
 This subsystem answers one question for any expensive step in either the test or release pipeline: has anything this target depends on changed since it last built successfully? It answers using content hashes rather than timestamps, so neither pipeline ever duplicates work that doesn't need to be done.
 
-It is not itself a pipeline with sequential stages the way `test_invoker`/`release_invoker` are. It's a service both pipelines call into, over and over, as a pure oracle. It never decides *when* to run a build step or *what order* steps happen in -- that's each pipeline's own stage list -- it only answers, at the moment a step is about to run, whether that build step actually needs to run.
+It is not itself a pipeline with sequential stages the way `test_invoker`/`release_invoker` are. It’s a service both pipelines call into, over and over, as a pure oracle. It never decides *when* to run a build step or *what order* steps happen in -- that’s each pipeline’s own stage list -- it only answers, at the moment a step is about to run, whether that build step actually needs to run.
 
-## The subsystem's shape
+## The subsystem’s shape
 
 Two classes matter most, and everything else exists to serve them:
 
-- **`Dependinator`** is the thin, Ceedling-specific facade every pipeline actually talks to. Every method it has except one is a one-line passthrough to `DependencyTracker` below; what it actually owns is the convention for where a given pipeline's cache file lives on disk, keyed by a symbolic identifier (`:test`, `:release`, or a plugin's own symbol), and the vocabulary translation at `flush` time from the tracker's own `prune:` parameter to the pipeline-facing `refresh_dependencies:` (see below). Its whole reason for existing is so pipeline code speaks in build verbs -- register this, ask if that's stale, mark this fresh -- and never has to know anything about cache file paths or tracker internals.
-- **`DependencyTracker`** is the actual engine, and it's deliberately generic and application-agnostic -- nothing in it knows what a "test" or a "release" is. It holds this run's registrations in memory and a loaded cache of the previous run's hashes, and its whole job is answering staleness queries against the two.
+- **`Dependinator`** is the thin, Ceedling-specific facade every pipeline actually talks to. Every method it has except one is a one-line passthrough to `DependencyTracker` below; what it actually owns is the convention for where a given pipeline’s cache file lives on disk, keyed by a symbolic identifier (`:test`, `:release`, or a plugin’s own symbol), and the vocabulary translation at `flush` time from the tracker’s own `prune:` parameter to the pipeline-facing `refresh_dependencies:` (see below). Its whole reason for existing is so pipeline code speaks in build verbs -- register this, ask if that’s stale, mark this fresh -- and never has to know anything about cache file paths or tracker internals.
+- **`DependencyTracker`** is the actual engine, and it’s deliberately generic and application-agnostic -- nothing in it knows what a "test" or a "release" is. It holds this run’s registrations in memory and a loaded cache of the previous run’s hashes, and its whole job is answering staleness queries against the two.
 
 `DependencyTracker`, in turn, composes six narrowly-scoped collaborators, each doing exactly one job and nothing else. Four serve every run:
 
 - `DependencyHasher` decides what makes two things count as "the same."
 - `DependencyPathNormalizer` decides what makes two spellings of a path count as "the same target."
 - `DependencyCacheStore` owns the on-disk cache file itself.
-- `GccDependencyParser` turns a compiler's own dependency output into something registerable.
+- `GccDependencyParser` turns a compiler’s own dependency output into something registerable.
 
 The other two, `DependencyDebugTree` and `DependencyDiffer`, serve only the troubleshooting aid described last. The sections below cover each in turn.
 
 ## The `register` → `stale?` → `mark_fresh` idiom
 
-Every call site in both pipelines, regardless of what expensive thing it's guarding -- a preprocessor subprocess, a mock generation, a real compiler or linker invocation -- follows the same three-part shape. `register` tells the tracker what a target depends on: its antecedent files, and a `meta` value carrying anything else that should count toward staleness even though it isn't a file at all -- compile flags, `-D` defines, search paths, a toolchain version string, a coverage-on-or-off flag. An object file's target is the `.o` itself; its files are its source plus whatever headers it actually pulled in; its meta is the flags and defines that, if they change, should force a recompile even though the source file itself didn't change a byte.
+Every call site in both pipelines, regardless of what expensive thing it’s guarding -- a preprocessor subprocess, a mock generation, a real compiler or linker invocation -- follows the same three-part shape. `register` tells the tracker what a target depends on: its antecedent files, and a `meta` value carrying anything else that should count toward staleness even though it isn't a file at all -- compile flags, `-D` defines, search paths, a toolchain version string, a coverage-on-or-off flag. An object file’s target is the `.o` itself; its files are its source plus whatever headers it actually pulled in; its meta is the flags and defines that, if they change, should force a recompile even though the source file itself didn't change a byte.
 
-`register` is additive on purpose. Calling it more than once for the same target unions the file lists and merges the meta rather than replacing what came before, so a target's dependencies can be built up incrementally across more than one call site without either caller needing full knowledge of what the other already supplied. This is exactly what lets a step register its source file and known flags in one call, then separately register the dependencies its tool discovered, described next.
+`register` is additive on purpose. Calling it more than once for the same target unions the file lists and merges the meta rather than replacing what came before, so a target’s dependencies can be built up incrementally across more than one call site without either caller needing full knowledge of what the other already supplied. This is exactly what lets a step register its source file and known flags in one call, then separately register the dependencies its tool discovered, described next.
 
 ### Dependencies discovered by the tool
 
@@ -32,38 +32,40 @@ A step can't know every file it depends on until its tool has run, since the hea
 
 Two sources supply these files:
 
-- **A compile's `.d` file.** `-MMD -MF` writes gcc's own dependency output, in Makefile dialect, and `register_gcc_deps_file` (or `register_gcc_deps_string`, for content already in hand) parses it with `GccDependencyParser` and `register`s each target it describes with that target's dependency list. A single `.d` file can describe more than one target at once; it's a small manifest, not just one file's dependency list.
-- **A preprocessing run's line markers.** Two stages of the test pipeline preprocess each test file. Stage 4 runs it with `-fdirectives-only`, following its `#include`s, including headers a mock stand-in includes. Stage 11 fully expands it, reading every header it reaches. Each output's line markers name every file it entered. The pipeline registers the project files among them (`Preprocessinator#preprocessed_dependencies`), so editing a header the test reaches reruns the step. Such a header's macro can decide which mocks the test needs (stage 4) or which test cases it has (stage 11). System headers are left out. So are generated files below the build root, such as stand-ins and the mocks that replace them, which derive from project files and would only ever look changed.
+- **A compile’s `.d` file.** `-MMD -MF` writes gcc’s own dependency output, in Makefile dialect, and `register_gcc_deps_file` (or `register_gcc_deps_string`, for content already in hand) parses it with `GccDependencyParser` and `register`s each target it describes with that target’s dependency list. A single `.d` file can describe more than one target at once; it’s a small manifest, not just one file’s dependency list.
+- **A preprocessing run’s line markers.** Two stages of the test pipeline preprocess each test file. Stage 4 runs it with `-fdirectives-only`, following its `#include`s, including headers a mock stand-in includes. Stage 11 fully expands it, reading every header it reaches. Each output’s line markers name every file it entered. The pipeline registers the project files among them (`Preprocessinator#preprocessed_dependencies`), so editing a header the test reaches reruns the step. Such a header’s macro can decide which mocks the test needs (stage 4) or which test cases it has (stage 11). System headers are left out. So are generated files below the build root, such as stand-ins and the mocks that replace them, which derive from project files and would only ever look changed.
 
 ### Fresh, stale, and invalidated
 
-Once a target's inputs are registered, `stale?` is a pure question with no side effects, and `mark_fresh` is the write that happens after a real (re)build succeeds, recording new hashes for the target itself, its meta, and every currently-existing dependency. A dependency that's gone missing is simply left out of what gets recorded -- `stale?` already treats a missing dependency as stale on its own, so there's nothing to gain by recording anything else about it. `invalidate` is the explicit override: it drops a target's cache entry outright, so its next `stale?` answers true whatever the hashes say, while leaving this run's registrations alone.
+Once a target’s inputs are registered, `stale?` is a pure question with no side effects, and `mark_fresh` is the write that happens after a real (re)build succeeds, recording new hashes for the target itself, its meta, and every currently-existing dependency. A dependency that’s gone missing is simply left out of what gets recorded -- `stale?` already treats a missing dependency as stale on its own, so there’s nothing to gain by recording anything else about it. `stale?` weighs only the dependencies registered this run, though. One dropped from the list leaves every remaining dependency as it was, so the drop goes unseen. A target whose dependency list can shrink while the rest stay unchanged -- a link, when a source or library leaves the build -- also registers the list itself as meta, so a shorter list is a meta change. A link’s dependencies include the library files `LibraryResolver` finds for its `:libraries` entries, and those the Dependencies plugin publishes in `LIBRARIES_LINK_FILES`. `invalidate` is the explicit override: it drops a target’s cache entry outright, so its next `stale?` answers true whatever the hashes say, while leaving this run’s registrations alone.
 
-Some steps also produce a file beside their target that a later run trusts, such as a test's cached results or a test file's cached build directives. These are a target's `outputs`, registered after the step produces them and before `mark_fresh`, which records their hashes in the target's cache entry. A fresh target vouches for those outputs: `stale?` reads them from the cache entry, not this run's registrations, and treats the target as stale when any one is missing or altered. Damage to a trusted artifact then costs one rerun of the step that wrote it. Throughout, only a regular file counts as present. A directory named as a dependency, as a truncated `.d` file can do, reads as missing rather than being hashed.
+Some steps also produce a file beside their target that a later run trusts, such as a test’s cached results or a test file’s cached build directives. These are a target’s `outputs`, registered after the step produces them and before `mark_fresh`, which records their hashes in the target’s cache entry. A fresh target vouches for those outputs: `stale?` reads them from the cache entry, not this run’s registrations, and treats the target as stale when any one is missing or altered. Damage to a trusted artifact then costs one rerun of the step that wrote it. Throughout, only a regular file counts as present. A directory named as a dependency, as a truncated `.d` file can do, reads as missing rather than being hashed.
 
 ## What makes two things "the same"
 
 Two decisions make this whole scheme trustworthy rather than merely convenient, and both trade a little efficiency for a lot of certainty.
 
-`DependencyHasher` hashes a file's exact on-disk bytes, whitespace included, with no format-aware normalization at all. Whitespace isn't reliably cosmetic: it's syntactically meaningful in YAML, and it can even change whether a C macro's line continuation is recognized. The two ways a hash-based staleness check can be wrong aren't equally bad, either. Reporting something stale when it isn't just costs a wasted rebuild. Reporting it fresh when it isn't means shipping a build that silently doesn't reflect its own source -- exactly the failure this whole subsystem exists to prevent. The same hasher also canonicalizes a `meta` value before hashing it. Hash-key order and Symbol-vs-String keys never matter as a result, while array element order is still preserved, since the order of a flags or defines list can itself be meaningful.
+`DependencyHasher` hashes a file’s exact on-disk bytes, whitespace included, with no format-aware normalization at all. A file unchanged since before the build began is hashed once a build, keyed by its size and modification time, since many targets share one dependency. Anything modified as the build began or since is hashed every time. Whitespace isn't reliably cosmetic: it’s syntactically meaningful in YAML, and it can even change whether a C macro’s line continuation is recognized. The two ways a hash-based staleness check can be wrong aren't equally bad, either. Reporting something stale when it isn't just costs a wasted rebuild. Reporting it fresh when it isn't means shipping a build that silently doesn't reflect its own source -- exactly the failure this whole subsystem exists to prevent. The same hasher also canonicalizes a `meta` value before hashing it. Hash-key order and Symbol-vs-String keys never matter as a result, while array element order is still preserved, since the order of a flags or defines list can itself be meaningful.
 
 `DependencyPathNormalizer` solves a narrower but equally important problem. It resolves a path to its real, on-disk casing before that path is ever used as a cache key. A `#include "Foo.h"` written one way and a file that actually exists on disk as `foo.h` can then never be mistaken for two different targets -- nor can the same real file assembled from project configuration two different ways.
 
 ## The on-disk cache
 
-Each pipeline identifier gets exactly one JSON cache file of its own, never shared with any other identifier. Test and release builds each get their own, and a plugin introducing a third pipeline would get a third, just by choosing its own identifier. This isn't incidental. It's what makes the pruning behavior described below safe at all -- a shared cache would mean a full test run's prune could silently evict every cached release entry along with it.
+Each pipeline identifier gets exactly one JSON cache file of its own, never shared with any other identifier. Test and release builds each get their own, and a plugin introducing a third pipeline would get a third, just by choosing its own identifier. This isn't incidental. It’s what makes the pruning behavior described below safe at all -- a shared cache would mean a full test run’s prune could silently evict every cached release entry along with it.
 
-Loading that cache is deliberately forgiving in one direction and strict in another. Any structural surprise in the file as a whole degrades the entire store to empty rather than raising. A missing cache, malformed cache, or incompatible version simply triggers a full rebuild instead of a hard failure. A single malformed *entry* within an otherwise-good file, similarly, only drops that one entry (i.e. requires a rebuild step). On every load, entries whose target has since been deleted from disk are pruned, automatically and unconditionally. This pruning is always safe, even during a partial run that only touches a handful of targets, because it only ever removes an entry once its target is confirmed gone from disk -- never merely unregistered this run. That's exactly what distinguishes it from the opt-in pruning described next. Persisting the cache back out is done via a write-to-temp-file-then-rename, so a reader can never observe a half-written cache.
+Loading that cache is deliberately forgiving in one direction and strict in another. Any structural surprise in the file as a whole degrades the entire store to empty rather than raising. A missing cache, malformed cache, or incompatible version simply triggers a full rebuild instead of a hard failure. A single malformed *entry* within an otherwise-good file, similarly, only drops that one entry (i.e. requires a rebuild step). On every load, entries whose target has since been deleted from disk are pruned, automatically and unconditionally. This pruning is always safe, even during a partial run that only touches a handful of targets, because it only ever removes an entry once its target is confirmed gone from disk -- never merely unregistered this run. That’s exactly what distinguishes it from the opt-in pruning described next. Persisting the cache back out is done via a write-to-temp-file-then-rename, so a reader can never observe a half-written cache.
+
+The cache’s schema version also marks the build layout. Its keys are build paths, so a release that moves where a build puts its output bumps the version even when the file’s shape is unchanged. Version 2 marks 1.2.2’s layout, which places each test’s output by the test’s path from the project root, and each object, preprocessed file, and dependencies file by its source’s path from the project root. A cache from an earlier version is set aside like any other incompatible one, so the first build after upgrading rebuilds everything once, and no target can trust an old directory that a different test’s new name happens to reuse. Its warning says so and advises `ceedling clobber`. The old store’s entries go with it, so nothing in the cache refers to the old layout. Only the old files stay on disk, where gcov’s coverage reports would still read them until `clobber` removes them.
 
 ## `refresh_dependencies`: the special feature
 
-A normal `flush` just persists whatever this run registered and learned. Passing `refresh_dependencies: true` additionally drops every cache entry whose target was *not* registered at some point during this run, before persisting. In effect, this treats this run's registrations as the complete, authoritative set of everything that should still have a cache entry at all. That's exactly correct for a run that's guaranteed to have touched every target that matters, such as a full `test:all` or a full release build. It's exactly wrong for anything less than that. A single test file, `test:pattern`, `test:path`, or the ad hoc single-file `release:compile:`/`release:assemble:` tasks would each, if this were set true, silently wipe out cache entries for every target the partial run simply didn't happen to touch -- forcing an unnecessary full rebuild on the next real run. Both pipelines are careful to set it only when they can actually make that guarantee.
+A normal `flush` just persists whatever this run registered and learned. Passing `refresh_dependencies: true` additionally drops every cache entry whose target was *not* registered at some point during this run, before persisting. In effect, this treats this run’s registrations as the complete, authoritative set of everything that should still have a cache entry at all. That’s exactly correct for a run that’s guaranteed to have touched every target that matters, such as a full `test:all` or a full release build. It’s exactly wrong for anything less than that. A single test file, `test:pattern`, `test:path`, or the ad hoc single-file `release:compile:`/`release:assemble:` tasks would each, if this were set true, silently wipe out cache entries for every target the partial run simply didn't happen to touch -- forcing an unnecessary full rebuild on the next real run. Both pipelines are careful to set it only when they can actually make that guarantee.
 
 ## Debugging a stale/fresh surprise
 
 Separately from anything either pipeline does on a normal run, this subsystem carries an opt-in troubleshooting aid for a human trying to understand *why* something did or didn't rebuild. Setting the `CEEDLING_DEP_DEBUG` developer-only environment variable turns on a tiered snapshot mechanism, off by default. `CEEDLING_DEP_DEBUG=meta` captures just the canonicalized meta a target was judged against. `CEEDLING_DEP_DEBUG=full` additionally captures the actual content of the target and its dependencies at the moment they were last marked fresh. Both tiers write into a human-readable file tree alongside the cache, capped in size per file so one enormous generated header can't silently balloon that tree.
 
-Those snapshots exist to back one thing: `DependencyTracker#diagnose`, an always-available "explain yourself" query that reports which of a target's self-hash, meta-hash, dependency hashes, or output hashes changed. `diagnose` runs regardless of whether `CEEDLING_DEP_DEBUG` is set. Without a snapshot to compare against, it can still report *that* a given hash changed, but not *what* changed. With one available, from a run where the matching tier was active, it produces an actual before/after diff instead.
+Those snapshots exist to back one thing: `DependencyTracker#diagnose`, an always-available "explain yourself" query that reports which of a target’s self-hash, meta-hash, dependency hashes, or output hashes changed. `diagnose` runs regardless of whether `CEEDLING_DEP_DEBUG` is set. Without a snapshot to compare against, it can still report *that* a given hash changed, but not *what* changed. With one available, from a run where the matching tier was active, it produces an actual before/after diff instead.
 
 `DependencyDebugTree` writes and reads those snapshots, and `DependencyDiffer` turns a snapshot and the live content into that diff. The differ uses the `diff-lcs` gem, a development-only dependency that an installed release gem does not require. Without it, the explanation degrades to a plain before/after dump rather than failing.
 

data/lib/ceedling/dependencies/dependency_cache_store.rb

@@ -55,7 +55,9 @@ require 'ceedling/dependencies/dependency_hasher'
 # only when it knows a run's registrations were comprehensive.
 class DependencyCacheStore
 
-  CACHE_SCHEMA_VERSION = 1 unless const_defined?(:CACHE_SCHEMA_VERSION, false)
+  # Cache keys are build paths, so a change to where a build puts its output changes the
+  # schema too. 2: per-test output is placed by the test's path from the project root.
+  CACHE_SCHEMA_VERSION = 2 unless const_defined?(:CACHE_SCHEMA_VERSION, false)
 
   constructor :file_wrapper, :loginator
 
@@ -73,6 +75,7 @@ class DependencyCacheStore
     parsed = JSON.parse( raw )
 
     return degrade( store_path, "does not contain a JSON object" ) unless parsed.is_a?( Hash )
+    return upgrade( store_path ) if earlier_layout?( parsed['schema_version'] )
     return degrade( store_path, "has an incompatible or missing schema_version" ) unless parsed['schema_version'] == CACHE_SCHEMA_VERSION
     return degrade( store_path, "has an incompatible or missing hash_algorithm" ) unless parsed['hash_algorithm'] == DependencyHasher::HASH_ALGORITHM
     return degrade( store_path, "is missing its entries object" ) unless parsed['entries'].is_a?( Hash )
@@ -83,7 +86,7 @@ class DependencyCacheStore
       if !valid_entry?( entry )
         @loginator.log(
           "Dependency cache entry for '#{target}' in #{store_path} is malformed -- dropping just that entry.",
-          Verbosity::COMPLAIN, LogLabels::WARNING
+          Verbosity::COMPLAIN
         )
         pruned << target
       elsif !@file_wrapper.exist?( target )
@@ -136,7 +139,21 @@ class DependencyCacheStore
   def degrade(store_path, reason)
     @loginator.log(
       "Dependency cache #{store_path} #{reason} -- treating it as absent (full rebuild for tracked targets).",
-      Verbosity::COMPLAIN, LogLabels::WARNING
+      Verbosity::COMPLAIN
+    )
+    empty_store
+  end
+
+  def earlier_layout?(version)
+    return version.is_a?( Integer ) && version.between?( 1, CACHE_SCHEMA_VERSION - 1 )
+  end
+
+  # An earlier build layout leaves its output where this build no longer looks
+  def upgrade(store_path)
+    @loginator.log(
+      "Dependency cache #{store_path} comes from an earlier Ceedling build layout -- rebuilding everything once. " \
+      "Run `ceedling clobber` to remove any orphaned build output left in the old layout.",
+      Verbosity::COMPLAIN
     )
     empty_store
   end

data/lib/ceedling/dependencies/dependency_hasher.rb

@@ -30,8 +30,18 @@ class DependencyHasher
   HASH_ALGORITHM = 'sha256'.freeze unless const_defined?(:HASH_ALGORITHM, false)
   DIGEST_RE = /\A[0-9a-f]{64}\z/.freeze unless const_defined?(:DIGEST_RE, false)
 
+  # The coarsest file timestamp a filesystem keeps (FAT's two seconds), so a file modified as
+  # the build began never counts as older than it
+  TIMESTAMP_TICK_S = 2
+
   constructor :file_wrapper
 
+  def setup
+    @digests = {}
+    @digests_lock = Mutex.new
+    @unchanged_before = Time.now - TIMESTAMP_TICK_S
+  end
+
   # SHA-256 hex digest of a file's exact on-disk bytes.
   #
   # Read via `read_binary` rather than `read`: Ruby's default text-mode read
@@ -62,8 +72,22 @@ class DependencyHasher
   #   reflect the source -- exactly the failure this class exists to prevent.
   #   Any normalization narrows the margin between those two outcomes in
   #   exchange for skipping some rebuilds; that trade isn't worth it here.
+  #
+  # A file unchanged since before the build began hashes once a build, keyed by its size and
+  # modification time. Many targets share one dependency, as every test executable shares a
+  # library it links. A file modified as the build began or since, as anything the build
+  # writes is, hashes afresh every time: a timestamp is too coarse to show a rewrite within
+  # one tick.
   def hash_of_file(path)
-    Digest::SHA256.hexdigest( @file_wrapper.read_binary( path ) )
+    status = @file_wrapper.stat( path )
+    return digest_of( path ) unless memoizable?( status )
+
+    key = [path, status.size, status.mtime]
+    @digests_lock.synchronize { return @digests[key] if @digests.key?( key ) }
+
+    digest = digest_of( path )
+    @digests_lock.synchronize { @digests[key] = digest }
+    return digest
   end
 
   # SHA-256 hex digest of a canonicalized `meta` value, or nil for nil/empty
@@ -91,4 +115,14 @@ class DependencyHasher
     end
   end
 
+  private
+
+  def digest_of(path)
+    return Digest::SHA256.hexdigest( @file_wrapper.read_binary( path ) )
+  end
+
+  def memoizable?(status)
+    return !status.nil? && status.mtime < @unchanged_before
+  end
+
 end

data/lib/ceedling/filename_extension.rb

@@ -5,7 +5,7 @@
 #   SPDX-License-Identifier: MIT
 # =========================================================================
 
-require 'ceedling/file_path_utils'
+require 'ceedling/filesystem/file_path_utils'
 
 # A configured filename-extension setting for one file type (source, header,
 # assembly, and so on). A project may name a file type with a single
@@ -51,6 +51,12 @@ class FilenameExtension
     @extensions.include?(File.extname(filepath))
   end
 
+  # A regex source matching any one of this type's extensions, each escaped, as one
+  # non-capturing group, for a pattern naming a file of this type by its ending
+  def pattern
+    return "(?:#{@extensions.map { |ext| Regexp.escape( ext ) }.join( '|' )})"
+  end
+
   # Every filename `basename` could take on under this type, one per
   # configured extension. Used to search a directory for a file of this
   # type when its extension isn't known in advance -- a build input file

data/lib/ceedling/filesystem/FILE_TREE.md

@@ -0,0 +1,253 @@
+# File Trees
+
+Ceedling copies trees of files whose types it must not misjudge. `FileTree` walks and copies those trees without `lstat`. It lists each directory with `Dir.children` and classifies each entry with `File.stat`. It brings a copy up to date file by file, writes each file whole, and replaces any entry of the wrong type. This is best effort. It removes the dependence on `lstat`, and trusts `stat` in its place, which no filesystem guarantees.
+
+This document is for developers and coding agents changing code that walks or copies trees. It begins with orientation and ends with detail. The orientation explains the problem, where Ceedling copies trees, the rules the code follows, and the invariants a change must keep. Later sections trace each rule to its methods, then cover limits, platforms, and testing.
+
+Statements of fact carry an evidence label where the evidence is not the code itself.
+
+| Label | Meaning |
+| --- | --- |
+| **[verified]** | Run and observed, as the testing section describes |
+| **[sourced]** | Taken from vendor documentation, not reproduced here |
+| **[inferred]** | Reasoned from sources, not confirmed |
+| **[unverified]** | A hypothesis awaiting a run on the platform in question |
+
+## Orientation
+
+### The problem
+
+`stat` and `lstat` both report a path's type. `stat` follows a link and reports its target. `lstat` reports the link itself. Code that must not follow links, such as a recursive copy or a recursive delete, classifies entries with `lstat`. Ruby's `FileUtils.cp_r` and `FileUtils.rm_rf` do.
+
+On some filesystems `lstat` reports a regular file as a directory while `stat` reports it correctly. Windows reparse points are the known case. A reparse point is a file or directory carrying a tag that a filesystem filter interprets. Some tags make an entry a link to another name, such as a symbolic link or a junction. These are name surrogates. Others mark a transparent storage optimization, such as Data Deduplication (`0x80000013`) or Windows Overlay Filter compression (`0x80000017`). These are not links, and reading such a file returns its contents as usual **[sourced]**. Ruby on Windows can report a non-link reparse point as a directory from `lstat` **[inferred]**. A file on a deduplicated network share has been observed to do so, `stat` reporting it as a file **[sourced]**.
+
+A copy that classifies by `lstat` then makes a directory where the file belongs, and fails reading the file as a directory **[verified]**. The stray directory remains, and every later copy collides with it. Retrying cannot help, because the misreport and the leftover both persist.
+
+A glob is no safer a walk. `Dir.glob( '**/*' )` lists a linked directory without descending into it **[verified]**. It reads `[`, `{`, `*`, and `?` in a pattern as syntax. How it decides an entry's type on Windows is unknown **[unverified]**.
+
+### Where Ceedling copies trees
+
+| Copy | Path through the code | When |
+| --- | --- | --- |
+| Unity, CMock, and CException into `build/vendor/` | `ConfiguratorSetup#vendor_frameworks_and_support_files` → `FileWrapper#sync_tree` → `FileTree.sync` | Before build tasks, for a test or release build only |
+| `ceedling.h` and the gdb backtrace script | `ConfiguratorSetup#vendor_test_support_files` → `FileWrapper#sync_file` → `FileTree.sync_file` | Before test build tasks, when Partials or gdb backtraces are on |
+| Ceedling into a project's `vendor/ceedling/`, examples, and docs | `Actionator#copy_directory` → `FileWrapper#tree_files` → `FileTree.each_file` | `ceedling new --local`, `upgrade`, `example`, and `--docs` |
+
+### The rules
+
+1. **Classify by `stat`.** Every source entry is classified with `File.stat`. A destination entry is checked with `File.directory?`, `File.file?`, and `File.exist?`, which follow links as `stat` does.
+2. **List with `Dir.children`.** A directory's names are read as names, never as a pattern, and sorted.
+3. **Follow links.** A link's target is copied, so a copy holds only regular files and directories. A link loop, a dangling link, a special file such as a socket, or an entry that cannot be read or listed raises a `CeedlingException` naming it.
+4. **Write whole.** A file is written to a temporary name beside its destination and renamed over it. A reader sees the old file or the new one, never part of either.
+5. **Write only what differs.** A destination file matching its source in size and content is left alone, and keeps its modification time.
+6. **Replace the wrong type.** A directory where a file belongs, or a file or dangling link where a directory belongs, is removed and logged before the copy goes on.
+7. **Retry only what clears.** A file's whole sync, from its type check to its write, is retried after each of a series of delays when it fails with `EACCES`, `EBUSY`, `EPERM`, or `EINTR`. Any other error fails at once.
+
+### Code organization
+
+| File | Module or class | Role |
+| --- | --- | --- |
+| `file_tree.rb` | `FileTree` | The walk, the sync, healing, and retries. A pure module, used by `lib/` and `bin/` alike |
+| `file_wrapper.rb` | `FileWrapper` | `#sync_tree`, `#sync_file`, and `#tree_files` give `FileTree` Ceedling's publish step, retry delays, and logging |
+| `../config/configurator_setup.rb` | `ConfiguratorSetup` | Choosing which frameworks a build vendors, and when |
+| `../../../bin/actionator.rb` | `Actionator` | Copying trees for the command line, with its own status lines and skip rules |
+
+### Invariants
+
+Every change to this code must keep these invariants. Specs pin each one.
+
+- No walk classifies by `lstat`. A copy uses it only on the destination: `FileUtils.rm_r` on a directory with contents, and `File.symlink?` to find a link.
+- A copy holds only regular files and directories.
+- A file that matches its source is never written.
+- A destination file is never visible half-written.
+- A walk ends. A link loop raises rather than recursing.
+- An error that retrying cannot clear surfaces at once.
+
+## Walking: `FileTree.each_file` and `.each_entry`
+
+`each_entry( root, skip: )` yields each directory and regular file below `root`, a directory ahead of its contents, as its path relative to `root`, its full path, and whether it is a directory. `each_file( root, skip: )` yields only the files, as relative and full paths. Without a block, it returns an Enumerator. `skip`, when given, is a callable given each entry's name. An entry it accepts is neither yielded nor entered, nor even stat'd.
+
+Each directory's names come from `Dir.children` and are sorted. A directory that cannot be listed raises "could not be listed". The walk stats each entry. A directory is descended, a regular file is yielded, and anything else raises a `CeedlingException` naming the entry and its type. An entry `stat` cannot read raises "could not be read".
+
+`ENOENT` or `ELOOP` from `stat` means a link points nowhere, and raises "is a dangling or unresolvable link". Before descending, the walk resolves the directory with `File.realpath` and compares it to the real paths of the directories above it. A match raises "Link loop". A directory whose real path cannot be resolved stands for itself, so the walk goes on without that safeguard rather than stopping. A root that is not a directory raises at once.
+
+## Copying: `FileTree.sync` and `.sync_file`
+
+`sync( source, destination, publish:, retry_delays:, sleep:, log: )` brings every file below `source` up to date below `destination`. It returns a `Result` counting files `copied`, `unchanged`, and `healed`. `sync_file( source, destination_dir, … )` does the same for one regular file.
+
+`publish` is a callable given a source and a destination path. It does the writing, so the caller decides how a file is written and checked. `retry_delays` lists seconds to wait between attempts. `sleep` and `log` are seams for specs and for Ceedling's logging. `log` receives a message, a `Verbosity` level, and a `LogLabels` value.
+
+### Directories
+
+A destination directory that exists is kept. A file or a dangling link in its place is removed, counted as healed, and logged. Then `FileUtils.mkdir_p` makes the directory, so empty source directories appear in the copy too.
+
+### Files
+
+A destination directory in a file's place is removed. A link to a directory is unlinked, `Dir.rmdir` removes an empty directory, and `FileUtils.rm_r` one with contents. Any error removing it reaches the retries. Another run may heal the same entry first. `ENOENT` means it removed the directory, and `ENOTDIR` that it already published the file there. This run then counts no heal, and publishes over that file rather than deleting it, so a reader always finds one. A destination file of the same size and the same bytes, by `FileUtils.compare_file`, is counted unchanged and left alone. Otherwise `publish` writes the file.
+
+### Retries
+
+A file's sync raising one of `TRANSIENT_ERRORS` at any step waits the next delay and tries the file again. A scanner or sync client can hold the destination open as well as the file being written. Once the delays are spent, the failure is logged and raised as a `CeedlingException` naming the file and the number of retries. Any other error propagates at once, unchanged. Its message names the path.
+
+### Log messages
+
+| Event | Verbosity, label | Message |
+| --- | --- | --- |
+| A directory replaced by a file | COMPLAIN, NOTICE | `Replaced a directory where the file '<path>' belongs` |
+| A file replaced by a directory | COMPLAIN, NOTICE | `Replaced a file where the directory '<path>' belongs` |
+| A retry | DEBUG | `Retrying copy to '<path>' after a transient error: <message>` |
+| Retries spent | ERRORS | `Failed to copy to '<path>' after <n> retries ⏩️ <message>`, then advice to delete `build/` or reinstall the gem |
+
+## FileWrapper's part
+
+`#sync_tree` and `#sync_file` pass `FileTree` four things:
+- `#replace_file` as the publish step;
+- `TRANSIENT_IO_RETRY_DELAYS`, four delays totalling 0.37 s;
+- `sleep`;
+- the Loginator.
+
+`#replace_file` copies the source to `.<name>.<pid>.<thread id>.tmp` beside the destination and renames it into place. It checks that temporary path's length against the platform's limit first. It makes the copy owner-writable, and a read-only destination file too, since Windows refuses to rename over a read-only file. A destination link is replaced, and the file it names keeps its mode. Execute bits carry over from the source.
+
+`#tree_files( dir, skip: )` returns every regular file below `dir` as a full path, sorted as one list. Sorting the full paths puts `a.c` ahead of `a/b.c`, the same order a sorted glob listing gives.
+
+## Actionator's part
+
+`#copy_directory` copies each file `FileWrapper#tree_files` lists, in that order. It passes `skip: Actionator::HIDDEN`, so entries named with a leading dot are never listed or entered, and nothing inside a hidden directory can stop a copy. It leaves out the junk files its exclude pattern matches too. It compares, forces, and reports each file as it always does, so its status lines match Thor's.
+
+## Limits
+
+- **`stat` is trusted.** The design assumes `stat` reports a type correctly wherever `lstat` does not. That is the observation behind it, not a guarantee.
+- **`lstat` remains on the destination.** `FileUtils.rm_r` classifies by `lstat` when it removes a destination directory with contents, and never follows a link out of it. `File.symlink?` asks `lstat` whether a destination entry is a link. Both work on the destination, which Ceedling creates, not on a source a filesystem may misreport.
+- **Special files raise.** A socket, a FIFO, or a device in a source tree stops the copy.
+- **Extra files stay.** A copy never removes destination files its source lacks.
+- **Every file is read.** Comparing a file reads both copies on every build. Vendored trees hold a few hundred small files, so this costs little. Comparing by modification time would cost less, but would trust timestamps.
+- **Timestamps.** A written file takes the time of writing. A skipped file keeps its own. Source times are not preserved, and nothing in Ceedling depends on them.
+- **Concurrent builds.** Each file is renamed into place whole, so two builds sharing one `build/vendor/` never read a partial file. A heal never deletes a file another build already published, and publishes over it instead. A spec simulates that race. No spec runs two processes.
+
+## Platforms
+
+- **Windows.**
+  - Reparse points are the case this design exists for.
+  - A junction is a link to a directory. Ruby reports it as one, `File.symlink?` true and `lstat` a link, and the walk descends it **[verified]**. Windows makes junctions only on local NTFS volumes, never on a UNC share **[verified]**.
+  - `File.rename` replaces an existing file. Windows refuses to replace a read-only one, though Ruby may clear the attribute itself **[inferred]**. `#replace_file` clears it first either way.
+  - Names cannot hold `*` or `?`.
+  - Paths stop near 260 characters. The temporary name adds about 20 characters beyond the destination's, and `check_path_length` warns as a path nears the limit.
+- **macOS and Linux.**
+  - Symlinks are followed.
+  - `rename` over a read-only file succeeds, since a directory's permissions govern it.
+  - On Linux, `Dir.glob` makes no stat-family calls of its own, reading types from the directory instead **[verified]**.
+- **FAT filesystems.**
+  - FAT has no links, refuses `*` and `?` in names, and keeps time to 2 s.
+  - The specs skip what FAT cannot hold.
+
+## Testing and coverage
+
+### Approach
+
+The specs test the shape of the problem rather than the infrastructure that produces it. Real network shares and real Data Deduplication are too heavy for CI, and unnecessary once the failure is expressed as `lstat` misreporting a type.
+
+### The stand-in
+
+`spec/support/file_tree_helper.rb` defines a `LyingStat`, a stat result that claims a directory. `lie_about_lstat( pattern )` stubs `File.lstat` through RSpec to return one for each path matching `pattern`. `FileUtils` classifies entries by calling `File.lstat` from Ruby, so the stub reaches it **[verified]**. RSpec removes the stub after each example. A global patch of `File` would leak between examples.
+
+Three kinds of example stub `File.lstat`. The control and the survival examples use the stand-in, and the guard a stricter stub.
+
+1. **The control.** `FileUtils.cp_r` fails under the stand-in, making a directory where the file belongs **[verified]**. This shows the stand-in reproduces the failure. If a later Ruby stops failing here, review the control rather than deleting it.
+2. **Survival.** `FileTree.sync` copies every file as a file while every entry lies, and `FileTree.each_file` lists every file **[verified]**.
+3. **The guard.** With `File.lstat` stubbed to raise on any call, a sync that heals an empty stray directory still succeeds **[verified]**.
+
+The stand-in proves `FileTree` never classifies by `lstat`. It does not prove a real filesystem behaves as the stand-in does, or that `stat` is right everywhere. It exercises the symptom, not the cause.
+
+### Real trees
+
+`spec/units/filesystem/file_tree_spec.rb` builds real trees in a temporary directory and covers:
+- nested trees, dotfiles, and empty directories;
+- names holding glob metacharacters;
+- an unchanged file keeping its modification time, and a changed file replaced, even at the same size;
+- extra destination files left alone;
+- each kind of healing, the destination root, a dangling link, and a link to a directory included, and a heal another run made first;
+- links: a linked root, a linked file, a linked directory, a loop, and a dangling link;
+- retries on each transient error, on a destination that cannot yet be read, and on each kind of heal, retries spent, and errors that fail at once;
+- entries `skip` names, directories that cannot be listed, entries that cannot be read, and real paths that cannot be resolved;
+- `.sync_file`.
+
+**The canary.** One example shows `Dir.glob( '**/*' )` skipping a linked directory's contents. It characterizes Ruby, not `FileTree`. If it ever fails, Ruby changed, and rule 2's reasoning deserves another look.
+
+Examples needing links skip themselves where links cannot be made. The glob-name examples write only names the filesystem accepts.
+
+Specs elsewhere cover the callers:
+- `spec/units/filesystem/file_wrapper_spec.rb` covers retries through `#sync_tree`, `#replace_file`'s writable bits and linked destinations, and `#tree_files`.
+- `spec/integration/vendor_copy_transient_io_retry_spec.rb` vendors real trees through `ConfiguratorSetup`: a stray directory at any file, a misreporting `lstat`, read-only files, and faults that clear, persist, or cannot clear.
+- `spec/integration/actionator_file_operations_spec.rb` covers sorted status lines, hidden entries and the hidden directories never entered, glob-named directories, linked directories, and a misreporting `lstat`.
+
+### Windows reparse points
+
+`spec/units/filesystem/file_tree_windows_spec.rb` runs only on Windows.
+- **Junction.** The first example makes a junction with `mklink /J` and copies through it. A canary line prints what `File.symlink?`, `lstat`, and `stat` report for the junction, without asserting any of them. On a UNC share, where Windows cannot make a junction, it skips without trying.
+- **WOF canary.** The second compresses a file with `compact /exe:lzx`, confirms its reparse tag is `0x80000017`, and prints the same canary line for the file. It then asserts the copy is a regular file of full size.
+- Each skips when its runner cannot make the reparse point. The skip names the command that failed, its exit status, and its output. A failed `compact` also reports the state of the WOF driver.
+
+On hosted Windows runners:
+- The junction example passes on the `C:` and `D:` drives and on a VHD-backed NTFS volume. Its canary reads `symlink?=true lstat.ftype=link stat.ftype=directory` on each **[verified]**.
+- The WOF example cannot run on a hosted runner, though the WOF driver runs there **[verified]**. It skips on every volume, saying why:
+
+  | Volume | What happens |
+  | --- | --- |
+  | `D:`, and a VHD-backed NTFS volume | `compact /exe` fails with "The file system does not support compression" |
+  | `C:` | `compact /exe` succeeds, but the file carries no reparse point |
+  | The UNC share | `compact /exe` fails with "The request is not supported" |
+
+- Whether Ruby misreports a WOF file is therefore unknown **[unverified]**. The example still runs, and its canary still prints, on a Windows machine that can compress a file this way.
+
+### Other filesystems
+
+Every `FileTree` spec carries `filesystem: true`, so `rspec spec/units/filesystem --tag filesystem` selects them. `CEEDLING_TEST_FS_ROOT` names a directory on another filesystem, and the specs build their trees there instead of the system temporary directory.
+
+| Filesystem | What differs | Where it runs | Status |
+| --- | --- | --- | --- |
+| APFS | Case-insensitive by default | macOS test legs, and locally | **[verified]** |
+| ext4 | The Linux baseline | Linux test legs, and locally in Docker | **[verified]** |
+| tmpfs | Memory-backed | The Linux Ruby 3.3 leg, with `CEEDLING_TEST_FS_ROOT=/dev/shm` | **[verified]** in CI and locally in Docker |
+| NTFS | The Windows baseline | Windows test legs | **[verified]** |
+| UNC administrative share | Reaches the local drive through the SMB client | The Filesystem Substrates job | **[verified]**, the junction and WOF examples skipping |
+| System drive `C:` | The Windows system volume | The Filesystem Substrates job | **[verified]**, the WOF example skipping |
+| VHD-backed NTFS volume | A separate volume made with `diskpart` | The Filesystem Substrates job | **[verified]**, the WOF example skipping |
+| Loopback FAT image | No links, restricted names, 2 s timestamps | The Filesystem Substrates job | **[verified]**, the link examples skipping |
+
+A failing spec in the Filesystem Substrates job fails CI Gate. A step that cannot set up its filesystem warns, names why, and skips, since that says nothing about Ceedling. Each step runs whatever the others did. The UNC step first enables Windows' evaluation of links on a share that name places on a share.
+
+### Proof the specs can fail
+
+**Mutations.** `FileTree` was broken on purpose, one change at a time **[verified]**:
+- `File.stat` changed to `File.lstat` in the walk made 7 examples fail;
+- `Dir.children` changed to `Dir.glob` in the walk made 5 fail.
+
+**Seed sweeps.** 100 random orderings of the spec file pass on APFS, ext4, and tmpfs **[verified]**.
+
+### Coverage
+
+| Scenario | Layer | Linux | macOS | Windows |
+| --- | --- | --- | --- | --- |
+| `lstat` misreports a file | Stand-in | yes | yes | yes |
+| No walk or copy calls `File.lstat` | Stand-in guard | yes | yes | yes |
+| Tree shapes, glob names, unchanged and changed files | Real trees | yes | yes | yes |
+| Healing entries of the wrong type | Real trees | yes | yes | yes |
+| Links | Real trees, skipping without links | yes | yes | where permitted |
+| Junctions | Windows real trees, canary | n/a | n/a | yes, except on a UNC share, which cannot hold one |
+| WOF reparse points | Windows real trees, canary | n/a | n/a | not on hosted runners, which cannot make a WOF file |
+| Retries and failures | Injected faults | yes | yes | yes |
+| Other filesystems | `CEEDLING_TEST_FS_ROOT` | tmpfs, FAT | none | UNC, `C:`, VHD |
+
+### Not covered
+
+- A real network share whose `lstat` answers change from call to call, and real Data Deduplication.
+- Any non-link reparse point on a hosted runner. WOF compression, the one tried, cannot be applied there.
+- Antivirus and cloud-sync software holding files open.
+- Two Ceedling processes copying into one `build/vendor/` at once.
+- Windows paths near 260 characters with the temporary name's suffix.
+- `stat` misreporting a type. The design assumes it does not.
+
+## Sources
+
+- Microsoft, [Reparse Tags](https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4) (MS-FSCC 2.1.2.1), for tag values and name surrogates.
+- Microsoft, [Data Deduplication overview](https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/overview), for how optimized files are stored.