ceedling 1.0.1 → 1.1.0

data/docs/mkdocs/plugins/gcov/overview.md

@@ -0,0 +1,145 @@
+# Plugin Overview
+
+!!! note
+    When enabled, this plugin creates a new set of `gcov:` tasks that mirror
+    Ceedling’s existing `test:` tasks. A `gcov:` task executes one or more tests
+    with coverage enabled for the source files exercised by those tests.
+
+    `gcov:` tasks entirely duplicate `test:` tasks and test builds because of 
+    the needs of coverage instrumentation at compile time.
+
+This plugin also provides an extensive set of options for generating various
+coverage reports for your project. The simplest is text-based coverage 
+summaries printed to the console after a `gcov:` test task is executed.
+
+## Simple Coverage Summaries
+
+In its simplest usage, this plugin outputs coverage statistics to the console
+for each source file exercised by a test. These console-based coverage 
+summaries are provided after the standard Ceedling test results summary. Other 
+than enabling the plugin and ensuring `gcov` is installed, no further set up 
+is necessary to produce these summaries.
+
+!!! tip
+    [Automatic summaries may be disabled](setup.md#disabling-automatic-coverage-summaries).
+
+When the Gcov plugin is active it enables Ceedling tasks like this:
+
+```shell
+ > ceedling gcov:Model
+```
+
+… that then generate output like this:
+
+```
+--------------------------
+GCOV: OVERALL TEST SUMMARY
+--------------------------
+TESTED:  1
+PASSED:  1
+FAILED:  0
+IGNORED: 0
+
+---------------------------
+GCOV: CODE COVERAGE SUMMARY
+---------------------------
+
+TestModel
+---------
+Model.c | Lines executed:100.00% of 4
+Model.c | No branches
+Model.c | No calls
+TimerModel.c | Lines executed:0.00% of 3
+TimerModel.c | No branches
+TimerModel.c | No calls
+```
+
+## Advanced Coverage Reports
+
+For more advanced visualizations and reporting, this plugin also supports a 
+variety of report generation options.
+
+Advanced report generation uses [gcovr] and / or [ReportGenerator] to generate
+HTML, XML, JSON, or text-based reports from coverage-instrumented test runs. 
+See the tools' respective sites for examples of the reports they can generate.
+
+In the default configuration, if reports are enabled, this plugin automatically
+generates reports in the build’s `artifacts/` directory after each execution of
+a `gcov:` task.
+
+An optional setting documented below disables automatic report generation, 
+providing a separate Ceedling task instead. Reports can then be generated
+on demand after test suite runs.
+
+[gcovr]: https://www.gcovr.com/
+[ReportGenerator]: https://reportgenerator.io
+
+## Summaries vs. Reports
+
+Coverage summaries and coverage reports provide different levels of fidelity
+and usability. Summaries are relatively unsophisticated while reports are 
+sophisticated. As such, both provide different capabilities and levels of 
+usability.
+
+### Coverage summaries
+
+Optional coverage summaries are intentionally simple. They require no
+configuration and, to oversimplify, are largely filtered output from the `gcov`
+tool.
+
+Coverage summaries are reported to the console for each source file exercised by
+the tests executed by `gcov:` tasks. That is, coverage summaries correspond to
+the tests executed, and in turn, the source code that your tests call. This
+could be all tests (and thus all source code) or a subset of tests (and some
+subset of source code). The `gcov` tool is run multiple times after test suite
+execution in direct relation to the set of tests you ran with `gcov:` testing 
+tasks. In short, the scope of coverage summaries is guaranteed to match the
+test suite you run.
+
+Coverage summaries do not include any sort of grand total, final tallies. This 
+is the domain of full coverage reports.
+
+Note that Ceedling can exercise the same source code under multiple scenarios
+using multiple test files. Practically, this means that the same source file
+may be listed in the coverage summaries more than once. That said, its coverage
+statistics will be the same each time — the aggregate result of all tests that
+exercised it.
+
+### Coverage reports
+
+Coverage reports provide both much more detail and better overviews of coverage
+than the console-based coverage summaries. However, with this comes the need
+for more sophisticated configuration and certain caveats on what is reported.
+
+Later sections detail how to configure the reports this plugin can generate.
+
+Of note is a consequence of how reports are generated and the limits of the
+tools that do so. Reports are generated using coverage results on disk. The
+report generation tools slurp up the coverage results they find in the `gcov/`
+build output directory. This means that previous test suite runs can "pollute"
+coverage reports. The solution is simple if blunt — run the `clobber` task
+before running a coverage-instrumented test suite. This will yield a coverage
+report with scope that matches that of the test suite you run.
+
+Both the `gcovr` and `reportgeneator` reporting utilities include powerful
+filters that can limit the scope of reports. Hypothetically, it’s possible for
+coverage reports to have the same clear scope as coverage summaries. However,
+in large projects, these filters would cause impractically long command lines.
+Both tools provide configuration file options that would solve the command line
+problem. However, this feature is "experimental" for `gcovr` and considerable 
+work to implement for both reporting utilities. At present, running 
+`ceedling clobber` before generating reports is the best option to ensure 
+accurate reports.
+
+## References
+
+Much of the text describing report generations options in this document was 
+taken from the [Gcovr User Guide][gcovr-user-guide] and the
+[ReportGenerator Wiki][report-generator-wiki].
+
+The text is repeated here to provide as useful documenation as possible.
+
+[gcovr-user-guide]: https://www.gcovr.com/en/stable/guide.html
+[report-generator-wiki]: https://github.com/danielpalme/ReportGenerator/wiki
+
+<br/><br/>

data/docs/mkdocs/plugins/gcov/reportgenerator.md

@@ -0,0 +1,58 @@
+# ReportGenerator Configuration
+
+The `ReportGenerator` utility may be configured with the following configuration items.
+
+All generated reports are found in `<build root>/artifacts/gcov/ReportGenerator/`.
+
+```yaml
+:gcov:
+  :report_generator:
+    # Optional directory for storing persistent coverage information.
+    # Can be used in future reports to show coverage evolution.
+    :history_directory: <path>
+
+    # Optional plugin files for custom reports or custom history storage (separated by semicolon).
+    :plugins: <plugin.dll>;<*.dll>
+
+    # Optional list of assemblies that should be included or excluded in the report (separated by semicolon).
+    # Exclusion filters take precedence over inclusion filters.
+    # Wildcards are allowed, but not regular expressions.
+    :assembly_filters: +<included>;-<excluded>
+
+    # Optional list of classes that should be included or excluded in the report (separated by semicolon).
+    # Exclusion filters take precedence over inclusion filters.
+    # Wildcards are allowed, but not regular expressions.
+    :class_filters: +<included>;-<excluded>
+
+    # Optional list of files that should be included or excluded in the report (separated by semicolon).
+    # Exclusion filters take precedence over inclusion filters.
+    # Wildcards are allowed, but not regular expressions.
+    # Example: "-./vendor/*;-./build/*;-./test/*;-./lib/*;+./src/*"
+    :file_filters: +<included>;-<excluded>
+
+    # The verbosity level of the log messages.
+    # Values: Verbose, Info, Warning, Error, Off (defaults to Warning)
+    :verbosity: <level>
+
+    # Optional tag or build version.
+    :tag: <tag>
+
+    # Optional list of one or more regular expressions to exclude gcov notes files that match these filters.
+    :gcov_exclude:
+      - <regex>
+      - ...
+
+    # Optionally set the number of threads to use in parallel. Defaults to 1.
+    :threads: <count>
+
+    # Optional list of one or more command line arguments to pass to Report Generator.
+    # Useful for configuring Risk Hotspots and Other Settings.
+    # https://github.com/danielpalme/ReportGenerator/wiki/Settings
+    # Note: This can be accomplished with Ceedling's tool configuration options outside of plugin 
+    #       configuration but is supported here to collect configuration options in one place.
+    :custom_args:
+      - <argument>
+      - ...
+```
+
+<br/><br/>

data/docs/mkdocs/plugins/gcov/reporting.md

@@ -0,0 +1,147 @@
+# Reporting Configuration
+
+Various reports are available. Each must be enabled in `:gcov` ↳ `:reports`.
+
+If no report types are specified, report generation (but not coverage summaries) 
+is disabled regardless of any other setting.
+
+Most report types can only be generated by `gcovr` or `ReportGenerator`. Some 
+can be generated by both. This means that your selection of report is impacted by
+which generation utility is enabled. In fact, in some cases, the same report type 
+could be generated by each utility (to different artifact build output folders).
+
+!!! note
+    Coverage results are cumulative across builds. That is, if you run a single
+    test file after having run a previous test file, coverage results for both
+    will be present in generated reporting.
+
+    For clean and clear results, run `ceedling clobber` before any coverage 
+    tasks.
+
+Reports are configured with:
+
+1. General or common options for each report generation utility
+1. Specific options for types of report per each report generation utility
+
+For tool-specific plugin configuration options, see:
+
+- [GCovr Configuration](gcovr.md)
+- [ReportGenerator Configuration](reportgenerator.md)
+
+See the [GCovr User Guide][gcovr-user-guide] and the 
+[ReportGenerator Wiki][report-generator-wiki] for full details on using
+these tools.
+
+[gcovr-user-guide]: https://www.gcovr.com/en/stable/guide.html
+[report-generator-wiki]: https://github.com/danielpalme/ReportGenerator/wiki
+
+| Report option | gcovr | ReportGenerator |
+|---|:---:|:---:|
+| `HtmlBasic` | ✓ | ✓ |
+| `HtmlDetailed` | ✓ | ✓ |
+| `Text` | ✓ | ✓ |
+| `Cobertura` | ✓ | ✓ |
+| `SonarQube` | ✓ | ✓ |
+| `JSON` | ✓ | |
+| `HtmlInline` | | ✓ |
+| `HtmlInlineAzure` | | ✓ |
+| `HtmlInlineAzureDark` | | ✓ |
+| `HtmlChart` | | ✓ |
+| `MHtml` | | ✓ |
+| `Badges` | | ✓ |
+| `CsvSummary` | | ✓ |
+| `Latex` | | ✓ |
+| `LatexSummary` | | ✓ |
+| `PngChart` | | ✓ |
+| `TeamCitySummary` | | ✓ |
+| `lcov` | | ✓ |
+| `Xml` | | ✓ |
+| `XmlSummary` | | ✓ |
+
+```yaml
+:gcov:
+  # Specify one or more reports to generate.
+  # Defaults to HtmlBasic.
+  :reports:
+    # Generate an HTML summary report.
+    # Supported utilities: gcovr, ReportGenerator
+    - HtmlBasic
+
+    # Generate an HTML report with line by line coverage of each source file.
+    # Supported utilities: gcovr, ReportGenerator
+    - HtmlDetailed
+
+    # Generate a Text report, which may be output to the console with gcovr or a file in both gcovr and ReportGenerator.
+    # Supported utilities: gcovr, ReportGenerator
+    - Text
+
+    # Generate a Cobertura XML report.
+    # Supported utilities: gcovr, ReportGenerator
+    - Cobertura
+
+    # Generate a SonarQube XML report.
+    # Supported utilities: gcovr, ReportGenerator
+    - SonarQube
+
+    # Generate a JSON report.
+    # Supported utilities: gcovr
+    - JSON
+
+    # Generate a detailed HTML report with CSS and JavaScript included in every HTML page. Useful for build servers.
+    # Supported utilities: ReportGenerator
+    - HtmlInline
+
+    # Generate a detailed HTML report with a light theme and CSS and JavaScript included in every HTML page for Azure DevOps.
+    # Supported utilities: ReportGenerator
+    - HtmlInlineAzure
+
+    # Generate a detailed HTML report with a dark theme and CSS and JavaScript included in every HTML page for Azure DevOps.
+    # Supported utilities: ReportGenerator
+    - HtmlInlineAzureDark
+
+    # Generate a single HTML file containing a chart with historic coverage information.
+    # Supported utilities: ReportGenerator
+    - HtmlChart
+
+    # Generate a detailed HTML report in a single file.
+    # Supported utilities: ReportGenerator
+    - MHtml
+
+    # Generate SVG and PNG files that show line and / or branch coverage information.
+    # Supported utilities: ReportGenerator
+    - Badges
+
+    # Generate a single CSV file containing coverage information per file.
+    # Supported utilities: ReportGenerator
+    - CsvSummary
+
+    # Generate a single TEX file containing a summary for all files and detailed reports for each files.
+    # Supported utilities: ReportGenerator
+    - Latex
+
+    # Generate a single TEX file containing a summary for all files.
+    # Supported utilities: ReportGenerator
+    - LatexSummary
+
+    # Generate a single PNG file containing a chart with historic coverage information.
+    # Supported utilities: ReportGenerator
+    - PngChart
+
+    # Command line output interpreted by TeamCity.
+    # Supported utilities: ReportGenerator
+    - TeamCitySummary
+
+    # Generate a text file in lcov format.
+    # Supported utilities: ReportGenerator
+    - lcov
+
+    # Generate a XML file containing a summary for all classes and detailed reports for each class.
+    # Supported utilities: ReportGenerator
+    - Xml
+
+    # Generate a single XML file containing a summary for all files.
+    # Supported utilities: ReportGenerator
+    - XmlSummary
+```
+
+<br/><br/>

data/docs/mkdocs/plugins/gcov/setup.md

@@ -0,0 +1,232 @@
+# Set up & Configuration
+
+## Toolchain dependencies
+
+### GNU Compiler Collection
+
+This plugin relies on the GNU compiler collection. Coverage instrumentation
+is enabled through `gcc` compiler flags. Coverage-insrumented executables 
+(i.e. test suites) output coverage result files to disk when run. `gcov`,
+`gcovr`, and `reportgenerator` (the tools managed by this plugin) all produce
+their coverage tallies from these files. `gcov` is part of the GNU compiler
+collection. The other tools — detailed below — require separate installation.
+
+Ceedling’s default toolchain is the same as needed by this plugin. If you
+are already running Ceedling test suites with the GNU compiler toolchain, 
+you are good to go. If you are using another toolchain for test suite and/or
+release builds you will need to install the GNU compiler collection to use
+this plugin. Depending on your needs you may also need to install the reporting
+utilities, `gcovr` and/or `reportgenerator`.
+
+### `gcovr` and `reportgenerator`’s dependence on `gcov`
+
+Both the `gcovr` and `reportgenerator` tools depend on the `gcov` tool. This
+dependency plays out in two different ways. In both cases, the report
+generation utilities ingest `gcov`’s output to produce their artifacts. As
+such, `gcov` must be available in your environment if using report generation.
+
+1. `gcovr` calls `gcov` directly.
+
+    Because it calls `gcov` directly, you are limited as to the
+    advanced Ceedling features you can employ to modify `gcov`’s execution.
+    However, with a configuration option (see below) you can instruct `gcovr`
+    to call something other than `gcov` (e.g. a script that intercepts and
+    modifies how `gcovr` calls out to `gcov`).
+
+    `gcovr` instructs `gcov` to generate `.gcov` files that it processes and
+    discards. A `gcovr` option documented below will retain the `.gcov` files.
+
+2. `reportgenerator` expects the existence of `.gcov` files to do its work.
+
+    This Ceedling plugin calls `gcov` appropriately to generate the `.gcov` 
+    files `reportgenerator` needs before then calling the report utility. 
+
+    You can use Ceedling’s features to modify how `gcov` is run before 
+    `reportgenerator`.
+
+## Enable this plugin
+
+To use this plugin it must be enabled in your Ceedling project file:
+
+```yaml
+:plugins:
+  :enabled:
+    - gcov
+```
+
+This simple configuration will create new `gcov:` command line tasks to run 
+tests with source coverage and output simple coverage summaries to the console 
+as above.
+
+## Disabling automatic coverage summaries
+
+To disable the coverage summaries generated immediately following `gcov:` tasks,
+simply add the following to a top-level `:gcov:` section in your project
+configuration file.
+
+```yaml
+:plugins:
+  :enabled:
+    - gcov
+
+:gcov:
+  :summaries: FALSE
+```
+
+## Report generation
+
+To generate reports:
+
+1. GCovr and / or ReportGenerator must installed or otherwise ready to run in
+   Ceedling’s environment.
+1. Reporting options must be configured in your project file beneath a `:gcov:`
+   entry.
+
+The next sections explain each of these steps.
+
+### Modified Condition / Decision Coverage
+
+As of version 14, the GNU Compiler Collection supports MC/DC. If your environment
+contains a minimum of GCC 14 you can enable MC/DC in coverage summaries.
+
+If your environment contains a minimum of GCC 14 and GCovr 8, you can enable MC/DC 
+in your generated coverage reports.
+
+NOTE: ReportGenerator does not support MC/DC reporting.
+
+```yaml
+:plugins:
+  :enabled:
+    - gcov
+
+:gcov:
+  :mcdc: TRUE
+```
+**Default:** `FALSE`
+
+### Coverage for untested sources
+
+This setting controls how the GCov plugin handles project source files that
+are not exercised by any test. It takes one of three values:
+
+* `:ignore` — Untested source files are not processed at all. Nothing is
+  logged, nothing is compiled, and these files are simply absent from the
+  coverage report.
+* `:list` — Untested source files are not compiled with coverage, but their
+  filepaths are logged as a warning so you know which files will not appear
+  in the coverage report.
+* `:compile` — All untested source files are compiled with coverage so they
+  appear in the final report with 0% coverage (since no test exercises
+  them). This causes all source files to appear in any generated reporting.
+  If a source file fails to compile, Ceedling logs guidance at the console,
+  and the build fails.
+
+```yaml
+:plugins:
+  :enabled:
+    - gcov
+
+:gcov:
+  :untested_sources: :list
+```
+**Default:** `:list`
+
+!!! warning
+    **Compiling all untested sources for 0% coverage reporting (`:compile`) will likely require additional work.**
+
+    Successful compilation of untested source files may require certain symbols 
+    to be defined, certain flags to be set, or entire stand-in shims for platform 
+    headers and code.
+    
+    Ceedling’s 
+    [`:defines`](../../configuration/reference/defines.md)
+    and 
+    [`:flags`](../../configuration/reference/flags.md)
+    matchers can provide these. For GCov tasks, symbols and flags are extracted from
+    the `:test` context beneath the `:defines` and `:flags` configuration sections 
+    by default. If you need something special for coverage builds, use the `:gcov` 
+    context for these matchers instead.
+
+Notes:
+
+* Versions of GCovr before 7.0 do not include the necessary options to produce 
+  0% coverage results for source files only compiled (but never executed).
+
+---
+
+When `:untested_sources` is `:compile`, an additional `gcov:untested_sources`
+build task becomes available:
+
+```shell
+ > ceedling gcov:untested_sources
+```
+
+This task exists to let you work through the compilation problems described
+in the warning above — missing symbols, flags, or platform header/code
+stand-ins — by re-running just the untested-source compilation step
+directly. No test suite build is needed while iterating on source compilation 
+fixes.
+
+### Reporting utilities installation
+
+!!! tip "Variants of the `madsciencelab` Docker images come with these tools preinstalled"
+    See [the Docker image options](../../getting-started/installation.md#madsciencelab-docker-images)
+    for running Ceedling.
+
+[gcovr] is available on any platform supported by Python.
+
+`gcovr` can be installed via pip like this:
+
+```shell
+ > pip install gcovr
+```
+
+[ReportGenerator] is available on any platform supported by .Net.
+
+`ReportGenerator` can be installed via .NET Core like so:
+
+```shell
+ > dotnet tool install -g dotnet-reportgenerator-globaltool
+```
+
+Either or both of `gcovr` or `ReportGenerator` may be used. Only one must 
+be installed for advanced report generation.
+
+[gcovr]: https://www.gcovr.com/
+[ReportGenerator]: https://reportgenerator.io
+
+### Enabling reporting utilities
+
+If reports are configured (see next sections) but no `:utilities:` subsection 
+exists, this plugin defaults to using `gcovr` for report generation.
+
+Otherwise, enable Gcovr and / or ReportGenerator to create coverage reports.
+
+```yaml
+:gcov:
+  :utilities:
+    - gcovr           # Use `gcovr` to create reports (default if no :utilities set).
+    - ReportGenerator # Use `ReportGenerator` to create reports.
+```
+
+### Automatic and manual report generation
+
+By default, if reports are specified, this plugin automatically generates 
+reports after any `gcov:` task is executed. To disable this behavior, add
+`:report_task: TRUE` to your project file’s `:gcov:` configuration.
+
+With this setting enabled, an additional Ceedling task `report:gcov` is enabled.
+It may be executed after `gcov:` tasks to generate the configured reports.
+
+For small projects, the default behavior is likely preferred. This alernative 
+setting allows large or complex projects to execute potentially time intensive 
+report generation only when desired.
+
+Enabling the manual report generation task looks like this:
+
+```yaml
+:gcov:
+  :report_task: TRUE
+```
+
+<br/><br/>

data/docs/mkdocs/plugins/gcov/tool-versions.md

@@ -0,0 +1,60 @@
+# Supported tool versions
+
+_**Last updated:**_ May 17, 2026
+
+At the time of the last major updates to the Gcov plugin, the following notes
+on version compatibility were known to be accurate.
+
+Keep in mind that for proper functioning, you do not necessarily need to
+install all the tooks the Gcov plugin works with. Depending on configuration
+options documented in later sections, any of the following tool combinations
+may be sufficient for your needs:
+
+1. `gcov`
+1. `gcov` + `gcovr`
+1. `gcov` + `reportgenerator`
+1. `gcov` + `gcovr` + `reportgenerator`
+
+## `gcov`
+
+The Gcov plugin is known to work with `gcov` packaged with GNU Compiler
+Collection 12 through at least 15.
+
+The maintainers of `gcov` introduced significant behavioral changes for
+version 12. Previous versions of `gcov` had a simple exit code scheme with only a
+single non-zero exit code upon fatal errors. Since version 12 `gcov` emits a
+variety of exit codes even if the noted issue is a non-fatal error. The Gcov
+plugin's logic assumes version 12 behavior and processes failure messages and
+exit codes appropriately, taking into account plugin configuration options.
+
+The Gcov plugin should be compatible with versions of `gcov` before version 12.
+That is, its improved `gcov` exit handling should not be broken by the prior
+simpler behavior. The Gcov plugin dependes on the `gcov` command line and has
+been compatible with it as far back as `gcov` version 7.
+
+Because long file paths are quite common in software development scenarios, by
+default, the Gcov plugin depends on the `gcov` `-x` flag. This flag hashes long
+file paths to ensure they are not a problem for certain platforms' file
+systems. This flag became available with `gcov` version 7. We do not recommend 
+using `gcov` version 6 and earlier. And, in fact, because of the Gcov plugin's 
+dependence on the `gcov` `-x` flag, attempting to use it will fail.
+
+GNU Compiler Collection 14 introduced changes in how coverage is instrumented.
+The `gcov` plugin implemented a revised means of processing coverage that is
+forward compatible with GCC 14+ and backwards compatible to the earliest
+versions of the collection.
+
+## `gcovr`
+
+The Gcov plugin is known to work with `gcovr` 5.2 through `gcovr` 8.x. The
+Gcov plugin supports `gcovr` command line conventions since version 4.2 and
+attempts to support `gcovr` command lines before version 4.2. We recommend 
+using `gcovr` 5 and later.
+
+## `reportgenerator`
+
+The Gcov plugin is known to work with `reportgenerator` 5.2.4. The command line
+for executing `reportgenerator` that the Gcov plugin relies on has largely been
+stable since version 4. We recommend using `reportgenerator` 5.0 and later.
+
+<br/><br/>