@cratis/pi 7.102.0 → 7.103.0

package/package/corpus/skills/microbenchmarking/LICENSE

@@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) .NET Foundation and Contributors
+
+All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/package/corpus/skills/microbenchmarking/NOTICE

@@ -0,0 +1,36 @@
+Benchmark Authoring Guidelines
+Copyright (c) .NET Foundation and Contributors
+Licensed under the MIT License. See LICENSE.
+
+Upstream source: https://github.com/dotnet/skills/blob/a6bf43ab61967c30919334986515163ab24d7635/plugins/dotnet-diag/skills/microbenchmarking/SKILL.md
+
+Changes made by Cratis:
+
+LICENSE:
+- Copy upstream MIT license and copyright verbatim.
+
+SKILL.md:
+- Package guidance defers to Central Package Management pins when `Directory.Packages.props` exists.
+- Reword the historical-measurements axis in neutral present-tense wording.
+- Normalize frontmatter serialization, add MIT license where absent; preserve name and description.
+- Add marked modification/source comment and applicability/Cratis precedence line.
+- Rewrite the description to the Cratis shape (what it does, Use when, Not for) and drop references to skills that are not part of this corpus.
+
+references/writing-benchmarks.md:
+- Repair references/-relative sibling link.
+- Add or normalize the top Contents list from existing headings; preserve section content.
+
+references/bdn-internals-and-tuning.md:
+- Reduce release-specific version mentions to the single behavior boundary.
+- Add or normalize the top Contents list from existing headings; preserve section content.
+
+references/comparison-strategies.md:
+- Add or normalize the top Contents list from existing headings; preserve section content.
+
+references/diagnosers-and-exporters.md:
+- Add or normalize the top Contents list from existing headings; preserve section content.
+
+references/project-setup-and-running.md:
+- Add or normalize the top Contents list from existing headings; preserve section content.
+
+NOTICE: Add this attribution, change list, and any download/execution disclosure.

package/package/corpus/skills/microbenchmarking/SKILL.md

@@ -0,0 +1,131 @@
+---
+name: microbenchmarking
+description: 'Designs, configures, runs, and reviews BenchmarkDotNet (BDN) benchmarks, including project setup, efficient runs, and side-by-side performance comparisons. Use when BenchmarkDotNet is involved, or a .NET performance question needs measurement and BenchmarkDotNet is the likely tool. Not for profiling or tracing (use profiling or dotnet-trace-collect), production telemetry, or load and stress testing.'
+license: MIT
+---
+
+<!-- Modified by Cratis from dotnet's original (MIT): frontmatter, applicability, navigation and the changes listed in NOTICE. Upstream: https://github.com/dotnet/skills (plugins/dotnet-diag/skills/microbenchmarking/SKILL.md; see NOTICE) -->
+
+# Benchmark Authoring Guidelines
+
+Cratis rules in `.cratis/ai/rules/` (including `csharp.md`, `specs.csharp.md`, and `orleans.md`) and the `cratis-*` skills win where they differ.
+
+BenchmarkDotNet (BDN) is a .NET library for writing and running microbenchmarks. Throughout this skill, "BDN" refers to BenchmarkDotNet.
+
+> **Note:** Evaluations of LLMs writing BenchmarkDotNet benchmarks have revealed common failure patterns caused by outdated assumptions about BDN's behavior — particularly around runtime comparison, job configuration, and execution defaults that have changed in recent versions. The reference files in this skill contain verified, current information. **You MUST read the reference files relevant to the task before writing any code** — your training data likely contains outdated or incorrect BDN patterns.
+
+## Key concepts
+
+- **Job** — describes how to run a benchmark: runtime, iteration counts, launch count, run strategy, and environment settings. Multiple jobs can be configured to run the same benchmarks under different conditions.
+- **Benchmark case** — one method × one parameter combination × one job. The atomic unit BDN measures.
+- **Operation** — the logical unit of work being measured. All BDN output columns (Mean, Error, etc.) report time per operation.
+- **Invocation** — a single call to the benchmark method. By default, 1 invocation = 1 operation. With `OperationsPerInvoke=N`, each invocation counts as N operations.
+- **Iteration** — a timed batch of invocations. BDN measures the total time for all invocations in an iteration, then divides by the total operation count to get per-operation time.
+
+## Benchmarks are comparative instruments
+
+A single benchmark number has limited value — it can confirm the order of magnitude of a measurement, but the exact value changes across machines, operating systems, and runtime configurations. Benchmarks produce the most useful information when compared against something. Before writing benchmarks, identify the **comparison axis** for the current task:
+
+- **Approaches (A vs B)**: comparing alternative implementations side-by-side in the same run.
+- **Runtimes**: comparing the same code across .NET versions (e.g., net8.0 vs net9.0).
+- **Package versions**: comparing different versions of a NuGet dependency.
+- **Builds (before/after)**: comparing a saved DLL of the old code against the current source.
+- **Runtime configuration (GC mode, JIT settings)**: understanding how runtime settings affect performance — compared via multiple jobs in a single run.
+- **Scale (N=100 vs N=1000)**: understanding how performance changes as input size grows.
+- **Hardware/OS**: comparing across different machines or operating systems — requires separate runs on each environment.
+- **Earlier measurements**: comparing against measurements recorded previously.
+
+BDN can compare the first six axes side-by-side in a single run, but each requires specific CLI flags or configuration that differ from what you might expect — read [references/comparison-strategies.md](references/comparison-strategies.md) for the correct approach for each strategy before configuring a comparison.
+
+## Use cases and benchmark lifecycle
+
+There are four distinct reasons a developer writes a benchmark, and each one changes how the benchmark should be designed and where it should live:
+
+1. **Coverage suite**: Write benchmarks to maximize coverage of real-world usage patterns so that regressions affecting most users are caught. These benchmarks are permanent — they belong in the project's benchmark suite, follow its conventions (directory structure, base classes, naming), and are checked in.
+
+2. **Issue investigation**: Someone has reported a specific performance problem. Write benchmarks to reproduce and diagnose that specific issue. These benchmarks are task-scoped — they persist across the investigation (reproduce → isolate → verify fix) but are not part of the permanent suite.
+
+3. **Change validation**: A developer has a PR or change and wants to understand its performance characteristics before merging. These benchmarks are task-scoped — they persist across the review cycle but are not checked in.
+
+4. **Development feedback**: A developer is actively working on a task and wants to use benchmarks to evaluate approaches and get information early. These benchmarks are task-scoped and throwaway — they persist across the development session but are deleted when the decision is made.
+
+For use case 1, add to the existing benchmark project following its conventions. For use cases 2–4, create a standalone project in a working directory that persists for the task but is clearly not part of the permanent codebase.
+
+For **coverage suite** benchmarks, design from the perspective of real callers — what code patterns use this API, what inputs they pass, and what performance characteristics matter to them. Each permanent benchmark should justify its maintenance cost through real-world relevance. For **temporary benchmarks**, keep the case count intentional — each additional test case costs wall-clock time (read [Cost awareness](#cost-awareness)).
+
+## Cost awareness
+
+Each benchmark case (one method × one parameter combination × one job) takes **15–25 seconds** with default settings. `[Params]` creates a Cartesian product: two `[Params]` with 3 and 4 values across 5 methods = 60 cases ≈ 20 minutes. Multiple jobs multiply this further. Before running, estimate the total case count and match the job preset to the situation:
+
+| Preset | Per-case time | When to use |
+|--------|--------------|-------------|
+| `--job Dry` | <1s | Validate correctness — confirms compilation and execution without measurement |
+| `--job Short` | 5–8s | Quick measurements during development or investigation |
+| *(default)* | 15–25s | Final measurements for a coverage suite |
+| `--job Medium` | 33–52s | Higher confidence when results matter |
+| `--job Long` | 3–12 min | High statistical confidence |
+
+If benchmark runs take longer than expected, results seem unstable, or you need to tune iteration counts or execution settings, read [references/bdn-internals-and-tuning.md](references/bdn-internals-and-tuning.md) for detailed information about BDN's execution pipeline and configuration options.
+
+## Entry points and configuration
+
+BDN programs use either **`BenchmarkSwitcher`** (provides interactive benchmark selection for humans, parses CLI arguments) or **`BenchmarkRunner`** (runs specified benchmarks directly). Both support CLI flags like `--filter` and `--runtimes`, but only when `args` is passed through — without it, CLI flags are silently ignored. When using `BenchmarkSwitcher`, always pass `--filter` to avoid hanging on an interactive prompt.
+
+BDN behavior is customized through **attributes**, **config objects**, and **CLI flags**.
+
+Read [references/project-setup-and-running.md](references/project-setup-and-running.md) for entry point setup, config object patterns, and CLI flags. If you need to collect data beyond wall-clock time — such as memory allocations, hardware counters, or profiling traces — read [references/diagnosers-and-exporters.md](references/diagnosers-and-exporters.md).
+
+## Running benchmarks
+
+BenchmarkDotNet console output is extremely verbose — hundreds of lines per case showing internal calibration, warmup, and measurement details. Redirect all output to a file to avoid consuming context on verbose iteration output:
+
+```
+dotnet run -c Release -- --filter "*MethodName" --noOverwrite > benchmark.log 2>&1
+```
+
+Each benchmark method can take several minutes. Rather than running all benchmarks at once, use `--filter` to run a subset at a time (e.g. one or two methods per invocation), read the results, then run the next subset. This keeps each invocation short — avoiding session or terminal timeouts — and lets you verify results incrementally. Read [references/project-setup-and-running.md](references/project-setup-and-running.md) for filter syntax, CLI flags, and project setup.
+
+After each run, read the Markdown report (`*-report-github.md`) from the results directory for the summary table. Only read `benchmark.log` if you need to investigate errors or unexpected results.
+
+## Writing new benchmarks
+
+### Step 1: Plan the test cases
+
+Before writing any code, determine:
+- Which use case this benchmark serves (coverage, investigation, change validation, or development feedback).
+- Which comparison axis applies (what will the number be compared against?).
+- What real-world scenarios to benchmark, based on how callers actually use the API.
+
+Each benchmark case should justify its cost. An uncovered scenario is usually more valuable than another parameter combination for one already covered, but when a specific parameter dimension genuinely affects performance characteristics, the depth is warranted.
+
+Decide on the list of test cases. For each test case, think through:
+
+- **How to express variation**: BenchmarkDotNet provides several mechanisms for parameterizing benchmarks — `[Params]` and `[ParamsSource]` for property-level parameters, `[Arguments]` and `[ArgumentsSource]` for method-level arguments, `[ParamsAllValues]` to enumerate all values of a `bool` or enum, and `[GenericTypeArguments]` for varying type parameters on generic benchmark classes. Choose the mechanism that best fits the dimension being varied. Read [references/writing-benchmarks.md](references/writing-benchmarks.md) for the full set of options and correctness patterns.
+- **Where input data comes from** — consider which sources are appropriate (these can be combined):
+  - Hard-coded values — small, fixed values where the exact input matters (e.g., specific strings, known edge-case sizes). Store in fields or `[Params]` to avoid constant folding.
+  - Asset files — static data that is too large or impractical to embed in source code such as binary blobs.
+  - Programmatically generated via `[ParamsSource]`/`[ArgumentsSource]`/`[GlobalSetup]` — when data shape matters more than specific content, or when input must be parameterized by size.
+- **Whether randomness is appropriate**: If using generated data, use seeded randomness for reproducibility. When generating random data, use a large enough sample that the generated distribution is representative (e.g., 4 random values may cluster in a narrow range, while 1000 will better exercise the full distribution).
+
+### Step 2: Implement the benchmarks
+
+For **coverage suite** benchmarks, add to the existing benchmark project and follow its conventions. For **temporary benchmarks** (investigation, change validation, development feedback), create a standalone project — read [references/project-setup-and-running.md](references/project-setup-and-running.md) for project setup and entry point configuration.
+
+**Adding the BenchmarkDotNet package**: In a repository with Central Package Management (`Directory.Packages.props`), use the version already pinned there and add a versionless `<PackageReference>`; otherwise use `dotnet add package BenchmarkDotNet` (no version) — this lets NuGet resolve the latest compatible version. Do NOT manually write a `<PackageReference>` with a version number into the `.csproj`; BDN versions in training data are outdated and may lack support for current .NET runtimes.
+
+Write the benchmark code. Follow the patterns in [references/writing-benchmarks.md](references/writing-benchmarks.md) to avoid common measurement errors — in particular:
+- **Return results** from benchmark methods to prevent dead code elimination
+- **Move initialization to `[GlobalSetup]`** — setup inside the benchmark method is measured; use `[IterationSetup]` only when the benchmark mutates state that must be reset between iterations
+- **Do not add manual loops** — BDN controls invocation count automatically
+- **Mark a baseline** when comparing alternatives — use `[Benchmark(Baseline = true)]` for method-level comparisons or `.AsBaseline()` on a job for multi-job comparisons so results show relative ratios
+- **Store inputs in fields or `[Params]`**, not as literals or `const` values — the JIT can fold constant expressions at compile time, making the benchmark measure a precomputed result instead of the actual computation
+
+### Step 3: Validate and run
+
+Validate before committing to a long run:
+
+1. Run with `--job Dry` first to catch compilation errors and runtime exceptions without spending time on measurement.
+2. Run a single representative case with default settings to verify the output looks correct and the numbers are in the expected range.
+3. Only run the full suite after validation passes.
+
+When iterating on benchmark design, use `--job Short` until confident, then switch to default for final numbers.

package/package/corpus/skills/microbenchmarking/references/bdn-internals-and-tuning.md

@@ -0,0 +1,143 @@
+# BenchmarkDotNet Execution and Configuration
+
+## Contents
+
+- Build and execution
+- Execution stages per case (default job settings)
+- Job preset configurations
+- Run strategies
+- Jobs and mutators
+- Execution-related attributes
+- Unroll factor
+- Execution environment
+
+## Build and execution
+
+BDN generates a standalone project containing the measurement harness, builds it **once per unique build configuration**, and runs each case in its **own process**. Cases are partitioned into separate builds when they differ in factors including runtime, toolchain, platform, JIT, GC settings, build configuration, or MSBuild arguments. All cases sharing the same build configuration share a single build (~5s).
+
+BDN determines the number of invocations per iteration automatically.
+
+`[Params]` creates a full Cartesian product: two `[Params]` properties with 3 and 4 values across 5 methods = 3 × 4 × 5 = 60 cases.
+
+Build overhead (~5s) is paid once regardless of case count. When running cases individually (separate `dotnet run` per case), each pays the build cost, adding ~5s per case.
+
+## Execution stages per case (default job settings)
+
+Each case runs through these stages sequentially. Some stage defaults have changed between BDN versions and may change in future releases — check which version of the BenchmarkDotNet NuGet package is installed when the exact behavior matters.
+
+1. **Process launch + JIT** — The benchmark method is invoked to trigger initial JIT compilation, ensuring the method is compiled before the pilot stage calibrates invocation count. This stage's time is not included in results.
+2. **Pilot** — Determines the invocation count per iteration. Fast methods get many invocations per iteration (targeting ~500ms per iteration). Methods slower than ~333ms/op (where two invocations would exceed the target) are fixed at one invocation per iteration.
+3. **Warmup** — Adaptive, typically 6–10 iterations for stable benchmarks (those with low variance). During warmup the runtime may further optimize the method (e.g., tiered JIT recompilation).
+4. **Actual** — Adaptive, typically 15 iterations for stable benchmarks. Stops when the confidence interval is sufficiently narrow (controlled by `MaxRelativeError`, default 2%).
+
+**Note (BDN before 0.16.0):** an additional **Overhead** stage ran between Pilot and Warmup by default. It measured the internal invocation loop's own cost and subtracted it from results. Newer releases leave this stage off by default, and the API to re-enable it (`[EvaluateOverhead]` / `.WithEvaluateOverhead()`) is marked obsolete.
+
+## Job preset configurations
+
+| Preset | Per-case time | Warmup | Target iterations | Launch count | Strategy |
+|--------|--------------|--------|-------------------|-------------|----------|
+| `Dry` | <1s | 0 | 1 | 1 | ColdStart |
+| `Short` | 5–8s | 3 | 3 | 1 | Throughput |
+| Default | 15–25s | 6–50 (adaptive) | 15–100 (adaptive) | 1 | Throughput |
+| `Medium` | 33–52s | 10 | 15 | 2 | Throughput |
+| `Long` | 3–12 min | 15 | 100 | 3 | Throughput |
+
+The `Dry` preset reuses the ColdStart strategy for minimal execution; its purpose is validation (confirming the benchmark compiles and runs), not first-call measurement.
+
+## Run strategies
+
+The `RunStrategy` controls how BDN calibrates and measures:
+
+- **Throughput** (default) — measures steady-state performance. BDN auto-calibrates invocation count via the pilot stage, runs warmup, and adapts iteration count. Use for microbenchmarks.
+- **ColdStart** — single invocation per iteration, skips all preliminary stages (JIT pre-trigger, pilot, warmup, overhead). The first measured invocation includes JIT compilation time. Use for startup time evaluation.
+- **Monitoring** — single invocation per iteration. Skips JIT pre-trigger, pilot, and overhead stages (like ColdStart), but includes the warmup stage (default 0 iterations, configurable). Use for macrobenchmarks with high variance that don't reach a steady state.
+
+## Jobs and mutators
+
+A **job** defines a complete set of run conditions — runtime, iteration counts, launch count, strategy, etc. A benchmark class can have **multiple jobs**, and BDN will run all benchmark methods under each job separately. For example, `[SimpleJob(RuntimeMoniker.Net80)]` and `[SimpleJob(RuntimeMoniker.Net90)]` on the same class runs every method twice — once per runtime.
+
+A **mutator** is not a standalone job. It is a partial override that gets applied to every existing job. This distinction matters when a class has multiple jobs:
+
+```csharp
+[SimpleJob(RuntimeMoniker.Net80)]      // Job A
+[SimpleJob(RuntimeMoniker.Net90)]      // Job B
+[WarmupCount(3)]                        // Mutator — applied to BOTH Job A and Job B
+public class MyBenchmarks { ... }
+```
+
+Without the mutator distinction, `[WarmupCount(3)]` would be a third job with just a warmup count override and default everything else — probably not what was intended.
+
+Jobs are resolved by collecting all standard jobs (from `[SimpleJob]`, etc.; `Job.Default` if none), then applying all mutators to every job. CLI flags like `--warmupCount 3` act as mutators — they override that setting on all jobs defined in source code. `--job Short` adds a standard job.
+
+## Execution-related attributes
+
+**Class or method level (mutator attributes):**
+These override specific job settings. When applied to a method, they affect only that method's cases. When applied to a class, they affect all methods in the class.
+
+| Attribute | Default | Config (on Job) | CLI flag |
+|-----------|---------|----------------|----------|
+| `[WarmupCount(N)]` | null (adaptive) | `.WithWarmupCount(N)` | `--warmupCount N` |
+| `[IterationCount(N)]` | null (adaptive) | `.WithIterationCount(N)` | `--iterationCount N` |
+| `[MinWarmupCount(N)]` | 6 | `.WithMinWarmupCount(N)` | `--minWarmupCount N` |
+| `[MaxWarmupCount(N)]` | 50 | `.WithMaxWarmupCount(N)` | `--maxWarmupCount N` |
+| `[MinIterationCount(N)]` | 15 | `.WithMinIterationCount(N)` | `--minIterationCount N` |
+| `[MaxIterationCount(N)]` | 100 | `.WithMaxIterationCount(N)` | `--maxIterationCount N` |
+| `[InvocationCount(N)]` | null (auto pilot) | `.WithInvocationCount(N)` | `--invocationCount N` |
+| `[IterationTime(ms)]` | 500ms | `.WithIterationTime(TimeInterval.FromMilliseconds(ms))` | `--iterationTime ms` |
+| `[ProcessCount(N)]` | 1 | `.WithLaunchCount(N)` | `--launchCount N` |
+| `[MaxRelativeError(pct)]` | 0.02 (= 2%) | `.WithMaxRelativeError(pct)` | — |
+
+Note: `[ProcessCount]` and `--launchCount` control the same setting (how many separate processes BDN launches per case). The naming differs between API surfaces.
+
+The `[RunOncePerIteration]` attribute (also available as `.RunOncePerIteration()` or `--runOncePerIteration`) is shorthand for setting `InvocationCount=1` and `UnrollFactor=1`.
+
+**Note (older BDN releases):** `[EvaluateOverhead(bool)]` / `.WithEvaluateOverhead(bool)` / `--noOverheadEvaluation` controlled overhead subtraction. These are obsolete because overhead evaluation is disabled by default.
+
+**Class or assembly level:**
+- Job attributes: `[SimpleJob(...)]`, `[DryJob]`, `[ShortRunJob]`, `[MediumRunJob]`, `[LongRunJob]`, `[VeryLongRunJob]`. AllowMultiple — can stack to run the same benchmarks under different job configurations.
+
+## Unroll factor
+
+BDN generates a measurement method that contains the benchmark code repeated `UnrollFactor` times (default 16). This reduces measurement loop overhead relative to the actual work.
+
+UnrollFactor is distinct from `OperationsPerInvoke`: UnrollFactor is an engine optimization, while `OperationsPerInvoke` is a user declaration that each invocation performs multiple logical operations.
+
+When `[IterationSetup]` or `[IterationCleanup]` is used, BDN defaults to `UnrollFactor=1` and `InvocationCount=1` so that setup/cleanup runs before and after each single invocation of the benchmark method. Explicitly setting `InvocationCount` or `UnrollFactor` overrides this behavior.
+
+| Attribute | Config API | CLI flag |
+|-----------|------------|----------|
+| `[UnrollFactor(N)]` | `.WithUnrollFactor(N)` | `--unrollFactor N` |
+
+## Execution environment
+
+The settings below configure the runtime environment the benchmark process runs in. They are orthogonal to the execution pipeline above — they don't change how many iterations run or how the engine measures, they change *what* is being measured.
+
+| Purpose | Attribute | Config API | CLI flag |
+|---------|-----------|------------|----------|
+| Target runtimes | `[SimpleJob(RuntimeMoniker.Net80)]` | `Job.Default.WithRuntime(CoreRuntime.Core80)` | `--runtimes net8.0 net9.0` (first is baseline) |
+| Environment variables | — | `job.WithEnvironmentVariable("key", "value")` | `--envVars KEY:VALUE` |
+
+Multiple runtimes multiply the total case count and each requires a separate build.
+
+### GC configuration
+
+| Setting | Attribute | Config API |
+|---------|-----------|------------|
+| Server GC | `[GcServer(true)]` | `.WithGcServer(true)` |
+| Concurrent GC | `[GcConcurrent(bool)]` | `.WithGcConcurrent(bool)` |
+
+### MemoryRandomization
+
+| Surface | Usage |
+|---------|-------|
+| Attribute | `[MemoryRandomization]` |
+| Config | `.WithMemoryRandomization(true)` |
+| CLI | `--memoryRandomization` |
+
+The `[MemoryRandomization]` attribute causes BDN to allocate random-sized memory between iterations, shifting heap object alignment. This reveals whether benchmark results depend on memory layout (cache line alignment, GC heap position) rather than algorithmic performance.
+
+When enabled, BDN:
+- Allocates random-sized Gen0 and LOH objects between iterations
+- Uses `stackalloc` with random size to shift stack alignment
+- Re-runs `[GlobalCleanup]` and `[GlobalSetup]` after every iteration
+- Disables outlier removal (since multimodal results may be genuine)

package/package/corpus/skills/microbenchmarking/references/comparison-strategies.md

@@ -0,0 +1,200 @@
+# Comparing benchmark results
+
+## Contents
+
+- Strategy 1: Side-by-side methods (preferred)
+- Strategy 2: Comparing runtimes
+- Strategy 3: Comparing NuGet package versions
+- Strategy 4: Comparing a saved build against current source (DLL reference)
+- Strategy 5: Comparing runtime configurations
+- Strategy 6: Comparing across input scale
+- How it works
+- Apples-to-apples comparison
+- Mann-Whitney equivalence test
+
+BenchmarkDotNet can compare multiple implementations or versions in a single run, producing a side-by-side table with ratio columns and statistical analysis. This is generally preferable to separate runs because it controls for environmental variance.
+
+## Strategy 1: Side-by-side methods (preferred)
+
+When both the old and new implementations can coexist in the same compilation, define separate benchmark methods:
+
+```csharp
+public class SortBenchmark
+{
+    private int[] _data;
+
+    [GlobalSetup]
+    public void Setup() => _data = Enumerable.Range(0, 1000).Reverse().ToArray();
+
+    [Benchmark(Baseline = true)]
+    public void BubbleSort() => Sorting.BubbleSort((int[])_data.Clone());
+
+    [Benchmark]
+    public void QuickSort() => Sorting.QuickSort((int[])_data.Clone());
+}
+```
+
+BDN displays a Ratio column showing each method's mean relative to the baseline. This is the simplest approach and should be preferred whenever both implementations are available at compile time.
+
+## Strategy 2: Comparing runtimes
+
+To compare the same benchmark across different .NET runtime versions, use `--runtimes` on the command line or configure jobs. The `.csproj` must use `<TargetFrameworks>net8.0;net9.0</TargetFrameworks>` (plural) listing all targets:
+
+```
+dotnet run -c Release --framework net9.0 -- --runtimes net8.0 net9.0
+```
+
+```csharp
+var config = DefaultConfig.Instance
+    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core80).AsBaseline())
+    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core90));
+```
+
+Each runtime becomes a separate job. BDN builds separate executables per runtime and shows them side-by-side with ratio columns.
+
+## Strategy 3: Comparing NuGet package versions
+
+To compare different versions of a NuGet dependency, use MSBuild properties to control the package version per job. Each job gets a separate build with the specified version.
+
+**.csproj** — use an MSBuild property for the version:
+```xml
+<PropertyGroup>
+  <MyLibVersion Condition="'$(MyLibVersion)' == ''">2.0.0</MyLibVersion>
+</PropertyGroup>
+<ItemGroup>
+  <PackageReference Include="MyLibrary" Version="$(MyLibVersion)" />
+</ItemGroup>
+```
+
+**Config** — create a job per version:
+```csharp
+var config = DefaultConfig.Instance
+    .AddJob(Job.Default
+        .WithMsBuildArguments("/p:MyLibVersion=1.0.0")
+        .WithId("v1.0.0")
+        .AsBaseline())
+    .AddJob(Job.Default
+        .WithMsBuildArguments("/p:MyLibVersion=2.0.0")
+        .WithId("v2.0.0"));
+```
+
+All versions must be source-compatible with the benchmark code (same namespace, same method signatures). BDN builds a separate exe per job because `WithArguments` changes the build partition.
+
+This works with packages from any NuGet source — public registries, private feeds, or local folders created with `dotnet pack`.
+
+## Strategy 4: Comparing a saved build against current source (DLL reference)
+
+When the code is not published as a NuGet package but you want to compare before/after a code change, you can save the build output of the baseline version and reference it directly.
+
+**Step 1** — build and save the baseline:
+```
+dotnet build ../MyLib -c Release -o ./saved-baseline
+```
+
+**Step 2** — make your changes to MyLib.
+
+**Step 3** — set up the benchmark .csproj with conditional references:
+```xml
+<ItemGroup>
+  <ProjectReference Condition="'$(BaselineDll)' == ''" Include="..\MyLib\MyLib.csproj" />
+  <Reference Condition="'$(BaselineDll)' != ''" Include="MyLib">
+    <HintPath>$(BaselineDll)</HintPath>
+  </Reference>
+</ItemGroup>
+```
+
+**Step 4** — configure two jobs:
+```csharp
+var config = DefaultConfig.Instance
+    .AddJob(Job.Default
+        .WithMsBuildArguments("/p:BaselineDll=../saved-baseline/MyLib.dll")
+        .WithId("baseline")
+        .AsBaseline())
+    .AddJob(Job.Default
+        .WithId("current"));
+```
+
+The `baseline` job builds with a direct DLL reference to the saved output. The `current` job builds with the `ProjectReference` to the current source. BDN produces a side-by-side table with ratio columns.
+
+This approach works for any class library. The namespace and public API must be the same between the two versions.
+
+## Strategy 5: Comparing runtime configurations
+
+To compare how different runtime settings (GC mode, JIT options, PGO) affect performance, define multiple jobs with different environment settings. This works with any `Job` setting — GC mode, environment variables, platform, JIT settings:
+
+```csharp
+// GC mode comparison
+var config = DefaultConfig.Instance
+    .AddJob(Job.Default.WithGcServer(false).WithId("Workstation GC").AsBaseline())
+    .AddJob(Job.Default.WithGcServer(true).WithId("Server GC"));
+
+// JIT setting comparison
+var config = DefaultConfig.Instance
+    .AddJob(Job.Default.WithId("Default").AsBaseline())
+    .AddJob(Job.Default.WithEnvironmentVariable("DOTNET_TieredCompilation", "0").WithId("No Tiered"));
+```
+
+## Strategy 6: Comparing across input scale
+
+Use `[Params]` on a size property to understand how performance changes with input size. BDN runs each parameter value as a separate case. Geometrically spaced values (e.g., 10×, 100×) help reveal scaling behavior:
+
+```csharp
+[Params(100, 1_000, 10_000)]
+public int N;
+
+private int[] _data;
+
+[GlobalSetup]
+public void Setup()
+{
+    var rng = new Random(42);
+    _data = Enumerable.Range(0, N).OrderBy(_ => rng.Next()).ToArray();
+}
+
+[Benchmark]
+public void Sort() => Array.Sort((int[])_data.Clone());
+```
+
+## How it works
+
+All multi-version strategies rely on the same BDN mechanism:
+
+1. **Build partitioning** — different MSBuild arguments, runtimes, or platform → separate compiled executables. Each case runs in its own process.
+2. **Ratio calculation** — when a baseline is marked, BDN adds `Ratio` and `RatioSD` columns. `Ratio` is the mean of current/baseline per-operation time ratios. A `Ratio` of `0.85` means ~15% faster.
+
+### Marking a baseline
+
+For side-by-side methods (Strategy 1), mark one method with `[Benchmark(Baseline = true)]`. This compares methods within each job.
+
+For multi-job strategies (Strategies 2–5), mark one job with `.AsBaseline()` in the config. This compares jobs against each other. For `--runtimes`, the first runtime listed is the baseline.
+
+These are separate mechanisms — using the wrong one produces no ratio columns.
+
+Without a baseline, BDN shows absolute numbers only — no ratio columns.
+
+## Apples-to-apples comparison
+
+When comparing across jobs (Strategies 2–5), the default behavior runs each job's pilot stage independently, which may choose different invocation counts per job for the same benchmark case. This creates asymmetric measurement conditions that can bias the comparison.
+
+The `--apples` flag forces symmetric measurement by running in two phases:
+
+1. **Pilot phase** — BDN runs the baseline job once per benchmark case (method × parameter combination) to determine the invocation count for that case. Different cases get different invocation counts based on their speed. This produces a first results table in the log showing only the baseline with `Ratio = 1.00`.
+2. **Measurement phase** — for each benchmark case, ALL jobs run with that case's fixed invocation count, ensuring every job measures the same number of operations per iteration. The final comparison table includes `Ratio` and `InvocationCount` columns and is written to `BenchmarkDotNet.Artifacts/`.
+
+Outlier removal is disabled in this mode since the controlled setup makes outliers more meaningful.
+
+```
+dotnet run -c Release --framework net9.0 -- --runtimes net8.0 net9.0 --apples --job short
+```
+
+Because the pilot runs only once per benchmark case instead of once per job, `--apples` is faster than the default when there are many benchmark cases. The savings grow with the number of methods and parameter combinations.
+
+Requirements:
+- At least two jobs, with exactly one marked as baseline (via `.AsBaseline()` on the job, not `[Benchmark(Baseline = true)]` on the method). When using `--runtimes`, the first runtime listed is automatically treated as the baseline.
+- An explicit iteration count, such as by using `--job short` (which sets `IterationCount=3`) or `--iterationCount N`. Adaptive iteration count is disabled in this mode.
+
+This does not apply to Strategy 1 (side-by-side methods) or Strategy 6 (input scale) because those use a single job — `--apples` requires multiple job objects.
+
+## Mann-Whitney equivalence test
+
+BDN can add a `Faster` / `Same` / `Slower` column based on a Mann-Whitney equivalence test. Add it via config: `.AddColumn(StatisticalTestColumn.Create(threshold: "5%"))`. The `threshold` is the equivalence margin — results within that percentage are reported as `Same`.