duel-retained 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 731c988c83f260d7d2b1f72f8bcaea04a3c2353d95eef6018acbf5107b6bc8f6
+  data.tar.gz: 0e482b0d4bbff58e2c459807da6378ce62e5874d38d6db7ce7e943ff2139448e
+SHA512:
+  metadata.gz: 4de0db7edeaadcb3bc5848249b87e54f7746c82be8153eb84ebdb1022c0bac4af822b6d638756e930a376444d9aafdcba68e097f887e85f7f10a1243ba6ef943
+  data.tar.gz: 18f36afd00357c1d3250f5fd1dabc40efd8cba87aa6591c9c455d958ccd2b3b47ff01c64da61b369c1c8d6fb9c0f3631bffbc546da09893e17971c119202d574

data/CHANGELOG.md

@@ -0,0 +1,36 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-10-24
+
+The first release. duel-retained is a Duel plugin, open source under
+the MIT license. It needs Duel 0.1 and Ruby 3.4 or later, and measures
+on CRuby alone (on JRuby and TruffleRuby it leaves itself out).
+
+### Added
+
+- `c.use :retained`: the `retained_objects` metric (CRuby), the
+  objects each benchmark's iterations leave alive after a full garbage
+  collection, with Duel's allowance of 16 a sample (for what's still on
+  the machine stack when it stops) and tolerance of one per two
+  iterations, both changed with `c.measure("retained_objects", ...)`.
+  It measures straight through `untimed` blocks. Where a sample retained
+  more than it's allowed (as `c.measure` leaves its allowance and
+  tolerance), the regression shows what, an iteration: the
+  objects' classes and where they were made, the three most (from
+  allocation tracing during the sample).
+- `retained_objects` is in objects (`unit: %w[object objects]`), so
+  reports read "1 object → 2 objects an iteration" (the summary line,
+  "retained_objects 3→5531", leaves out the unit its name ends in).
+- Mutation tested: mutant kills every change to the plugin's code, in
+  CI.
+- RBS signatures in sig/, checked with Steep.
+
+[Unreleased]: https://github.com/sferik/duel-ruby/compare/duel-retained-v0.1.0...HEAD
+[0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/duel-retained-v0.1.0

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Erik Berlin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,155 @@
+# duel-retained
+
+[![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
+[![duel-retained](https://github.com/sferik/duel-ruby/actions/workflows/plugin-retained.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-retained.yml)
+[![Gem Version](https://badge.fury.io/rb/duel-retained.svg)](https://rubygems.org/gems/duel-retained)
+
+A [Duel](https://github.com/sferik/duel-ruby) plugin: every benchmark is
+judged on what its iterations leave alive too, the Ruby objects still
+there after a full garbage collection.
+
+| Metric | What it is | A regression |
+|---|---|---|
+| `retained_objects` | the objects a sample's iterations left alive, beyond a few a sample (CRuby) | one more per two iterations |
+
+So a leak, a cache that grows with every call, or memoization keyed by
+input that never repeats, fails, though it allocates no more and takes
+no longer than before. Allocations count what a benchmark makes, and CPU
+time what it costs now, but neither sees what it keeps. Duel's `peak_memory`
+does, coarsely: resident memory, in fresh workers, judged by the
+megabyte. This counts the objects, exactly.
+
+## Installation
+
+```ruby
+# Gemfile
+group :development do
+  gem "duel-retained"
+end
+```
+
+> [!NOTE]
+> It measures on CRuby. On JRuby and TruffleRuby, whose garbage collectors
+> can't be held off, or their live objects counted, it leaves itself out:
+> benchmarks run, and are judged, as without it.
+
+## Usage
+
+```ruby
+# .duel.rb
+Duel.configure { |c| c.use :retained }
+```
+
+Then a benchmark whose code keeps what it's given fails:
+
+```ruby
+# lib/my_formatter.rb, the candidate's: memoized, and never forgets
+FORMATTED = {}
+def self.format(value) = FORMATTED[value] ||= value.round(2).to_s.rjust(10)
+
+# duel/formatter_duel.rb
+duel("format: a number") { MyFormatter.format(rand) }
+```
+
+```
+! format: a number    0.4s n=5528   ...   retained_objects 3→5531  allocations 11056
+REGRESSION format: a number: worse retained_objects in every measurement
+  retained_objects: 0.0005427 objects → 1.001 objects an iteration
+  retained_objects (candidate): an iteration: 1 String (lib/my_formatter.rb:3)
+```
+
+What it retained beyond what's let through is shown by class, and where
+it was made (the three most): see [What it retained](#what-it-retained).
+
+Install it after any other plugin's metrics: those added after it are
+read inside its measurement, and what they keep of their readings counts
+toward its allowance.
+
+A sample may leave 16 objects alive before any counts against it (its
+allowance), and beyond that it's one more per two iterations (its
+tolerance) that's a regression: see [below](#what-it-cant-help-counting).
+Duel's `c.measure` changes either, or leaves the metric out:
+
+```ruby
+Duel.configure do |c|
+  c.use :retained
+  c.measure("retained_objects", allowance: 32, tolerance: 1) # or skip: true
+end
+```
+
+## How it measures
+
+Duel holds the garbage collector off during a sample, so that when it
+runs isn't up to chance. `GC.start` still collects (CRuby leaves the
+collector disabled after): before each sample it counts in, the plugin
+collects all the garbage there is, a full collection twice (what a
+finalizer, or a dynamic symbol, keeps alive goes in the second), and
+counts the live objects (`GC.stat(:heap_live_slots)`). After the sample,
+it does it again. What's left is what the sample's iterations kept.
+
+That's outside the built-in metrics, so none of it is in a benchmark's
+allocations, instructions or CPU time. And only in the samples Duel
+counts in (three of each version's a benchmark, and as many again for
+each time it measures one again before failing), not the timed ones. A
+full collection takes about half a millisecond with 30,000 live objects,
+and some 15 with a million: four a sample, so a fraction of a second a
+benchmark, for even a large application.
+
+What's measured is the count itself, every object more after the sample
+than before it (or fewer, for one that let go of more than it kept), and
+Duel judges it: a candidate that leaves more alive than the baseline, by
+more than the allowance a sample and the tolerance an iteration, fails.
+
+### What it can't help counting
+
+Not every object a sample leaves alive is the benchmark's. Duel's own
+record of the metrics inside this one is. And so is what's still on the
+machine stack when the sample stops: the last iteration's result (or the
+block's, given n) is no longer referenced, but CRuby scans the stack
+conservatively, so a stale reference to it there, or to anything it
+holds, keeps it alive through the collection. Nothing Duel or the plugin
+can do from Ruby clears it: running the iterations on a stack of their
+own (a Fiber) would, but it would change what `Thread.current[]` means to
+the benchmark. They're the same few from sample to sample, but not
+always to the object, so a sample is allowed 16 before any is counted.
+On top of that, it's one more per two iterations, not one more, that's a
+regression. So for a benchmark that runs thousands of iterations a
+sample, as most do, one object kept an iteration fails. For one that
+runs a few (a slow one, or one with `only: :count`, which runs four),
+it takes more an iteration.
+
+What a benchmark makes the first time and keeps (a constant it
+memoizes, a method cache) isn't a leak: Duel runs each benchmark many
+times before it counts, and none of that is counted.
+
+Benchmarks with `only: :timing` aren't counted at all, so they aren't
+judged on this either.
+
+### Untimed blocks
+
+Duel pauses every metric around an `untimed` block, and this one would
+collect the garbage, twice, on each: in a benchmark with an `untimed`
+block in every iteration, that's thousands of collections a sample. So
+it doesn't pause: it measures the whole sample, and what an `untimed`
+block makes counts only if it's still alive when the sample is over.
+What it usually makes, the input a benchmark works on, isn't.
+
+## What it retained
+
+Where a sample retained more than Duel lets through (its allowance, and
+its tolerance for its iterations, as `c.measure` leaves them), the
+plugin says what, under the regression: the objects' classes, and where
+they were made, the three most, an iteration (`1 String
+(lib/my_formatter.rb:3)`), of the sample whose count was judged, on
+each version that did. Files are from the working directory, the
+project's, so the baseline's show their whole path, in its checkout
+next to the project. Objects Duel makes for itself aren't shown.
+
+It knows from Ruby's allocation tracing (`objspace`'s
+`ObjectSpace.trace_object_allocations`), on during each sample it counts
+in: that makes each allocation some four times slower, in samples that
+aren't timed. Telling what's left apart walks every live object (some
+twenty times as long as a full collection), so it's done only for a
+sample that retained more than Duel lets through. A benchmark that
+traces allocations itself, or clears the trace, shares the one Ruby
+has.

data/lib/duel/retained/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Duel
+  module Retained
+    VERSION = "0.1.0"
+  end
+end

data/lib/duel/retained.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+require "duel"
+require "objspace"
+require_relative "retained/version"
+
+module Duel
+  # What each benchmark leaves alive: the Ruby objects still there after
+  # a full garbage collection that weren't before it (CRuby).
+  #
+  # - retained_objects: the objects a sample's iterations left alive
+  #   (counted; beyond ALLOWANCE a sample, one more per two iterations is
+  #   a regression)
+  #
+  #   # .duel.rb
+  #   Duel.configure { |c| c.use :retained }
+  #
+  # So a leak, a cache that grows with every call, or memoization keyed
+  # by input that never repeats, fails, though it allocates no more and
+  # takes no longer than before.
+  #
+  # Duel holds the garbage collector off during a sample, but GC.start
+  # still collects: the metric collects, twice (what a finalizer or a
+  # dynamic symbol keeps alive goes in the second), and counts the live
+  # objects before the sample and again after it. It's outside the
+  # built-in metrics, so none of that is in allocations or CPU time, and
+  # only in the samples Duel counts in (a few a benchmark), not the timed
+  # ones. When a sample retained more than Duel lets through, the metric
+  # says what (Objects#details), by class and where it was made, under
+  # the regression: "an iteration: 1 String (lib/my_cache.rb:12)".
+  module Retained
+    Duel.plugin(:retained, self)
+
+    # The retained objects a sample is allowed (Duel's allowance) before
+    # any is counted against it: Duel's own record of the metrics inside
+    # this one, and what's still on the machine stack when the sample
+    # stops (the last iteration's result, or the block's, given n),
+    # which are the same few from sample to sample, but not always to the
+    # object.
+    ALLOWANCE = 16
+
+    # Retained objects per iteration that still aren't a regression.
+    TOLERANCE = 0.5
+
+    # The kinds of object (by class, and where they were made) a sample
+    # that retained more than its allowance shows, most first.
+    SHOWN = 3
+
+    # Configures Duel for the plugin (c.use :retained): the metric
+    # (CRuby only: elsewhere the garbage collector can't be held off or
+    # its live objects counted). Returns the metric.
+    def self.configure(config) = config.metric(Objects)
+
+    # Collects all the garbage there is: a full collection (GC.start's
+    # own: a full mark, and an immediate sweep), twice (the first leaves
+    # what has a finalizer to run, and a dynamic symbol's string, for the
+    # second). It collects though the garbage collector is disabled, and
+    # leaves it as it was.
+    def self.collect = 2.times { GC.start }
+
+    # The objects alive after collecting the garbage.
+    def self.live
+      collect
+      GC.stat(:heap_live_slots)
+    end
+
+    # Where Duel's own files are: what it makes during a sample (its
+    # record of the metrics inside this one) is in the allowance, and
+    # isn't shown.
+    # (Duel's metric.rb, written in Ruby: it has a source location.)
+    metric_rb, = Metric.instance_method(:details).source_location #: [String, Integer]
+    DUEL = File.join(File.dirname(metric_rb), "")
+
+    # What each of the objects that a sample made (and Objects traced)
+    # and left alive is, and where it was made: {[class, "file:line"] =>
+    # how many}, most first, but Duel's own. A walk of every live object.
+    def self.traced
+      counts = Hash.new(0) #: Hash[[String, String], Integer]
+      class_of = Kernel.instance_method(:class)
+      here = File.join(Dir.pwd, "")
+      ObjectSpace.each_object do |object|
+        next unless ObjectSpace.allocation_generation(object)
+
+        file = ObjectSpace.allocation_sourcefile(object)
+        next if file.to_s.start_with?(DUEL)
+
+        klass = class_of.bind_call(object)
+        counts[[klass.name || klass.inspect, site(file, ObjectSpace.allocation_sourceline(object), here)]] += 1
+      end
+      counts.sort_by { |(klass, site), count| [-count, klass, site] }.to_h
+    end
+
+    # Where an object was made, from its +file+ and +line+ (nil where it
+    # wasn't made in Ruby code): the file from +here+ (a directory, with
+    # a trailing slash) where it's under it.
+    def self.site(file, line, here) = file ? "#{file.delete_prefix(here)}:#{line}" : "?"
+
+    # The metric: the objects alive after the sample less those before
+    # it (fewer, when the sample let go of more than it kept). Duel takes
+    # the allowance and the tolerance into account when it compares them.
+    class Objects < Metric
+      def self.measures = {"retained_objects" => {unit: %w[object objects], allowance: ALLOWANCE, tolerance: TOLERANCE}}
+
+      # mutant:disable -- mutant runs on CRuby, where this is always true
+      def self.available? = RUBY_ENGINE == "ruby"
+
+      # It traces where objects are made (ObjectSpace's allocation
+      # tracing) during the sample, for #details: what it retained, it
+      # can tell apart from the rest.
+      def start_sample(sample)
+        @n = sample.n
+        ObjectSpace.trace_object_allocations_clear
+        ObjectSpace.trace_object_allocations_start
+        @live = Retained.live
+      end
+
+      def stop_sample
+        @retained = Retained.live - @live
+      ensure
+        ObjectSpace.trace_object_allocations_stop
+      end
+
+      # It goes on through an untimed block: stopping and starting would
+      # collect the garbage, twice, on each, for every untimed block (an
+      # iteration's, maybe). So it measures the whole sample, and what an
+      # untimed block made (the input the benchmark works on, usually)
+      # counts only if it's still alive when the sample is over.
+      def pause_sample
+      end
+
+      def resume_sample(_sample)
+      end
+
+      # What the sample retained, an iteration, by class and where it was
+      # made (the SHOWN most), when it retained more than Duel lets through
+      # (its allowance a sample, and its tolerance an iteration, as
+      # configured): only then is it worth walking every live object for
+      # (Retained.traced). Nothing for one that retained less. Duel asks
+      # once, after the sample: the trace is cleared either way.
+      def details
+        return if within?(@retained, @n)
+
+        kinds = Retained.traced.first(SHOWN).map { |(klass, site), count| format("%.4g %s (%s)", count.fdiv(@n), klass, site) }
+        "an iteration: #{kinds.join(", ")}" unless kinds.empty? # (none it can see: internal objects, or a trace cleared)
+      ensure
+        ObjectSpace.trace_object_allocations_clear
+      end
+    end
+  end
+end

data/sig/duel/retained.rbs

@@ -0,0 +1,62 @@
+module Duel
+  # What each benchmark leaves alive: the Ruby objects still there after
+  # a full garbage collection that weren't before it (retained_objects).
+  module Retained
+    VERSION: String
+
+    # The retained objects a sample is allowed (Duel's allowance) before
+    # any is counted against it.
+    ALLOWANCE: Integer
+
+    # Retained objects per iteration that still aren't a regression.
+    TOLERANCE: Float
+
+    # The kinds of object a sample that retained more than its allowance
+    # shows, most first.
+    SHOWN: Integer
+
+    # Configures Duel for the plugin (c.use :retained): the metric (CRuby
+    # only). Returns it.
+    def self.configure: (Config config) -> singleton(Metric)
+
+    # Collects all the garbage there is (a full collection, twice).
+    def self.collect: () -> void
+
+    # The objects alive after collecting the garbage.
+    def self.live: () -> Integer
+
+    # What each of the objects a sample made and left alive is, and where
+    # it was made: {[class, "file:line"] => how many}, most first.
+    def self.traced: () -> Hash[[String, String], Integer]
+
+    # Where Duel's own files are: what it makes isn't shown.
+    DUEL: String
+
+    # Where an object was made, from its +file+ and +line+ (nil where it
+    # wasn't made in Ruby code).
+    def self.site: (String? file, Integer? line, String here) -> String
+
+    # The metric: the objects alive after the sample less those before
+    # it.
+    class Objects < Metric
+      @live: Integer
+      @n: Integer
+      @retained: Integer
+
+      # It traces where objects are made during the sample, for #details.
+      def start_sample: (Sample sample) -> void
+
+      def stop_sample: () -> Integer
+
+      # It goes on through an untimed block: it measures the whole sample.
+      def pause_sample: () -> nil
+
+      def resume_sample: (Sample _sample) -> nil
+
+      # What the sample retained, an iteration, by class and where it was
+      # made, when it retained more than its allowance and tolerance let
+      # through.
+      def details: () -> String?
+    end
+  end
+end

metadata

@@ -0,0 +1,67 @@
+--- !ruby/object:Gem::Specification
+name: duel-retained
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Erik Berlin
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: duel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+description: 'A Duel plugin: every benchmark is judged on the Ruby objects its iterations
+  leave alive after a full garbage collection, so a leak, or a cache that grows with
+  every call, fails.'
+email:
+- sferik@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/duel/retained.rb
+- lib/duel/retained/version.rb
+- sig/duel/retained.rbs
+homepage: https://github.com/sferik/duel-ruby/tree/main/plugins/retained
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/sferik/duel-ruby/tree/main/plugins/retained
+  changelog_uri: https://github.com/sferik/duel-ruby/blob/main/plugins/retained/CHANGELOG.md
+  bug_tracker_uri: https://github.com/sferik/duel-ruby/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.4'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.23
+specification_version: 4
+summary: 'What each Duel benchmark leaves alive: objects that outlive the garbage
+  collector'
+test_files: []