serialbench 0.1.1 → 0.1.2

data/README.adoc

@@ -13,7 +13,7 @@ Serialbench is a comprehensive benchmarking suite that evaluates the performance
 
 **Key Metrics**: Parsing speed, generation speed, memory usage, streaming capabilities, and feature completeness
 
-**Docker Support**: Multi-Ruby version benchmarking with automated result aggregation and GitHub Pages generation
+**Multi-Environment Support**: Docker and ASDF-based multi-Ruby version benchmarking with automated result aggregation and HTML site generation
 
 == Supported serialization libraries
 
@@ -80,202 +80,49 @@ Serialbench is a comprehensive benchmarking suite that evaluates the performance
 | https://github.com/emancu/toml-rb[TOML-RB]
 | v2.2.0
 | Pure Ruby TOML parser
+
+| TOML
+| https://github.com/fbernier/tomlrb[tomlrb]
+| v2.0.3
+| A Racc based TOML Ruby parser (Only supports parsing, no support for dumping/writing.)
+
 |===
 
-== Benchmark report JSON format
 
-=== Single Ruby version results
+== Data formats and schema
 
-The benchmark results are saved as JSON files with the following structure:
+Serialbench generates structured YAML output for benchmark results, with
+different formats for single-environment and multi-environment runs.
 
-[source,json]
-----
-{
-  "environment": {
-    "ruby_version": "3.3.8",
-    "ruby_platform": "aarch64-linux",
-    "serializer_versions": {
-      "rexml": "3.4.1",
-      "ox": "2.14.23",
-      "nokogiri": "1.18.8",
-      "oga": "3.4",
-      "libxml": "4.1.2",
-      "json": "2.12.2",
-      "oj": "3.16.11",
-      "yajl": "1.4.3",
-      "psych": "5.1.2",
-      "syck": "1.5.1.1",
-      "toml-rb": "2.2.0",
-      "tomlib": "0.7.3"
-    },
-    "timestamp": "2025-06-07T09:00:25+00:00"
-  },
-  "parsing": {
-    "small": {
-      "xml": {
-        "rexml": {
-          "time_per_iterations": 0.002514416000053643,
-          "time_per_iteration": 0.00012572080000268216,
-          "iterations_per_second": 7954.133285650949,
-          "iterations_count": 20
-        },
-        "ox": {
-          "time_per_iterations": 0.00005258399994545471,
-          "time_per_iteration": 0.0000026291999972727353,
-          "iterations_per_second": 380343.83121759404,
-          "iterations_count": 20
-        }
-      },
-      "json": {
-        "json": {
-          "time_per_iterations": 0.000029707999942729657,
-          "time_per_iteration": 0.000001485399997136483,
-          "iterations_per_second": 673219.3361571126,
-          "iterations_count": 20
-        },
-        "oj": {
-          "time_per_iterations": 0.00003158300000905001,
-          "time_per_iteration": 0.0000015791500004525006,
-          "iterations_per_second": 633252.0658034089,
-          "iterations_count": 20
-        }
-      }
-    },
-    "medium": {
-      "xml": { /* ... */ },
-      "json": { /* ... */ }
-    },
-    "large": {
-      "xml": { /* ... */ },
-      "json": { /* ... */ }
-    }
-  },
-  "ruby_version": "3.3.8",
-  "ruby_platform": "aarch64-linux",
-  "timestamp": "2025-06-07T09:01:02+00:00"
-}
-----
+The data formats include:
 
-=== Multi-Ruby version merged results
+* **Single benchmark results**: Individual benchmark run output
+* **Result set data structure**: Multi-platform benchmark aggregation
+* **JSON schema specification**: Complete schema validation rules
+* **Configuration file formats**: Docker and ASDF configuration examples
 
-When results from multiple Ruby versions are merged, the structure becomes:
+== Prerequisites
 
-[source,json]
-----
-{
-  "environments": {
-    "3_3_8_aarch64_linux": {
-      "ruby_version": "3.3.8",
-      "ruby_platform": "aarch64-linux",
-      "source_file": "docker-results/ruby-3.3/data/results.json",
-      "timestamp": "2025-06-07T09:01:02+00:00",
-      "environment": {
-        "ruby_version": "3.3.8",
-        "ruby_platform": "aarch64-linux",
-        "serializer_versions": { /* ... */ },
-        "timestamp": "2025-06-07T09:00:25+00:00"
-      }
-    },
-    "3_4_4_aarch64_linux": {
-      "ruby_version": "3.4.4",
-      "ruby_platform": "aarch64-linux",
-      /* ... */
-    }
-  },
-  "combined_results": {
-    "parsing": {
-      "small": {
-        "xml": {
-          "rexml": {
-            "3_3_8_aarch64_linux": {
-              "time_per_iterations": 0.002514416000053643,
-              "time_per_iteration": 0.00012572080000268216,
-              "iterations_per_second": 7954.133285650949,
-              "iterations_count": 20
-            },
-            "3_4_4_aarch64_linux": {
-              "time_per_iterations": 0.0025308329999234047,
-              "time_per_iteration": 0.00012654164999617023,
-              "iterations_per_second": 7902.536437846866,
-              "iterations_count": 20
-            }
-          }
-        }
-      }
-    }
-  },
-  "metadata": {
-    "merged_at": "2025-06-07T17:01:48+08:00",
-    "ruby_versions": ["3.3.8", "3.4.4"],
-    "platforms": ["aarch64-linux"]
-  }
-}
-----
+=== System requirements
+
+* **Ruby**: 3.0 or later (3.3+ recommended for best performance)
+* **Operating system**: Linux, macOS, or Windows
+* **Architecture**: x86_64 or ARM64
 
-=== JSON schema specification
+=== Library dependencies
 
-The benchmark results follow this schema:
+**System dependencies** (required for some native extensions):
 
-[source,json]
+[source,bash]
 ----
-{
-  "$schema": "http://json-schema.org/draft-07/schema#",
-  "title": "Serialbench Results",
-  "type": "object",
-  "properties": {
-    "environment": {
-      "type": "object",
-      "properties": {
-        "ruby_version": { "type": "string" },
-        "ruby_platform": { "type": "string" },
-        "serializer_versions": {
-          "type": "object",
-          "additionalProperties": { "type": "string" }
-        },
-        "timestamp": { "type": "string", "format": "date-time" }
-      },
-      "required": ["ruby_version", "ruby_platform", "serializer_versions", "timestamp"]
-    },
-    "parsing": {
-      "type": "object",
-      "properties": {
-        "small": { "$ref": "#/definitions/sizeResults" },
-        "medium": { "$ref": "#/definitions/sizeResults" },
-        "large": { "$ref": "#/definitions/sizeResults" }
-      }
-    },
-    "ruby_version": { "type": "string" },
-    "ruby_platform": { "type": "string" },
-    "timestamp": { "type": "string", "format": "date-time" }
-  },
-  "definitions": {
-    "sizeResults": {
-      "type": "object",
-      "properties": {
-        "xml": { "$ref": "#/definitions/formatResults" },
-        "json": { "$ref": "#/definitions/formatResults" },
-        "yaml": { "$ref": "#/definitions/formatResults" },
-        "toml": { "$ref": "#/definitions/formatResults" }
-      }
-    },
-    "formatResults": {
-      "type": "object",
-      "additionalProperties": {
-        "$ref": "#/definitions/serializerResults"
-      }
-    },
-    "serializerResults": {
-      "type": "object",
-      "properties": {
-        "time_per_iterations": { "type": "number" },
-        "time_per_iteration": { "type": "number" },
-        "iterations_per_second": { "type": "number" },
-        "iterations_count": { "type": "integer" }
-      },
-      "required": ["time_per_iterations", "time_per_iteration", "iterations_per_second", "iterations_count"]
-    }
-  }
-}
+# macOS with Homebrew
+$ brew install libxml2 libxslt
+
+# Ubuntu/Debian
+$ sudo apt-get install libxml2-dev libxslt1-dev build-essential
+
+# CentOS/RHEL/Fedora
+$ sudo yum install libxml2-devel libxslt-devel gcc gcc-c++
 ----
 
 == Installation
@@ -301,900 +148,298 @@ Or install it yourself as:
 $ gem install serialbench
 ----
 
-=== XML library dependencies
-
-To run benchmarks for all supported XML libraries, install the following gems:
 
+== Command line interface
 
-[source]
-----
-# Core XML libraries
-$ gem install ox nokogiri libxml-ruby oga
+Serialbench provides a comprehensive Thor-based CLI with four main subcommands
+for managing environments, benchmarks, result sets, and Ruby builds.
 
-# Additional format libraries (for comparison)
-$ gem install oj toml-rb
+=== Main Commands Overview
 
-# Memory profiling support
-$ gem install memory_profiler
+[source,sh]
 ----
+$ serialbench
+Serialbench - Benchmarking Framework for Ruby Serialization Libraries
 
+USAGE:
+  serialbench COMMAND [SUBCOMMAND] [OPTIONS]
 
-NOTE: REXML, JSON, and Psych (YAML) are included with Ruby and require no additional installation.
+COMMANDS:
+  environment   Manage benchmark environments (Docker, ASDF, Local)
+  benchmark     Manage individual benchmark runs
+  resultset     Manage benchmark resultsets (collections of runs)
+  ruby-build    Manage Ruby-Build definitions for validation
+  version       Show version information
+  help          Show this help message
 
-=== Library-specific installation notes
+EXAMPLES:
+  # Create a Docker environment
+  serialbench environment new docker-test docker
 
-==== Ox
+  # Run multi-environment benchmarks
+  serialbench environment multi-execute asdf --config=serialbench-asdf.yml
+  serialbench environment multi-execute docker --config=serialbench-docker.yml
 
-High-performance C extension requiring compilation:
+  # Create and execute a benchmark
+  serialbench benchmark create my-benchmark
+  serialbench benchmark execute my-benchmark.yml
 
-[source]
-----
-$ gem install ox
-----
-
-==== Nokogiri
+  # Create a result set for comparison
+  serialbench resultset create comparison-set
+  serialbench resultset add-result comparison-set results/my-benchmark
 
-May require system dependencies on some platforms:
-
-[source]
+  # Generate static sites
+  serialbench benchmark build-site results/my-benchmark
+  serialbench resultset build-site resultsets/comparison-set
 ----
-# macOS with Homebrew
-$ brew install libxml2 libxslt
-$ gem install nokogiri
-
-# Ubuntu/Debian
-$ sudo apt-get install libxml2-dev libxslt1-dev
-$ gem install nokogiri
-----
-
-==== LibXML
-
-Ruby bindings for libxml2:
-
-[source]
-----
-# macOS with Homebrew
-$ brew install libxml2
-$ gem install libxml-ruby
 
-# Ubuntu/Debian
-$ sudo apt-get install libxml2-dev
-$ gem install libxml-ruby
-----
-
-==== Oga
+=== Environment management
 
-Pure Ruby implementation with no system dependencies:
+The `environment` subcommand manages environment configurations and executes
+benchmarks across different Ruby environments.
 
 [source]
 ----
-$ gem install oga
+$ serialbench environment help
+Commands:
+  serialbench environment execute ENVIRONMENT_CONFIG BENCHMARK_CONFIG RESULT_PATH  # Execute benchmark in environment
+  serialbench environment help [COMMAND]                                           # Describe subcommands or one specific subcommand
+  serialbench environment new NAME KIND RUBY_BUILD_TAG                             # Create a new environment configuration
+  serialbench environment prepare ENVIRONMENT_CONFIG                               # Prepare environment for benchmarking
 ----
 
-== Docker containers and cross-platform benchmarking
-
-Serialbench provides pre-built Docker containers and supports cross-platform benchmarking on Windows, macOS, and Linux. Our containers are automatically built and published to GitHub Container Registry for easy access.
-
-=== Published Docker containers
-
-We publish multi-architecture Docker containers for all supported Ruby versions:
 
-**Registry**: `ghcr.io/metanorma/serialbench`
+=== Benchmark management
 
-**Available tags**:
-* `ghcr.io/metanorma/serialbench:main-ruby-3.1` - Ruby 3.1 (linux/amd64, linux/arm64)
-* `ghcr.io/metanorma/serialbench:main-ruby-3.2` - Ruby 3.2 (linux/amd64, linux/arm64)
-* `ghcr.io/metanorma/serialbench:main-ruby-3.3` - Ruby 3.3 (linux/amd64, linux/arm64)
-* `ghcr.io/metanorma/serialbench:main-ruby-3.4` - Ruby 3.4 (linux/amd64, linux/arm64)
-
-**Container features**:
-* All serialization libraries pre-installed (XML, JSON, YAML, TOML)
-* Multi-architecture support (Intel x64 and ARM64)
-* Optimized for CI/CD and local development
-* Automatic updates on code changes
-
-=== Running benchmarks on your computer
-
-==== Windows (PowerShell/Command Prompt)
-
-[source,powershell]
-----
-# Pull and run latest Ruby 3.3 container
-docker pull ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# Create results directory
-mkdir results
-
-# Run benchmarks with volume mounting
-docker run --rm -v ${PWD}/results:/app/results ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# View results
-dir results
-----
-
-==== macOS (Terminal)
-
-[source,bash]
-----
-# Pull and run latest Ruby 3.3 container
-docker pull ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# Create results directory
-mkdir -p results
-
-# Run benchmarks with volume mounting
-docker run --rm -v $(pwd)/results:/app/results ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# View results
-ls -la results/
-----
-
-==== Ubuntu/Linux (Terminal)
-
-[source,bash]
-----
-# Pull and run latest Ruby 3.3 container
-docker pull ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# Create results directory
-mkdir -p results
-
-# Run benchmarks with volume mounting
-docker run --rm -v $(pwd)/results:/app/results ghcr.io/metanorma/serialbench:main-ruby-3.3
-
-# View results
-ls -la results/
-----
-
-=== Multi-Ruby version comparison
-
-Run benchmarks across all supported Ruby versions:
-
-==== Windows (PowerShell)
-
-[source,powershell]
-----
-# Create results directories
-$versions = @("3.1", "3.2", "3.3", "3.4")
-foreach ($version in $versions) {
-    mkdir "results-ruby-$version" -Force
-    docker pull "ghcr.io/metanorma/serialbench:main-ruby-$version"
-    docker run --rm -v "${PWD}/results-ruby-${version}:/app/results" "ghcr.io/metanorma/serialbench:main-ruby-$version"
-}
-
-# View all results
-dir results-ruby-*
-----
-
-==== macOS/Linux (Bash)
-
-[source,bash]
-----
-# Run benchmarks for all Ruby versions
-for version in 3.1 3.2 3.3 3.4; do
-    echo "Running benchmarks for Ruby $version..."
-    mkdir -p "results-ruby-$version"
-    docker pull "ghcr.io/metanorma/serialbench:main-ruby-$version"
-    docker run --rm \
-        -v "$(pwd)/results-ruby-$version:/app/results" \
-        "ghcr.io/metanorma/serialbench:main-ruby-$version"
-done
-
-# View all results
-ls -la results-ruby-*/
-----
-
-=== Custom benchmark configuration
-
-Run benchmarks with custom parameters:
-
-[source,bash]
-----
-# Run specific formats only
-docker run --rm \
-    -v $(pwd)/results:/app/results \
-    ghcr.io/metanorma/serialbench:main-ruby-3.3 \
-    bundle exec serialbench benchmark --formats xml json --iterations 10
-
-# Run with memory profiling
-docker run --rm \
-    -v $(pwd)/results:/app/results \
-    ghcr.io/metanorma/serialbench:main-ruby-3.3 \
-    bundle exec serialbench benchmark --memory-profiling
-
-# List available serializers
-docker run --rm ghcr.io/metanorma/serialbench:main-ruby-3.3 \
-    bundle exec serialbench list
-----
-
-=== Quick start with Docker
-
-==== Prerequisites
-
-* Docker installed and running
-* Command line access (PowerShell, Terminal, or Bash)
-
-==== Running multi-Ruby benchmarks
+The `benchmark` subcommand handles individual benchmark runs and site generation.
 
 [source]
 ----
-# From the project root directory (if you have the source)
-$ ./docker/run-benchmarks.sh
-
-# Or using published containers directly
-$ docker pull ghcr.io/metanorma/serialbench:main-ruby-3.3
-$ docker run --rm -v $(pwd)/results:/app/results ghcr.io/metanorma/serialbench:main-ruby-3.3
+$ serialbench benchmark help
+Commands:
+  serialbench benchmark _docker_execute ENVIRONMENT_CONFIG_PATH BENCHMARK_CONFIG_PATH  # (Private) Execute a benchmark run
+  serialbench benchmark build-site RUN_PATH [OUTPUT_DIR]                               # Generate HTML site for a run
+  serialbench benchmark create [NAME]                                                  # Generate a run configuration file
+  serialbench benchmark execute ENVIRONMENT_CONFIG_PATH BENCHMARK_CONFIG_PATH          # Execute a benchmark run
+  serialbench benchmark help [COMMAND]                                                 # Describe subcommands or one specific subcommand
+  serialbench benchmark list                                                           # List all available runs
 ----
 
-This will:
+The `_docker_execute` command is a private command used internally by the
+`execute` command to run benchmarks in Docker environments.
 
-. **Pull pre-built containers** from GitHub Container Registry
-. **Run comprehensive benchmarks** in isolated environments
-. **Generate detailed reports** with performance comparisons
-. **Output results** to your local `results/` directory
 
-==== Results structure
+=== Result set management
 
-Results are organized in `docker-results/`:
+The `resultset` subcommand manages collections of benchmark runs for comparison analysis.
 
 [source]
 ----
-docker-results/
-├── ruby-3.0/              # Ruby 3.0 individual results
-│   ├── benchmark.log       # Execution log
-│   ├── data/
-│   │   ├── results.json    # Raw benchmark data
-│   │   └── results.yaml    # YAML format results
-│   ├── reports/
-│   │   └── benchmark_report.html
-│   └── assets/
-├── ruby-3.1/              # Ruby 3.1 individual results
-├── ruby-3.2/              # Ruby 3.2 individual results
-├── ruby-3.3/              # Ruby 3.3 individual results
-├── ruby-3.4/              # Ruby 3.4 individual results
-├── merged/                 # Aggregated cross-version results
-│   └── merged_results.json # Combined performance data
-└── docs/                   # GitHub Pages ready output
-    ├── index.html          # Interactive comparison report
-    ├── styles.css          # Report styling
-    └── merged_results.json # Data for interactive charts
+$ serialbench resultset help
+Commands:
+  serialbench resultset add-result RESULT_PATH RESULTSET_PATH     # Add a run to a resultset
+  serialbench resultset build-site RESULTSET_PATH [OUTPUT_DIR]    # Generate HTML site for a resultset
+  serialbench resultset create NAME PATH                          # Create a new resultset
+  serialbench resultset help [COMMAND]                            # Describe subcommands or one specific subcommand
+  serialbench resultset list                                      # List all available resultsets
+  serialbench resultset remove-result RESULTSET_PATH RESULT_PATH  # Remove a run from a resultset
 ----
 
-=== Manual Docker usage
 
-==== Build image for specific Ruby version
+=== ruby-build management
 
-[source]
-----
-$ docker build \
-  --build-arg RUBY_VERSION=3.3 \
-  -t serialbench:ruby-3.3 \
-  -f docker/Dockerfile.benchmark \
-  .
-----
-
-==== Run benchmarks in container
-
-[source]
-----
-# Create results directory
-$ mkdir -p results
-
-# Run benchmarks with volume mounting
-$ docker run \
-  --rm \
-  -v $(pwd)/results:/app/results \
-  serialbench:ruby-3.3
-----
-
-==== Custom configuration
-
-[source]
-----
-# Use custom config file
-$ docker run \
-  --rm \
-  -v $(pwd)/results:/app/results \
-  -v $(pwd)/config:/app/config \
-  serialbench:ruby-3.3 \
-  bundle exec serialbench benchmark --config config/ci.yml
-----
-
-=== Supported Ruby versions
-
-The Docker setup supports the following Ruby versions:
-
-* **Ruby 3.0** - Stable release with good performance baseline
-* **Ruby 3.1** - Improved performance and new features
-* **Ruby 3.2** - Enhanced YJIT and memory optimizations
-* **Ruby 3.3** - Latest stable with performance improvements
-* **Ruby 3.4** - Current development version
-
-Each version includes all supported serialization libraries:
-
-* **XML**: REXML (built-in), Ox, Nokogiri, Oga, LibXML
-* **JSON**: JSON (built-in), Oj, YAJL
-* **YAML**: Psych (built-in), Syck
-* **TOML**: TOML-RB, Tomlib
-
-=== Environment variables
-
-The Docker images support these environment variables:
-
-* `BUNDLE_PATH` - Bundle installation path
-* `BUNDLE_BIN` - Bundle binary path
-* `PATH` - System PATH including bundle binaries
-* `RUBY_VERSION` - Ruby version for build-time configuration
-
-=== Result aggregation and GitHub Pages
-
-==== Merging multi-version results
-
-The Docker workflow automatically merges results from all Ruby versions:
+The `ruby-build` subcommand manages Ruby build definitions and version information.
 
-[source]
-----
-# Manual result merging
-$ serialbench merge_results \
-  docker-results/ruby-3.0 \
-  docker-results/ruby-3.1 \
-  docker-results/ruby-3.2 \
-  docker-results/ruby-3.3 \
-  docker-results/ruby-3.4 \
-  docker-results/merged
-----
-
-==== GitHub Pages generation
-
-Generate interactive HTML reports ready for GitHub Pages deployment:
-
-[source]
-----
-# Generate GitHub Pages from multi-version results
-$ serialbench github_pages \
-  docker-results/ruby-3.0 \
-  docker-results/ruby-3.1 \
-  docker-results/ruby-3.2 \
-  docker-results/ruby-3.3 \
-  docker-results/ruby-3.4 \
-  docker-results/docs
-----
-
-The generated GitHub Pages include:
-
-* **Interactive Performance Charts**: Compare serializers across Ruby versions
-* **Multi-Version Analysis**: See how performance changes between Ruby releases
-* **Environment Details**: Ruby versions, platforms, and library versions
-* **Responsive Design**: Works on desktop and mobile devices
-* **Direct Deployment**: Ready for GitHub Pages, Netlify, or any static hosting
+Serialbench uses ruby-build definitions of Ruby interpreter types and versions
+for identification.
 
-==== Deploying to GitHub Pages
-
-. **Commit the generated files**:
-+
 [source]
 ----
-$ git add docker-results/docs/
-$ git commit -m "Add multi-Ruby benchmark results"
-$ git push origin main
+$ serialbench ruby-build help
+Commands:
+  serialbench ruby_build cache-info      # Show information about the Ruby-Build definitions cache
+  serialbench ruby_build help [COMMAND]  # Describe subcommands or one specific subcommand
+  serialbench ruby_build list [FILTER]   # List available Ruby-Build definitions
+  serialbench ruby_build show TAG        # Show details for a specific Ruby-Build definition
+  serialbench ruby_build suggest         # Suggest Ruby-Build tag for current Ruby version
+  serialbench ruby_build update          # Update Ruby-Build definitions from GitHub
+  serialbench ruby_build validate TAG    # Validate a Ruby-Build tag
 ----
 
-. **Enable GitHub Pages** in repository settings:
-.. Go to Settings → Pages
-.. Set source to "Deploy from a branch"
-.. Select branch containing the `docs/` folder
-.. Set folder to `/docker-results/docs`
-
-. **Access your results** at: `https://yourusername.github.io/yourrepo/`
-
-=== Troubleshooting Docker issues
 
-==== Build failures
+== Workflow examples
 
-Check build logs for specific Ruby versions:
+=== Docker-based testing
 
-[source]
-----
-$ cat docker-results/build-ruby-3.3.log
-----
+NOTE: This works.
 
-Common build issues:
-
-* **Missing system dependencies**: Ensure libxml2-dev and libxslt1-dev are available
-* **Network timeouts**: Retry the build or use a different network
-* **Disk space**: Ensure sufficient disk space for multiple Ruby images
-
-==== Runtime failures
-
-Check benchmark execution logs:
-
-[source]
-----
-$ cat docker-results/ruby-3.3/benchmark.log
+[source,bash]
 ----
+# 1. Prepare Docker environment
+$ bundle exec serialbench environment prepare config/environments/docker-ruby-3.1.yml
 
-Common runtime issues:
-
-* **Memory constraints**: Increase Docker memory allocation
-* **Timeout issues**: Some benchmarks may take longer on slower systems
-* **Permission errors**: Ensure proper volume mounting permissions
+# 2. Run benchmark
+$ bundle exec serialbench environment execute config/environments/docker-ruby-3.1.yml config/benchmarks/short.yml results/runs/docker-ruby-3.1-results
 
-==== Docker system issues
+# 3. Create a resultset
+$ bundle exec serialbench resultset create docker-comparison results/sets/docker-comparison
 
-Verify Docker is running properly:
+# 3a. (Optional) Build the site from the result if you want to visualize results
+$ bundle exec serialbench benchmark build-site results/runs/docker-ruby-3.1-results/ --output_dir=_site_result
 
-[source]
-----
-$ docker info
-$ docker system df  # Check disk usage
-$ docker system prune  # Clean up unused resources
-----
+# 4. Add the result to the resultset
+$ bundle exec serialbench resultset add-result results/sets/docker-comparison/ results/runs/docker-ruby-3.1-results/
 
-Clean up Serialbench Docker resources:
+# 5. Build the site from the resultset
+$ bundle exec serialbench resultset build-site results/sets/docker-comparison/
 
-[source]
-----
-# Remove all Serialbench images
-$ docker rmi $(docker images serialbench -q)
-
-# Remove all containers
-$ docker container prune
+# 6. Open the generated site
+$ open _site/index.html
 ----
 
-=== Customization options
+=== ASDF-based testing
 
-==== Adding Ruby versions
-
-Edit the `RUBY_VERSIONS` array in `docker/run-benchmarks.sh`:
+WARNING: THIS IS NOT YET WORKING.
 
 [source,bash]
 ----
-RUBY_VERSIONS=("3.0" "3.1" "3.2" "3.3" "3.4" "head")
-----
+# 1. Validate configuration
+$ bundle exec serialbench benchmark validate serialbench-asdf.yml
 
-==== Custom benchmark configuration
+# 2. Prepare Ruby environments
+$ bundle exec serialbench benchmark prepare asdf --config=serialbench-asdf.yml
 
-Create custom config files in the `config/` directory:
+# 3. Run benchmarks across all Ruby versions
+$ bundle exec serialbench benchmark execute asdf --config=serialbench-asdf.yml
 
-[source,yaml]
-----
-# config/custom.yml
-formats:
-  - xml
-  - json
-iterations: 50
-warmup: 5
-data_sizes:
-  - small
-  - medium
+# 4. Results are automatically merged and dashboard generated
+$ open asdf-results/_site/index.html
 ----
 
-Reference the custom config in the run script:
 
-[source,bash]
-----
-# In docker/run-benchmarks.sh
-CONFIG_FILE="config/custom.yml"
-----
-
-==== Output directory customization
-
-Change the output directory in the run script:
 
-[source,bash]
-----
-# In docker/run-benchmarks.sh
-OUTPUT_DIR="my-benchmark-results"
-----
 
-=== Integration with CI/CD
+== Configuration Files
 
-==== GitHub Actions integration
+=== Environment configuration
 
-The Docker setup integrates seamlessly with GitHub Actions:
+Environment configuration files define how benchmarks are executed in different runtime environments.
 
+.Environment configuration for Docker (`config/environments/docker-ruby-3.4.yml`)
 [source,yaml]
 ----
-# .github/workflows/benchmark.yml
-name: Multi-Ruby Benchmarks
-
-on:
-  schedule:
-    - cron: '0 2 * * 0'  # Weekly on Sunday at 2 AM
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-
-concurrency:
-  group: "pages"
-  cancel-in-progress: false
-
-jobs:
-  benchmark:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Setup Docker Buildx
-        uses: docker/setup-buildx-action@v3
-
-      - name: Run Docker Benchmarks
-        run: ./docker/run-benchmarks.sh
-
-      - name: Upload Results
-        uses: actions/upload-artifact@v4
-        with:
-          name: benchmark-results
-          path: docker-results/
-
-      - name: Setup Pages
-        uses: actions/configure-pages@v4
-
-      - name: Upload Pages artifact
-        uses: actions/upload-pages-artifact@v3
-        with:
-          path: ./docker-results/docs
-
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
-    needs: benchmark
-    steps:
-      - name: Deploy to GitHub Pages
-        id: deployment
-        uses: actions/deploy-pages@v4
-----
-
-==== Performance considerations
-
-* **Parallel builds**: Docker builds can run in parallel for faster execution
-* **Build caching**: Subsequent runs use cached layers for faster builds
-* **Memory profiling**: Enabled by default but can be disabled for faster runs
-* **Result compression**: Large result files can be compressed for storage
-
-=== Security considerations
-
-* **Containers run with minimal privileges**: No root access required
-* **No network access during benchmarks**: Isolated execution environment
-* **Volume mounting**: Results written only to specified mounted volumes
-* **Image scanning**: Regular security updates for base Ruby images
-
-== Usage
-
-=== Command line interface
-
-==== Basic usage
-
-Run benchmarks for all available formats:
-
-[source]
-----
-$ serialbench benchmark
-----
-
-List all available serializers:
-
-[source]
-----
-$ serialbench list
-----
-
-Show help information:
-
-[source]
-----
-$ serialbench help
-$ serialbench help benchmark
-----
-
-Show version:
-
-[source]
-----
-$ serialbench version
-----
-
-==== Format-specific benchmarks
-
-===== XML benchmarks
-
-Run all XML library benchmarks:
-
-[source]
-----
-$ serialbench benchmark --formats xml
-----
-
-Test specific XML libraries:
-
-[source]
-----
-$ serialbench benchmark --formats xml --parsers ox,nokogiri
-$ serialbench benchmark --formats xml --parsers rexml,oga,libxml
-----
-
-XML-only parsing performance:
-
-[source]
-----
-$ serialbench benchmark --formats xml --parsing-only
+---
+name: docker-ruby-3.4
+kind: docker
+created_at: '2025-06-13T15:18:43+08:00'
+ruby_build_tag: "3.4.1"
+description: Docker environment for Ruby 3.4 benchmarks
+docker:
+  image: 'ruby:3.4-slim'
+  dockerfile: '../../docker/Dockerfile.ubuntu'
 ----
 
-XML generation benchmarks:
-
-[source]
-----
-$ serialbench benchmark --formats xml --generation-only
-----
-
-XML streaming/SAX parsing:
-
-[source]
-----
-$ serialbench benchmark --formats xml --streaming-only
-----
-
-===== JSON benchmarks
-
-Run all JSON library benchmarks:
-
-[source]
-----
-$ serialbench benchmark --formats json
-----
-
-Test specific JSON libraries:
-
-[source]
-----
-$ serialbench benchmark --formats json --parsers oj,json
-$ serialbench benchmark --formats json --parsers yajl,oj
-----
-
-===== TOML benchmarks
-
-Run all TOML library benchmarks:
-
-[source]
-----
-$ serialbench benchmark --formats toml
-----
-
-Test specific TOML libraries:
-
-[source]
+.Environment configuration for ASDF (`config/environments/asdf-ruby-3.3.yml`)
+[source,yaml]
 ----
-$ serialbench benchmark --formats toml --parsers tomlib,toml-rb
+---
+name: ruby-332-asdf
+kind: asdf
+created_at: '2025-06-12T22:53:24+08:00'
+ruby_build_tag: 3.3.2
+description: ASDF environment
+asdf:
+  auto_install: true
 ----
 
-==== Cross-format comparisons
-
-Compare XML vs JSON performance:
-
-[source]
-----
-$ serialbench benchmark --formats xml json
-----
+=== Benchmark configuration
 
-Compare all supported formats:
+Benchmark configuration files control what tests to run and how to run them.
 
-[source]
-----
-$ serialbench benchmark --formats xml json toml
+.Short configuration (CI-friendly) (`config/benchmarks/short.yml`)
+[source,yaml]
 ----
+name: short-benchmark
 
-==== Advanced options
-
-Memory profiling across formats:
-
-[source]
-----
-$ serialbench benchmark --memory-profiling
-----
+data_sizes:
+- small
 
-Generate detailed reports:
+formats:
+- xml
+- json
+- yaml
+- toml
 
-[source]
-----
-$ serialbench benchmark --detailed-reports
-----
+iterations:
+  small: 5
+  medium: 2
+  large: 1
 
-Output results in JSON format:
+operations:
+- parse
+- generate
+- streaming
 
-[source]
-----
-$ serialbench benchmark --output-format json
+warmup: 2
 ----
 
-Custom data sizes and iterations:
-
-[source]
-----
-$ serialbench benchmark --data-sizes small,medium --iterations 100
+.Full configuration (Comprehensive) (`config/benchmarks/full.yml`)
+[source,yaml]
 ----
+name: full-benchmark
 
-=== Multi-Ruby version comparison
+data_sizes:
+- small
+- medium
+- large
 
-Merge benchmark results from multiple Ruby versions:
+formats:
+- xml
+- json
+- yaml
+- toml
 
-[source]
-----
-$ serialbench merge_results ruby-3.0/results ruby-3.1/results ruby-3.2/results merged_output/
-----
+iterations:
+  small: 20
+  medium: 5
+  large: 2
 
-Generate GitHub Pages HTML from multiple benchmark runs:
+operations:
+- parse
+- generate
+- streaming
+- memory
 
-[source]
-----
-$ serialbench github_pages ruby-3.0/results ruby-3.1/results ruby-3.2/results docs/
+warmup: 3
 ----
 
-This creates an interactive HTML report with:
+== Results structure
 
-* **Multi-version charts**: Compare performance across Ruby versions
-* **Interactive navigation**: Switch between parsing, generation, streaming, and memory usage
-* **Environment details**: Ruby versions, platforms, and serializer versions
-* **GitHub Pages ready**: Deploy directly to GitHub Pages for public sharing
+=== Individual run results
 
-=== Cross-platform performance analysis
+Results are stored in a structured directory format, with each run containing
+raw benchmark data and execution logs.
 
-Analyze performance data from multiple benchmark runs across different platforms and Ruby versions:
+The directory is located at `results/runs/{name}/`, where `{name}` is the name
+of the environment used for the benchmark.
 
 [source]
 ----
-$ serialbench analyze_performance artifacts/benchmark-results-*/ performance_analysis.json
+results/runs/docker-ruby-33-results/
+├── results.yaml                    # Raw benchmark data
+└── benchmark.log                   # Execution log
 ----
 
-This command:
+=== ResultSet structure
 
-* **Processes multiple result directories** from different platforms and Ruby versions
-* **Extracts platform and Ruby version** information from directory names
-* **Generates comprehensive JSON** with detailed performance metrics
-* **Handles both parsing and generation** benchmark results
-* **Provides summary statistics** about processed data
-
-Generate platform comparison reports:
+ResultSets aggregate multiple benchmark runs for comparison. They are stored in
+a structured directory format at `results/sets/{name}/`, where `{name}` is the
+name of the result set.
 
 [source]
 ----
-$ serialbench platform_comparison performance_analysis.json platform_comparison.json
-----
-
-This creates a JSON report with:
-
-* **Cross-platform statistics**: Average, min, max performance by platform
-* **Format-specific analysis**: Performance breakdown by serialization format
-* **Operation comparison**: Separate analysis for parsing vs generation
-* **Sample counts**: Number of data points for statistical confidence
-* **Standard deviation**: Statistical variance for performance consistency
-
-==== Example workflow for cross-platform analysis
-
-[source]
-----
-# 1. Run benchmarks on different platforms (or use CI artifacts)
-$ serialbench benchmark --formats xml json yaml toml
-
-# 2. Collect results from multiple platforms/Ruby versions
-$ mkdir analysis
-$ cp -r platform1-ruby3.2/results analysis/benchmark-results-ubuntu-ruby-3.2
-$ cp -r platform2-ruby3.3/results analysis/benchmark-results-macos-ruby-3.3
-$ cp -r platform3-ruby3.4/results analysis/benchmark-results-windows-ruby-3.4
-
-# 3. Generate performance analysis
-$ serialbench analyze_performance analysis/benchmark-results-*/ performance_analysis.json
-
-# 4. Create platform comparison report
-$ serialbench platform_comparison performance_analysis.json platform_comparison.json
-
-# 5. View results
-$ cat performance_analysis.json | jq '.summary'
-$ cat performance_analysis.json | jq '.platforms'
-$ cat platform_comparison.json | jq '.platforms'
-----
-
-The analysis commands are particularly useful for:
-
-* **CI/CD integration**: Automated cross-platform performance tracking
-* **Performance regression detection**: Compare results across builds
-* **Platform optimization**: Identify platform-specific performance characteristics
-* **Ruby version migration**: Analyze performance impact of Ruby upgrades
-
-=== Programmatic usage
-
-==== Basic benchmark execution
-
-[source,ruby]
-----
-require 'serialbench'
-
-# Run all benchmarks for all formats
-results = Serialbench.run_benchmarks
-
-# Run benchmarks for specific formats
-results = Serialbench.run_benchmarks(formats: [:xml, :json])
-
-# Generate comprehensive reports
-report_files = Serialbench.generate_reports(results)
-
-puts "HTML report: #{report_files[:html]}"
-puts "Charts generated: #{report_files[:charts].length}"
-----
-
-==== Custom benchmark configuration
-
-[source,ruby]
-----
-require 'serialbench'
-
-# Create a custom benchmark runner
-runner = Serialbench::BenchmarkRunner.new(formats: [:json, :xml])
-
-# Run specific benchmark categories
-parsing_results = runner.run_parsing_benchmarks
-generation_results = runner.run_generation_benchmarks
-memory_results = runner.run_memory_benchmarks
-
-# Format and display results
-formatter = Serialbench::ResultFormatter.new(runner.results)
-puts formatter.summary
-----
-
-==== Individual serializer testing
-
-[source,ruby]
-----
-require 'serialbench'
-
-# Test a specific JSON serializer
-oj_serializer = Serialbench::Serializers::Json::OjSerializer.new
-
-if oj_serializer.available?
-  json_content = '{"users": [{"name": "Alice", "age": 30}]}'
-
-  # Parse JSON
-  data = oj_serializer.parse(json_content)
-
-  # Generate JSON
-  json_output = oj_serializer.generate(data, pretty: true)
-
-  # Stream parsing (if supported)
-  if oj_serializer.supports_streaming?
-    oj_serializer.stream_parse(json_content) do |event, data|
-      puts "Event: #{event}, Data: #{data}"
-    end
-  end
-
-  puts "Serializer: #{oj_serializer.name}"
-  puts "Version: #{oj_serializer.version}"
-  puts "Format: #{oj_serializer.format}"
-  puts "Features: #{oj_serializer.features}"
-end
-----
-
-==== Check available serializers
-
-[source,ruby]
-----
-require 'serialbench'
-
-# List all available serializers
-Serialbench.available_serializers.each do |serializer_class|
-  serializer = serializer_class.new
-  puts "#{serializer.format}: #{serializer.name} v#{serializer.version}"
-end
-
-# List serializers for specific format
-Serialbench.available_serializers(:json).each do |serializer_class|
-  serializer = serializer_class.new
-  puts "JSON: #{serializer.name} v#{serializer.version}"
-end
+results/sets/ruby-version-comparison/
+└── resultset.yml                  # Result set configuration
 ----
 
 == Benchmark categories
@@ -1221,105 +466,32 @@ it, which processes data sequentially and is memory-efficient for large files.
 Profiles memory allocation and retention during serialization operations using
 the `memory_profiler` gem.
 
+== Interactive Dashboard Features
 
-== Output and reports
-
-=== Generated files
+The generated HTML sites provide comprehensive interactive dashboards with:
 
-Running benchmarks creates the following output structure:
+=== Navigation and Filtering
+* **Format tabs**: Dedicated views for XML, JSON, YAML, and TOML
+* **Operation sections**: Parsing, generation, streaming, and memory usage
+* **Dynamic filtering**: Platform, Ruby version, and environment selection
+* **Real-time updates**: Charts update instantly based on filter selections
 
-[source]
-----
-results/
-├── reports/
-│   ├── benchmark_report.html    # Main HTML report
-│   └── benchmark_report.adoc    # AsciiDoc source
-├── charts/
-│   ├── parsing_performance.svg
-│   ├── generation_performance.svg
-│   ├── streaming_performance.svg
-│   ├── memory_usage_comparison.svg
-│   └── format_comparison.svg
-├── data/
-│   ├── results.json             # Raw benchmark data
-│   └── results.csv              # CSV export
-└── assets/
-    └── css/
-        └── benchmark_report.css # Report styling
-----
-
-=== Report features
-
-* **Multi-format comparison**: Compare XML, JSON, and TOML performance
-* **Interactive charts**: SVG-based performance visualizations
-* **Comparative analysis**: Side-by-side library comparisons
-* **Performance rankings**: Fastest to slowest for each category
-* **Memory profiling**: Detailed memory allocation analysis
-* **Feature matrix**: Capability comparison across libraries
-* **Environment details**: Ruby version, platform, and library versions
-
-=== Sample output
-
-[example]
-====
-Serialbench - Comprehensive Serialization Performance Tests
-===========================================================
-Environment: Ruby 3.3.2 on arm64-darwin23
-Timestamp: 2025-06-07T10:30:00Z
-
-Available serializers: rexml, json, oj, toml-rb
-Test formats: xml, json, toml
-Test data sizes: small, medium, large
-
-Parsing Performance:
-  Small files:
-    JSON/oj: 0.08ms
-    JSON/json: 0.12ms
-    XML/rexml: 0.45ms
-    TOML/toml-rb: 0.52ms
-
-  Medium files:
-    JSON/oj: 8.23ms
-    JSON/json: 12.67ms
-    XML/rexml: 28.45ms
-    TOML/toml-rb: 35.21ms
-====
-
-== Methodology
+=== Visualization Capabilities
+* **Chart.js integration**: Interactive performance charts with hover details
+* **Multi-scale handling**: Automatic Y-axis scaling for different performance ranges
+* **Color-coded data**: Consistent color schemes across serializers and environments
+* **Responsive design**: Optimized for desktop and mobile viewing
 
-=== Performance measurement
+=== User Experience
+* **Theme toggle**: Light and dark mode with persistent preferences
+* **Keyboard navigation**: Full accessibility support
+* **Fast loading**: Optimized JavaScript for quick dashboard initialization
+* **Export capabilities**: JSON data export for further analysis
 
-* Each test runs multiple iterations with warmup iterations
-* Memory profiling uses 10 iterations to reduce noise
-* Results show average performance across all iterations
-* Benchmarks use Ruby's `Benchmark.realtime` for precise timing
-
-=== Test data
-
-==== Synthetic datasets
-
-The benchmark suite uses carefully crafted synthetic data that represents common real-world scenarios:
-
-* **Configuration files**: Small, nested structures typical of application settings
-* **API responses**: Medium-sized documents with repeated record structures
-* **Data exports**: Large documents with extensive hierarchical data
-
-==== Multi-format consistency
-
-* Equivalent data structures across XML, JSON, and TOML formats
-* Consistent complexity and nesting levels
-* Representative of real-world usage patterns
-
-=== Statistical considerations
-
-* Multiple iterations reduce timing variance
-* Warmup iterations eliminate JIT compilation effects
-* Memory measurements account for garbage collection
-* Results include both absolute and relative performance metrics
 
 == Development
 
-=== Running tests
+=== Running Tests
 
 [source]
 ----
@@ -1327,15 +499,7 @@ $ bundle exec rake
 $ bundle exec rspec
 ----
 
-=== Contributing
-
-. Fork the repository
-. Create your feature branch (`git checkout -b feature/my-new-feature`)
-. Commit your changes (`git commit -am 'Add some feature'`)
-. Push to the branch (`git push origin feature/my-new-feature`)
-. Create a new Pull Request
-
-=== Adding new serializers
+=== Adding a new serializers
 
 To add support for additional serialization libraries:
 
@@ -1345,76 +509,50 @@ To add support for additional serialization libraries:
 . Add the serializer to the registry in `lib/serialbench/serializers.rb`
 . Update documentation and tests
 
-==== Example: Adding a new JSON serializer
+=== Contributing
 
-[source,ruby]
-----
-# lib/serialbench/serializers/json/yajl_serializer.rb
-class YajlSerializer < BaseJsonSerializer
-  def available?
-    require_library('yajl')
-  end
-
-  def name
-    'yajl'
-  end
-
-  def version
-    require 'yajl'
-    Yajl::VERSION
-  end
-
-  def parse(json_string)
-    require 'yajl'
-    Yajl::Parser.parse(json_string)
-  end
-
-  def generate(object, options = {})
-    require 'yajl'
-    Yajl::Encoder.encode(object)
-  end
-end
-----
+. Fork the repository
+. Create your feature branch (`git checkout -b feature/my-new-feature`)
+. Commit your changes (`git commit -am 'Add some feature'`)
+. Push to the branch (`git push origin feature/my-new-feature`)
+. Create a new Pull Request
 
-== Architecture
 
-=== Serializer hierarchy
+== Known issues
 
-[source]
-----
-BaseSerializer
-├── BaseXmlSerializer
-│   └── RexmlSerializer
-├── BaseJsonSerializer
-│   ├── JsonSerializer
-│   └── OjSerializer
-└── BaseTomlSerializer
-    └── TomlRbSerializer
-----
+=== Syck YAML serializer segmentation faults
 
-=== Key components
+The Syck YAML serializer at version 1.5+ is known to cause segmentation faults
+on Ruby 3.1 and later versions. Serialbench automatically detects this
+problematic configuration and:
 
-* **Serializers**: Individual library implementations
-* **BenchmarkRunner**: Orchestrates benchmark execution
-* **ResultFormatter**: Formats and displays results
-* **ReportGenerator**: Creates HTML/AsciiDoc reports
-* **ChartGenerator**: Creates performance visualizations
-* **MemoryProfiler**: Analyzes memory usage patterns
+* Displays a warning message when Syck is detected on Ruby 3.1+
+* Skips Syck benchmarks to prevent crashes
+* Continues with other YAML serializers (Psych)
 
-== Research and references
+=== Syck overrides YAML constant
 
-This benchmarking suite was developed based on research from:
+On occasion after Syck is loaded, the constant `YAML` may be redefined to
+`Syck`, which can cause issues in other parts of the codebase. This can cause
+YAML output to fail when using libraries that expect `YAML` to have the
+`Psych` API.
+
+In `benchmark_cli.rb` there is therefore such code to ensure that
+`YAML` is defined as `Psych` when writing to file is needed:
+
+[source,ruby]
+----
+# Restore YAML to use Psych for output, otherwise lutaml-model's to_yaml
+# will have no output
+Object.const_set(:YAML, Psych)
+----
 
-* https://www.ohler.com/dev/xml_with_ruby/xml_with_ruby.html[XML with Ruby performance analysis]
-* https://gist.github.com/danneu/3977120[Ruby XML parser comparison]
-* https://gist.github.com/adilosa/d4277dc1c683da91990515352ffe5420[XML parsing benchmarks]
 
-== Copyright
+== License and copyright
 
-This gem is developed, maintained and funded by
-https://www.ribose.com[Ribose Inc.]
+Copyright Ribose.
 
-== License
+This gem is developed, maintained and funded by https://www.ribose.com[Ribose]
 
 The gem is available as open source under the terms of the
 https://opensource.org/licenses/BSD-2-Clause[2-Clause BSD License].