irt_ruby 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: eb346c93f7beebb572f5c80663276c1f0285f55fe3382b0c6dc43ee17e3c2d04
4
- data.tar.gz: 9fcc3c15cd54e969ffc531fd004d5911eed71234c1218a643ac137aca3ae85ea
3
+ metadata.gz: 520defda852bedfe0f27ea300cb471b2f9afc6b8fbf32c16cb75fc3569cca5df
4
+ data.tar.gz: 16c98bd94b5d88e1c898be6aab91e987d1db3e957bb1dc921e73af02f61d9377
5
5
  SHA512:
6
- metadata.gz: 1c78192318e3fb78c9ee9c33919fb5dced7d09e4c38af7f56b9f508c1d5e26fc4aab9665e4acdd12ce711dedae12299ef054182ac9d2a25458a5561c9d31a501
7
- data.tar.gz: 1577007bae07c567fc05b6d2ad6037b3264d0ec2a23086eabe317ec332d896416d94dd257eb18a4e63d9b7ba7a21356cf3a8579bd1a419d849946280934a8f58
6
+ metadata.gz: 2c7a636273fbccda8bce8c96acb157fbb5a3f0d209cf275e4ddff845d1b5ea139f0389c6b5c9c296dfdf98c01ab7e89ab6080e8ed21940319e05845701853ef7
7
+ data.tar.gz: c8b2d1447884a130e1f25ac82a2cae0ec9198171372023ea48ff8aa57168ff78724becd51e1da040e7c1e7e9f822d9775541baf2bfe77f5d0d4af51d7dc761ff
data/CHANGELOG.md ADDED
@@ -0,0 +1,71 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented in this file.
4
+
5
+ ## [0.4.0] - 2026-10-11
6
+
7
+ ### Fixed
8
+ - Fixed 3PL gradients for ability, difficulty, discrimination, and guessing; they now match finite differences of the log-likelihood.
9
+ - Only the largest strongly connected part of the response graph is estimated; other respondents and items are returned as `nil` instead of drifting with `max_iter` or keeping random initial values. This covers all-correct, all-incorrect, and unobserved rows or columns as well as perfectly separated or disconnected groups.
10
+ - 3PL keeps the limiting `log(1 - c)` contribution of excluded respondents who answered every estimated item incorrectly, and keeps guessing gradients finite when the response function underflows.
11
+ - Log-likelihoods are computed with a numerically stable log-sigmoid instead of adding `1e-15` inside `Math.log`, which flattened the objective for badly mispredicted responses.
12
+ - Models copy response data on construction and return fresh arrays from `fit`, so later mutation of either does not change model state.
13
+ - Model constructors validate optimization options and raise `ArgumentError` unless `max_iter` is a positive Integer, `tolerance`, `param_tolerance`, `gradient_tolerance`, and `learning_rate` are positive finite numbers, and `decay_factor` is strictly between 0 and 1.
14
+
15
+ ### Changed
16
+ - `fit` reports results on a fixed latent scale: Rasch abilities are centered at 0; 2PL and 3PL abilities are standardized to mean 0 and SD 1 with likelihood-preserving item transformations.
17
+ - `fit` results include `converged:` and `iterations:`. `converged` is true only at a stationary point on the reported scale, controlled by the new `gradient_tolerance:` option (default `1e-3`).
18
+ - The learning rate grows by 10% after each accepted step, so it can recover from earlier decreases.
19
+ - Clarified the gem summary and description to describe implemented capabilities rather than production-readiness claims.
20
+ - Added CI RuboCop and gem package build/install verification, and documented the local quality and packaging commands for contributors. RuboCop is pinned to 1.92.0 for Ruby 2.7 and newer.
21
+ - Clarified that response data must be a `Matrix` or array of arrays containing only integer `0`, integer `1`, or `nil`; floats, strings, booleans, and other values are rejected.
22
+
23
+ ### Added
24
+ - Added a `seed:` keyword to all models for reproducible initialization without resetting or consuming Ruby's global random number generator.
25
+ - Added finite-difference gradient, identification, extreme-pattern, and parameter-recovery specs.
26
+ - Added executable specs for README usage examples to keep the documented Rasch, 2PL, 3PL, and missing-data flows aligned with the public API.
27
+
28
+ ---
29
+
30
+ ## [0.3.0] - 2025-06-13
31
+
32
+ ### Changed
33
+ - **Code Quality**
34
+ - Updated RuboCop configuration to handle new cops and resolve style warnings
35
+ - Fixed operator precedence ambiguity in Three-Parameter Model calculations
36
+ - Added MFA requirement to gemspec metadata (RuboCop requirement)
37
+
38
+ ### Notes
39
+ - This release maintains **full backward compatibility** with previous versions
40
+ - All 46 existing tests continue to pass
41
+ - Comprehensive performance benchmarking suite remains available via `bundle exec rake benchmark:all`
42
+
43
+ ---
44
+
45
+ ## [0.2.0] - 2025-03-01
46
+
47
+ ### Added
48
+ - **Missing Data Strategies**
49
+ - Introduced a `missing_strategy` parameter for **Rasch**, **TwoParameterModel**, and **ThreeParameterModel** to handle `nil` responses:
50
+ - `:ignore` (default) – skip missing responses in log-likelihood and gradients.
51
+ - `:treat_as_incorrect` – interpret `nil` as `0`.
52
+ - `:treat_as_correct` – interpret `nil` as `1`.
53
+ - Updated RSpec tests to cover each strategy and ensure graceful handling of missing responses.
54
+
55
+ - **Expanded Test Coverage**
56
+ - Added tests for repeated fits, deterministic seeding, larger random datasets, and new edge cases (all-correct/all-incorrect).
57
+ - Improved specs for parameter clamping (discriminations, guessing in 2PL/3PL).
58
+
59
+ - **Adaptive Learning Rate Enhancements**
60
+ - Enhanced convergence checks combining log-likelihood changes and average parameter updates.
61
+ - Clearer revert-and-decay logic if the likelihood decreases on a given step.
62
+
63
+ ### Changed
64
+ - **Documentation / README**
65
+ - Updated the README to reflect new missing data strategies, advanced usage (adaptive learning rate, parameter clamping), and test instructions.
66
+ - Added examples showcasing how to set `missing_strategy` for each model.
67
+
68
+ ### Notes
69
+ - This release remains **backward-compatible** with `0.1.x` in terms of existing usage; the default `:ignore` missing-data approach matches prior behavior.
70
+ - If upgrading, simply update your gem and enjoy the new features.
71
+ - For more details, see the updated [README](./README.md) and expanded test suites.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2024 Alex Kholodniak
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,209 @@
1
+ # IrtRuby
2
+
3
+ IrtRuby is a Ruby gem that provides implementations of the **Rasch model**, the **Two-Parameter (2PL)** model, and the **Three-Parameter (3PL)** model for Item Response Theory (IRT). It allows you to estimate the **abilities** of individuals and the **difficulties** (and optionally **discriminations** and **guessing** parameters) of items based on their responses.
4
+
5
+ ## Installation
6
+
7
+ Add this line to your application's Gemfile:
8
+
9
+ ```ruby
10
+ gem 'irt_ruby'
11
+ ```
12
+
13
+ And then execute:
14
+
15
+ ```bash
16
+ bundle install
17
+ ```
18
+
19
+ Or install it yourself as:
20
+
21
+ ```bash
22
+ gem install irt_ruby
23
+ ```
24
+
25
+ ## Usage
26
+
27
+ Here's a quick example using the Rasch model:
28
+
29
+ ```ruby
30
+ require 'irt_ruby'
31
+ require 'matrix'
32
+
33
+ # Create a sample response matrix
34
+ data = Matrix[
35
+ [1, 0, 1],
36
+ [0, 1, 0],
37
+ [1, 1, 1]
38
+ ]
39
+
40
+ # Initialize the Rasch model with the response data
41
+ model = IrtRuby::RaschModel.new(data)
42
+
43
+ # Fit the model to estimate abilities and difficulties
44
+ result = model.fit
45
+
46
+ # Output the estimated abilities and difficulties
47
+ puts "Abilities: #{result[:abilities]}"
48
+ puts "Difficulties: #{result[:difficulties]}"
49
+ puts "Converged: #{result[:converged]} after #{result[:iterations]} iterations"
50
+ ```
51
+
52
+ The third respondent answered every item correctly, so their ability has no
53
+ finite estimate and is returned as `nil` (see [Estimation Details](#estimation-details)).
54
+
55
+ Response data passed to model constructors must be either a `Matrix` or an
56
+ array of arrays. Each response value must be the integer `0`, the integer `1`,
57
+ or `nil` for missing data; floats such as `0.0`/`1.0`, strings, booleans, and
58
+ other values are rejected.
59
+
60
+ ### Using 2PL and 3PL Models
61
+ ```ruby
62
+ two_pl_model = IrtRuby::TwoParameterModel.new(data)
63
+ two_pl_result = two_pl_model.fit
64
+ puts two_pl_result[:abilities]
65
+ puts two_pl_result[:difficulties]
66
+ puts two_pl_result[:discriminations]
67
+
68
+ three_pl_model = IrtRuby::ThreeParameterModel.new(data)
69
+ three_pl_result = three_pl_model.fit
70
+ puts three_pl_result[:abilities]
71
+ puts three_pl_result[:difficulties]
72
+ puts three_pl_result[:discriminations]
73
+ puts three_pl_result[:guessings]
74
+ ```
75
+
76
+ Three respondents are far too few for 2PL or 3PL: on this matrix `fit` returns
77
+ `converged: false`. Discrimination and guessing estimates need hundreds of
78
+ respondents.
79
+
80
+ ## Handling Missing Data
81
+ Real-world data often has missing responses. Each model (Rasch, 2PL, 3PL) accepts a `missing_strategy:` option to handle nil entries:
82
+
83
+ - `:ignore` (default): Skip `nil` responses entirely in the log-likelihood and gradient calculations.
84
+ - `:treat_as_incorrect`: Interpret `nil` as `0`.
85
+ - `:treat_as_correct`: Interpret `nil` as `1`.
86
+
87
+ For example:
88
+ ```ruby
89
+ data_with_missing = [
90
+ [1, nil, 0],
91
+ [nil, 1, 0],
92
+ [0, 1, 1]
93
+ ]
94
+
95
+ model = IrtRuby::RaschModel.new(
96
+ data_with_missing,
97
+ max_iter: 300,
98
+ learning_rate: 0.01,
99
+ missing_strategy: :treat_as_incorrect
100
+ )
101
+ result = model.fit
102
+
103
+ puts "Abilities: #{result[:abilities]}"
104
+ puts "Difficulties: #{result[:difficulties]}"
105
+ ```
106
+ This flexibility helps you handle datasets where missingness might signify a skipped item or an unanswered question.
107
+
108
+ ## Advanced Usage
109
+
110
+ ### Adaptive Learning Rate & Convergence
111
+ By default, each model uses a gradient ascent with:
112
+
113
+ - An adaptive learning rate: if log-likelihood decreases, the step is reverted and the rate is multiplied by `decay_factor`; after each accepted step the rate grows by 10%.
114
+ - A stopping rule on the change in log-likelihood and the average parameter update.
115
+ - A stationarity check that decides whether the stopping point is reported as converged.
116
+
117
+ You can customize:
118
+
119
+ - `max_iter`: The maximum number of iterations.
120
+ - `tolerance` and `param_tolerance`: Stopping thresholds for log-likelihood change and parameter updates.
121
+ - `gradient_tolerance` (default `1e-3`): `fit` reports `converged: true` only if, for every parameter, the gradient averaged over the responses it depends on is below this value.
122
+ - `learning_rate`: Initial learning rate.
123
+ - `decay_factor`: Factor by which the learning rate is reduced on a failed step.
124
+
125
+ `fit` returns `converged: false` when `max_iter` runs out or when the stopping rule fires away from a stationary point, for example because the learning rate became too small to move. Check this flag before using the estimates; a larger `learning_rate` or `max_iter` usually helps.
126
+
127
+ Example:
128
+ ```ruby
129
+ IrtRuby::TwoParameterModel.new(
130
+ data,
131
+ max_iter: 500,
132
+ tolerance: 1e-7,
133
+ param_tolerance: 1e-7,
134
+ learning_rate: 0.05,
135
+ decay_factor: 0.5
136
+ )
137
+ ```
138
+
139
+ ### Reproducible Initialization
140
+ Each model initializes parameters randomly. By default, constructors use Ruby's global random number generator, preserving the historical behavior and honoring any external `srand` calls. For reproducible model initialization without resetting or consuming global RNG state, pass `seed:`:
141
+
142
+ ```ruby
143
+ model_a = IrtRuby::ThreeParameterModel.new(data, seed: 1234)
144
+ model_b = IrtRuby::ThreeParameterModel.new(data, seed: 1234)
145
+
146
+ # Same data, options, and seed produce identical fitted results.
147
+ model_a.fit == model_b.fit #=> true
148
+ ```
149
+
150
+ The `seed:` keyword is available for `RaschModel`, `TwoParameterModel`, and `ThreeParameterModel`.
151
+
152
+ ### Parameter Clamping
153
+ For 2PL and 3PL:
154
+
155
+ - **Discriminations** (`a`) are clamped between `0.01` and `5.0` during optimization.
156
+ - **Guessings** (`c`, 3PL only) are clamped to `[0.0, 0.35]`.
157
+
158
+ These bounds are model restrictions: a 3PL item with a true lower asymptote above `0.35` cannot be represented. The bounds apply to the internal optimization scale; returned discriminations are multiplied by the internal ability SD (see below), so they can lie outside `[0.01, 5.0]`.
159
+
160
+ ## Estimation Details
161
+
162
+ All models estimate person and item parameters jointly by maximizing the likelihood (joint maximum likelihood, JMLE).
163
+
164
+ - **Which parameters are estimated.** Think of the responses as a directed graph: a correct answer links the item to the respondent, an incorrect answer links the respondent to the item. Finite estimates on a common scale exist only within a strongly connected part of this graph, so IrtRuby estimates the largest such part and returns `nil` for every other respondent and item. This excludes respondents who answered every observed item correctly or every item incorrectly, items that everyone answered correctly or incorrectly, rows and columns with no observed responses, groups that are perfectly separated from the rest (for example, a group of respondents who answered every item of another group correctly), and groups that share no responses with the rest. Missing-data strategies are applied first, so `:treat_as_correct` and `:treat_as_incorrect` can create such patterns. For the Rasch model, and for 2PL with its bounded discriminations, this condition guarantees that finite estimates exist (the optimizer may still need more iterations to reach them). For 3PL it is necessary but not sufficient: fits can drift toward boundary solutions in which an item's difficulty keeps growing while its guessing parameter carries the item. Treat very large difficulties, or results with `converged: false`, as unreliable.
165
+ - **3PL and all-incorrect respondents.** Under 3PL, an excluded respondent who answered every estimated item incorrectly still informs the guessing parameters: as their ability tends to minus infinity each response has probability `1 - c`, and that limiting term stays in the likelihood.
166
+ - **Scale identification.** Results are reported on a fixed scale: Rasch abilities are centered to mean 0; 2PL and 3PL abilities are standardized to mean 0 and standard deviation 1, with difficulties and discriminations transformed so that response probabilities are unchanged. Only estimated respondents enter these statistics. The model keeps its internal parameters, so calling `fit` again continues from where it stopped; convergence is checked on the reported scale.
167
+ - **Known limitations.** JMLE item estimates are biased when tests are short, and the bias does not vanish as the number of respondents grows. 2PL and 3PL JMLE is also slow to converge with plain gradient ascent. For high-stakes calibration, compare against marginal maximum likelihood software such as R's `mirt`.
168
+
169
+ ## Performance Benchmarks
170
+
171
+ IRT Ruby includes comprehensive performance benchmarks to help you understand the computational characteristics of different models:
172
+
173
+ ```bash
174
+ # Run all benchmarks (takes 8-15 minutes)
175
+ bundle exec rake benchmark:all
176
+
177
+ # Quick performance check (2-3 minutes)
178
+ bundle exec rake benchmark:quick
179
+
180
+ # Individual benchmark suites
181
+ bundle exec rake benchmark:performance
182
+ bundle exec rake benchmark:convergence
183
+ ```
184
+
185
+ The benchmarks test:
186
+ - **Performance**: Execution speed across dataset sizes (50 to 100,000 data points)
187
+ - **Memory Usage**: Object allocation and memory efficiency
188
+ - **Scaling**: How computational complexity grows with data size
189
+ - **Convergence**: Optimization behavior under different conditions
190
+
191
+ See `benchmarks/README.md` for detailed information about interpreting results.
192
+
193
+ ## Development
194
+
195
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec` for the test suite or `bundle exec rake` for the default local quality gate: the RSpec suite and RuboCop. Before tagging a release, run `bundle exec rake build` to verify the gemspec packages cleanly from the current file list; CI also installs the built gem and smoke-tests `require 'irt_ruby'`. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
196
+
197
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
198
+
199
+ ## Contributing
200
+
201
+ Bug reports and pull requests are welcome on GitHub at https://github.com/SyntaxSpirits/irt_ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/SyntaxSpirits/irt_ruby/blob/main/CODE_OF_CONDUCT.md).
202
+
203
+ ## License
204
+
205
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
206
+
207
+ ## Code of Conduct
208
+
209
+ Everyone interacting in the IrtRuby project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [code of conduct](https://github.com/SyntaxSpirits/irt_ruby/blob/main/CODE_OF_CONDUCT.md).
@@ -0,0 +1,135 @@
1
+ # IRT Ruby Performance Benchmarks
2
+
3
+ This directory contains comprehensive performance benchmarks for the IRT Ruby gem, helping users understand the computational characteristics and scaling behavior of the different IRT models.
4
+
5
+ ## Available Benchmarks
6
+
7
+ ### 1. Performance Benchmark (`performance_benchmark.rb`)
8
+
9
+ **Purpose**: Comprehensive performance analysis across different dataset sizes and model types.
10
+
11
+ **What it measures**:
12
+ - Execution time (iterations per second) for Rasch, 2PL, and 3PL models
13
+ - Memory usage analysis (allocated/retained objects and memory)
14
+ - Scaling behavior analysis (how performance changes with dataset size)
15
+ - Impact of missing data strategies on performance
16
+
17
+ **Dataset sizes tested**:
18
+ - Tiny: 10 people × 5 items (50 data points)
19
+ - Small: 50 people × 20 items (1,000 data points)
20
+ - Medium: 100 people × 50 items (5,000 data points)
21
+ - Large: 200 people × 100 items (20,000 data points)
22
+ - XLarge: 500 people × 200 items (100,000 data points)
23
+
24
+ ### 2. Convergence Benchmark (`convergence_benchmark.rb`)
25
+
26
+ **Purpose**: Detailed analysis of convergence behavior and optimization characteristics.
27
+
28
+ **What it measures**:
29
+ - Impact of tolerance settings on convergence time and success rate
30
+ - Learning rate optimization analysis
31
+ - Dataset characteristics impact on convergence
32
+ - Missing data pattern effects on convergence
33
+
34
+ **Key insights provided**:
35
+ - Optimal hyperparameter settings for different scenarios
36
+ - Convergence reliability across different conditions
37
+ - Trade-offs between speed and accuracy
38
+
39
+ ## Running the Benchmarks
40
+
41
+ ### Prerequisites
42
+
43
+ Install benchmark dependencies:
44
+ ```bash
45
+ bundle install
46
+ ```
47
+
48
+ ### Running Individual Benchmarks
49
+
50
+ ```bash
51
+ # Full performance benchmark suite (takes 5-10 minutes)
52
+ ruby benchmarks/performance_benchmark.rb
53
+
54
+ # Convergence analysis (takes 3-5 minutes)
55
+ ruby benchmarks/convergence_benchmark.rb
56
+ ```
57
+
58
+ ### Running All Benchmarks
59
+
60
+ ```bash
61
+ # Run both benchmark suites
62
+ ruby benchmarks/performance_benchmark.rb && ruby benchmarks/convergence_benchmark.rb
63
+ ```
64
+
65
+ ## Understanding the Results
66
+
67
+ ### Performance Benchmark Output
68
+
69
+ 1. **Iterations per Second (IPS)**: Higher is better
70
+ - Shows relative speed between Rasch, 2PL, and 3PL models
71
+ - Includes confidence intervals and comparison ratios
72
+
73
+ 2. **Memory Usage**:
74
+ - Total allocated: Memory used during computation
75
+ - Total retained: Memory still held after computation
76
+ - Object counts: Number of Ruby objects created
77
+
78
+ 3. **Scaling Analysis**:
79
+ - Shows computational complexity (O(n^x))
80
+ - Helps predict performance for larger datasets
81
+
82
+ ### Convergence Benchmark Output
83
+
84
+ 1. **Convergence Rate**: Percentage of runs that converged within tolerance
85
+ 2. **Average Iterations**: Typical number of iterations needed
86
+ 3. **Time**: Wall-clock time to convergence
87
+
88
+ ## Interpreting Results for Your Use Case
89
+
90
+ ### For Educational Assessment (typical: 100-1000 students, 20-100 items)
91
+ - Focus on Medium to Large dataset results
92
+ - Rasch model typically fastest, 3PL slowest but most flexible
93
+ - Missing data strategies have < 10% performance impact
94
+
95
+ ### For Psychological Testing (typical: 50-500 participants, 10-50 items)
96
+ - Focus on Small to Medium dataset results
97
+ - All models should complete in < 1 second
98
+ - Consider convergence reliability for different tolerance settings
99
+
100
+ ### For Large-Scale Analysis (1000+ participants)
101
+ - Review XLarge dataset results and scaling analysis
102
+ - Consider batching or parallel processing for very large datasets
103
+ - Monitor memory usage to avoid system limits
104
+
105
+ ## Customizing Benchmarks
106
+
107
+ You can modify the benchmark scripts to test your specific scenarios:
108
+
109
+ 1. **Custom Dataset Sizes**: Edit `DATASET_CONFIGS` array
110
+ 2. **Different Hyperparameters**: Modify tolerance, learning rate configs
111
+ 3. **Specific Missing Data Patterns**: Adjust missing data generation
112
+ 4. **Model-Specific Tests**: Focus on particular IRT models
113
+
114
+ ## Performance Tips
115
+
116
+ Based on benchmark results:
117
+
118
+ 1. **Choose the Right Model**: Rasch is fastest, use 2PL/3PL only when needed
119
+ 2. **Optimize Tolerance**: `1e-5` typically good balance of speed/accuracy
120
+ 3. **Adjust Learning Rate**: Start with `0.01`, increase for faster convergence
121
+ 4. **Handle Missing Data**: `:ignore` strategy typically fastest
122
+ 5. **Consider Iteration Limits**: 100-500 iterations usually sufficient
123
+
124
+ ## Comparing with Other IRT Libraries
125
+
126
+ These benchmarks can help you compare IRT Ruby against other implementations. Key metrics to compare:
127
+
128
+ - Time per data point processed
129
+ - Memory efficiency
130
+ - Convergence reliability
131
+ - Scaling behavior with dataset size
132
+
133
+ ---
134
+
135
+ *Note: Benchmark results will vary based on your hardware. Run benchmarks on your target deployment environment for most accurate performance estimates.*
@@ -0,0 +1,225 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ require "bundler/setup"
5
+ require "irt_ruby"
6
+ require "benchmark"
7
+
8
+ def generate_data(num_people, num_items, difficulty_range: (-2..2), ability_range: (-2..2))
9
+ # Generate realistic IRT data based on known parameters
10
+ true_abilities = Array.new(num_people) { rand(ability_range) }
11
+ true_difficulties = Array.new(num_items) { rand(difficulty_range) }
12
+
13
+ data = Array.new(num_people) do |person|
14
+ Array.new(num_items) do |item|
15
+ prob = 1.0 / (1.0 + Math.exp(-(true_abilities[person] - true_difficulties[item])))
16
+ rand < prob ? 1 : 0
17
+ end
18
+ end
19
+
20
+ { data: data, true_abilities: true_abilities, true_difficulties: true_difficulties }
21
+ end
22
+
23
+ puts "=" * 70
24
+ puts "IRT Ruby Convergence Analysis"
25
+ puts "=" * 70
26
+ puts
27
+
28
+ # Test convergence with different tolerance settings
29
+ tolerance_configs = [
30
+ { tolerance: 1e-3, param_tolerance: 1e-3, label: "Loose (1e-3)" },
31
+ { tolerance: 1e-4, param_tolerance: 1e-4, label: "Medium (1e-4)" },
32
+ { tolerance: 1e-5, param_tolerance: 1e-5, label: "Tight (1e-5)" },
33
+ { tolerance: 1e-6, param_tolerance: 1e-6, label: "Very Tight (1e-6)" }
34
+ ]
35
+
36
+ dataset = generate_data(100, 50)
37
+ data = dataset[:data]
38
+
39
+ puts "Convergence Analysis - Impact of Tolerance Settings"
40
+ puts "-" * 50
41
+
42
+ tolerance_configs.each do |config|
43
+ puts "\nTolerance: #{config[:label]}"
44
+
45
+ times = []
46
+ iterations = []
47
+ convergences = []
48
+
49
+ 5.times do
50
+ time = Benchmark.measure do
51
+ model = IrtRuby::RaschModel.new(
52
+ data,
53
+ max_iter: 2000,
54
+ tolerance: config[:tolerance],
55
+ param_tolerance: config[:param_tolerance],
56
+ learning_rate: 0.01
57
+ )
58
+ result = model.fit
59
+ iterations << result[:iterations]
60
+ convergences << result[:converged]
61
+ end.real
62
+ times << time
63
+ end
64
+
65
+ avg_time = times.sum / times.size
66
+ avg_iterations = iterations.sum.to_f / iterations.size
67
+ convergence_rate = convergences.count(true) / 5.0
68
+
69
+ printf(" Time: %6.3fs Iterations: %6.1f Convergence Rate: %4.0f%%\n",
70
+ avg_time, avg_iterations, convergence_rate * 100)
71
+ end
72
+
73
+ # Test convergence with different learning rates
74
+ puts "\n#{"=" * 70}"
75
+ puts "Learning Rate Impact Analysis"
76
+ puts "-" * 50
77
+
78
+ learning_rate_configs = [
79
+ { rate: 0.001, label: "Very Slow (0.001)" },
80
+ { rate: 0.01, label: "Slow (0.01)" },
81
+ { rate: 0.05, label: "Medium (0.05)" },
82
+ { rate: 0.1, label: "Fast (0.1)" },
83
+ { rate: 0.2, label: "Very Fast (0.2)" }
84
+ ]
85
+
86
+ learning_rate_configs.each do |config|
87
+ puts "\nLearning Rate: #{config[:label]}"
88
+
89
+ times = []
90
+ iterations = []
91
+ convergences = []
92
+
93
+ 5.times do
94
+ time = Benchmark.measure do
95
+ model = IrtRuby::RaschModel.new(
96
+ data,
97
+ max_iter: 1000,
98
+ tolerance: 1e-5,
99
+ param_tolerance: 1e-5,
100
+ learning_rate: config[:rate]
101
+ )
102
+ result = model.fit
103
+ iterations << result[:iterations]
104
+ convergences << result[:converged]
105
+ end.real
106
+ times << time
107
+ end
108
+
109
+ avg_time = times.sum / times.size
110
+ avg_iterations = iterations.sum.to_f / iterations.size
111
+ convergence_rate = convergences.count(true) / 5.0
112
+
113
+ printf(" Time: %6.3fs Iterations: %6.1f Convergence Rate: %4.0f%%\n",
114
+ avg_time, avg_iterations, convergence_rate * 100)
115
+ end
116
+
117
+ # Test convergence with different dataset characteristics
118
+ puts "\n#{"=" * 70}"
119
+ puts "Dataset Characteristics Impact"
120
+ puts "-" * 50
121
+
122
+ dataset_configs = [
123
+ { people: 50, items: 25, diff_range: (-1..1), ability_range: (-1..1), label: "Easy (narrow ranges)" },
124
+ { people: 100, items: 50, diff_range: (-2..2), ability_range: (-2..2), label: "Medium (standard ranges)" },
125
+ { people: 100, items: 50, diff_range: (-3..3), ability_range: (-3..3), label: "Hard (wide ranges)" },
126
+ { people: 200, items: 100, diff_range: (-2..2), ability_range: (-2..2), label: "Large (more data)" }
127
+ ]
128
+
129
+ dataset_configs.each do |config|
130
+ puts "\nDataset: #{config[:label]}"
131
+
132
+ times = []
133
+ iterations = []
134
+ convergences = []
135
+
136
+ 3.times do
137
+ dataset = generate_data(
138
+ config[:people],
139
+ config[:items],
140
+ difficulty_range: config[:diff_range],
141
+ ability_range: config[:ability_range]
142
+ )
143
+
144
+ time = Benchmark.measure do
145
+ model = IrtRuby::RaschModel.new(
146
+ dataset[:data],
147
+ max_iter: 1000,
148
+ tolerance: 1e-5,
149
+ param_tolerance: 1e-5,
150
+ learning_rate: 0.01
151
+ )
152
+ result = model.fit
153
+ iterations << result[:iterations]
154
+ convergences << result[:converged]
155
+ end.real
156
+ times << time
157
+ end
158
+
159
+ avg_time = times.sum / times.size
160
+ avg_iterations = iterations.sum.to_f / iterations.size
161
+ convergence_rate = convergences.count(true) / 3.0
162
+
163
+ printf(" Time: %6.3fs Iterations: %6.1f Convergence Rate: %4.0f%%\n",
164
+ avg_time, avg_iterations, convergence_rate * 100)
165
+ end
166
+
167
+ # Test different missing data patterns
168
+ puts "\n#{"=" * 70}"
169
+ puts "Missing Data Pattern Impact"
170
+ puts "-" * 50
171
+
172
+ missing_configs = [
173
+ { rate: 0.0, strategy: :ignore, label: "No Missing Data" },
174
+ { rate: 0.1, strategy: :ignore, label: "10% Missing (ignore)" },
175
+ { rate: 0.2, strategy: :ignore, label: "20% Missing (ignore)" },
176
+ { rate: 0.2, strategy: :treat_as_incorrect, label: "20% Missing (incorrect)" },
177
+ { rate: 0.2, strategy: :treat_as_correct, label: "20% Missing (correct)" }
178
+ ]
179
+
180
+ missing_configs.each do |config|
181
+ puts "\nMissing Data: #{config[:label]}"
182
+
183
+ # Generate data with missing values
184
+ base_data = generate_data(100, 50)[:data]
185
+
186
+ data_with_missing = if config[:rate].positive?
187
+ base_data.map do |row|
188
+ row.map { |resp| rand < config[:rate] ? nil : resp }
189
+ end
190
+ else
191
+ base_data
192
+ end
193
+
194
+ times = []
195
+ iterations = []
196
+ convergences = []
197
+
198
+ 3.times do
199
+ time = Benchmark.measure do
200
+ model = IrtRuby::RaschModel.new(
201
+ data_with_missing,
202
+ max_iter: 1000,
203
+ tolerance: 1e-5,
204
+ param_tolerance: 1e-5,
205
+ learning_rate: 0.01,
206
+ missing_strategy: config[:strategy]
207
+ )
208
+ result = model.fit
209
+ iterations << result[:iterations]
210
+ convergences << result[:converged]
211
+ end.real
212
+ times << time
213
+ end
214
+
215
+ avg_time = times.sum / times.size
216
+ avg_iterations = iterations.sum.to_f / iterations.size
217
+ convergence_rate = convergences.count(true) / 3.0
218
+
219
+ printf(" Time: %6.3fs Iterations: %6.1f Convergence Rate: %4.0f%%\n",
220
+ avg_time, avg_iterations, convergence_rate * 100)
221
+ end
222
+
223
+ puts "\n#{"=" * 70}"
224
+ puts "Convergence Analysis Complete!"
225
+ puts "=" * 70