env_mem 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7e90ecdef2c4f4bb09bd84c7bb0baadbc5371660
+  data.tar.gz: 76d080f3e473482803108641ce8b6e65ab1a50f6
+SHA512:
+  metadata.gz: 487540c9d340ab69397739b8c820fccb20f7cab07878026c22972d73f6e4991eec957f4bb7c277d1ca7c013eb09f4b948e0693fbc6981a70df465cacf671ece1
+  data.tar.gz: c415f78104f2cec46cb5c9fac45cef63de6765f0d0b3556df93c42280400fd293afdcd3be5b972894a7214652f48e1366b86725479100dc7e5d43ff1d29d0659

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.14.6

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at the.codefolio.guy@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in env_mem.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Noah Gibbs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,204 @@
+# EnvMem
+
+Ever read a web page about how to set your Ruby memory environment
+variables and thought, "but how do I know that's right for my app?"
+EnvMem is here to help you out.
+
+Specifically, if you have a long-running or high-memory Ruby process
+(server, batch, etc) then your process will do more garbage collecting
+than is necessary in getting up to its long-term size. You can save a
+bit of time and processor by setting its environment variables close
+to their steady-state or end-of-process values.
+
+This is the same thing you do when you set Ruby environment variables
+to more Rails-friendly, batch-friendly or your-server-friendly values
+from a web page. It's just that this way you can make sure it's a good
+match for your app, specifically.
+
+EnvMem generates a small, simple shellscript to set your environment
+variable values. To use it, just source the script before running your
+application. You can manually tweak it later if you like, or remove
+variables you don't want to set for some reason - such as OLDMALLOC
+values if you've compiled a Ruby without it, for instance.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'env_mem'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install env_mem
+
+## Usage
+
+EnvMem needs a dump of GC.stat values from your application in the
+configuration you want to match. If you have a long-running Rails
+server, that means after it has processed a bunch of HTTP requests. If
+you're using EnvMem to configure your batch script, that probably
+means dumping GC.stat after you've finished your batch work and your
+job's memory configuration is nice and stable.
+
+Since you'll need the GC.stat values from the process, you'll need to
+dump them. First, here's how to do it *without* EnvMem:
+
+~~~ ruby
+File.open("gc_stat_dump.txt", "w") { |f| f.write GC.stat.inspect }
+~~~
+
+You can use EnvMem itself to dump GC.stat, but then you're using it at
+runtime. Here's how:
+
+~~~ ruby
+require 'env_mem'
+EnvMem.dump_to_file("gc_stat_dump.txt")
+~~~
+
+To create the environment script from the stat dump, translate from one filename to another:
+
+~~~ ruby
+$ env_mem gc_stat_dump.txt > env_wrapper.sh
+~~~
+
+Keep in mind that your application may change over time, and so it may
+need different memory settings. A simple way to handle that is to run
+your app *without* any Ruby memory environment variables set and then
+dump GC.stat again and regenerate them.
+
+### Long-Running Servers and Other Challenges
+
+There isn't always an obvious way to get statistics at the start and
+the end of the process. Start is usually easy, but end can be a
+challenge. Here's something I've tried with a large Rails server that
+has worked okay:
+
+~~~ ruby
+pid = Process.pid
+File.open("gc_stats_#{pid}_start.txt", "w") { |f| f.print GC.stats.inspect }
+at_exit {
+  File.open("gc_stats_#{pid}_stop.txt", "w") { |f| f.print GC.stats.inspect }
+}
+~~~
+
+The "at_exit" block is saying that before the process exits, it should
+stop and write out the GC stats again. Doing this during teardown
+means you won't necessarily have an accurate count of how many active
+objects are currently sitting around... But most of your statistics
+will work great.
+
+You can get a tiny bit of extra accuracy by instead adding a
+controller action to dump the GC stats while the Rails server is still
+fully active. But for most purposes, this will do just fine.
+
+### What the Variables Mean
+
+Ruby has two obvious thresholds, "malloc" and "oldmalloc", that keep
+going up. The "malloc" limit is so that Ruby garbage-collects
+regularly every so many bytes allocated. The "oldmalloc" limit is to
+garbage collect as (its estimate of) the old-generation size in bytes
+increases.
+
+Ordinarily a Ruby process will increase in size asymptotically,
+approaching its "full size." This is common for things like server
+processes that add and retain long-term memory (e.g. classes, caches)
+while adding a much smaller amount of per-request memory that gets
+garbage collected soon after the request is finished.
+
+After each time the limit causes a major garbage collection (e.g. the
+total allocated size crosses the "malloc" limit), that limit is raised
+by a configurable "growth factor". For instance, with the default
+RUBY\_GC\_MALLOC\_LIMIT\_GROWTH\_FACTOR of 1.4, the malloc limit will
+get 40% bigger each time. With a growth factor of 1.6, it would get
+60% bigger. There can also be a LIMIT_MAX variable, so that the limit
+grows by the smaller of the growth factor or the limit max. For
+instance, with a growth factor of 1.6 and a limit max of 100,000, Ruby
+would grow its malloc limit by 60% each time until 60% was bigger than
+100,000, and then it would grow by 100,000 each time.
+
+Slots are slightly different than the malloc and oldmalloc limits -
+slots are fully managed by Ruby itself, while Ruby uses a system
+allocator to managed the malloc and oldmalloc systems.
+
+With slots, Ruby starts with RUBY\_GC\_HEAP\_INIT\_SLOTS of them
+allocated. Slots also have a growth factor
+(RUBY\_GC\_HEAP\_GROWTH\_FACTOR) and a maximum growth
+(RUBY\_GC\_HEAP\_GROWTH\_MAX\_SLOTS). But Ruby will only use them if
+you don't set ratios of free slots (see below.) By default, Ruby will
+aim for 40% of slots free, allocating more to reach this ratio. By
+default it will free pages of slots when at least 65% of its slots are free.
+
+Here is a list of the variables in question:
+
+* RUBY\_GC\_HEAP\_INIT\_SLOTS - initial number of slots
+* RUBY\_GC\_HEAP\_FREE\_SLOTS - minimum free slots allowable after GC
+* RUBY\_GC\_HEAP\_GROWTH\_FACTOR - growth factor for slots
+* RUBY\_GC\_HEAP\_GROWTH\_MAX\_SLOTS - maximum slots to add at one time
+* RUBY\_GC\_HEAP\_FREE\_SLOTS\_MIN\_RATIO - allocate additional slots when below this ratio
+* RUBY\_GC\_HEAP\_FREE\_SLOTS\_MAX\_RATIO - free pages of slots when  above this ratio
+* RUBY\_GC\_HEAP\_FREE\_SLOTS\_GOAL\_RATIO - allocate slots to get to this ratio free (if 0.0, use the growth factor)
+
+* RUBY\_GC\_HEAP\_OLDOBJECT\_LIMIT\_FACTOR - do a major GC when the
+  number of old objects is above this factor times the old objects
+  after the *last* major GC.
+
+* RUBY\_GC\_MALLOC\_LIMIT
+* RUBY\_GC\_MALLOC\_LIMIT\_MAX
+* RUBY\_GC\_MALLOC\_LIMIT\_GROWTH\_FACTOR
+
+* RUBY\_GC\_OLDMALLOC\_LIMIT
+* RUBY\_GC\_OLDMALLOC\_LIMIT\_MAX
+* RUBY\_GC\_OLDMALLOC\_LIMIT\_GROWTH\_FACTOR
+
+### Limitations
+
+There are a *lot* of things you can do with the Ruby environment
+variables, and many different applications with different needs. Right
+now, EnvMem tries to do a bit to help you. But there's always room for
+more.
+
+(You can view these as limitations in EnvMem. You can also view them
+as places *you* can begin optimization. Both are correct.)
+
+For instance:
+
+EnvMem doesn't try to preserve environment variable settings from
+before you ran it. If you changed any of the "growth factors," for
+instance, EnvMem won't currently change them. You may also want to
+reduce the growth factors for a fully mature application, or set some
+of the LIMIT\_MAX environment variables so that your app can't bloat as
+quickly. EnvMem won't do that for you either since it's so
+application-specific what "reasonable" behavior is.
+
+EnvMem also tries not to assert anything about the balance of old- and
+new-generation objects. In a tightly-optimized application you'd
+expect old objects to dominate, while an application that generates a
+lot of transient garbage may need different settings. It's possible to
+balance MALLOC\_LIMIT settings with OLDMALLOC\_LIMIT settings to
+affect this, but EnvMem doesn't try to.
+
+Similarly, you may want a much smaller FREE\_SLOTS ratio with a more
+mature, more tightly-tuned application. EnvMem doesn't look at this,
+either.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/noahgibbs/env_mem. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "env_mem"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/env_mem.gemspec

@@ -0,0 +1,27 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'env_mem/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "env_mem"
+  spec.version       = EnvMem::VERSION
+  spec.authors       = ["Noah Gibbs"]
+  spec.email         = ["the.codefolio.guy@gmail.com"]
+
+  spec.summary       = %q{EnvMem takes a GC.stat dump and produce a memory-optimized script for your large Ruby app.}
+  spec.description   = %q{EnvMem allows you to dump your GC.stat information from a long-running server, then produce a simple shellscript to set up your Ruby memory environment variables to match that configuration. This improves startup time slightly and allows you finer-grain control over your memory setup. If you've ever found Ruby memory environment variables on a web page and used them to try to speed up your app, this is a better approach.}
+  spec.homepage      = "https://github.com/noahgibbs/env_mem"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.14"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+end

data/exe/env_mem

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+if ARGV.size < 1
+  STDERR.puts "You must supply a filename for the GC.stat dump!"
+  exit -1
+end
+
+devel_dir = File.join(__dir__, "..", "lib")
+if File.exist? devel_dir
+  $LOAD_PATH.unshift devel_dir
+end
+require "env_mem"
+
+out = EnvMem.gc_stat_to_shell File.read(ARGV[0]).chomp
+
+puts out

data/exe/gc_stat_dump.txt

@@ -0,0 +1 @@
+{:count=>11, :heap_allocated_pages=>132, :heap_sorted_length=>133, :heap_allocatable_pages=>0, :heap_available_slots=>53801, :heap_live_slots=>49948, :heap_free_slots=>3853, :heap_final_slots=>0, :heap_marked_slots=>21045, :heap_swept_slots=>26874, :heap_eden_pages=>123, :heap_tomb_pages=>9, :total_allocated_pages=>132, :total_freed_pages=>0, :total_allocated_objects=>191166, :total_freed_objects=>141218, :malloc_increase_bytes=>4240, :malloc_increase_bytes_limit=>16777216, :minor_gc_count=>8, :major_gc_count=>3, :remembered_wb_unprotected_objects=>201, :remembered_wb_unprotected_objects_limit=>378, :old_objects=>20381, :old_objects_limit=>36098, :oldmalloc_increase_bytes=>463088, :oldmalloc_increase_bytes_limit=>16777216}

data/lib/env_mem.rb

@@ -0,0 +1,26 @@
+require "env_mem/version"
+
+module EnvMem
+  extend self
+
+  def dump_to_file(filename)
+    File.open(filename, "w") { |f| f.write GC.stat.inspect }
+  end
+
+  def gc_stat_to_shell(stats)
+    stats_hash = {}
+    stats.scan(/:([a-zA-Z_]+)\s*=>\s*([0-9]+)/).each { |key, val| stats_hash[key] = val.to_i }
+
+    <<SHELL
+# gc_params.heap_init_slots
+export RUBY_GC_HEAP_INIT_SLOTS=#{stats_hash["heap_live_slots"]}
+
+# gc_params.malloc_limit_min
+export RUBY_GC_MALLOC_LIMIT=
+
+# gc_params.oldmalloc_limit_min
+export RUBY_GC_OLDMALLOC_LIMIT=
+
+SHELL
+  end
+end

data/lib/env_mem/version.rb

@@ -0,0 +1,3 @@
+module EnvMem
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,108 @@
+--- !ruby/object:Gem::Specification
+name: env_mem
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Noah Gibbs
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-05-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.14'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+description: EnvMem allows you to dump your GC.stat information from a long-running
+  server, then produce a simple shellscript to set up your Ruby memory environment
+  variables to match that configuration. This improves startup time slightly and allows
+  you finer-grain control over your memory setup. If you've ever found Ruby memory
+  environment variables on a web page and used them to try to speed up your app, this
+  is a better approach.
+email:
+- the.codefolio.guy@gmail.com
+executables:
+- env_mem
+- gc_stat_dump.txt
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- env_mem.gemspec
+- exe/env_mem
+- exe/gc_stat_dump.txt
+- lib/env_mem.rb
+- lib/env_mem/version.rb
+homepage: https://github.com/noahgibbs/env_mem
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.14
+signing_key: 
+specification_version: 4
+summary: EnvMem takes a GC.stat dump and produce a memory-optimized script for your
+  large Ruby app.
+test_files: []