gargor 0.0.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,8 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in gargor.gemspec
+gemspec
+
+require 'json'
+require 'jsonpath'
+

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Yoshihiro TAKAHARA
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,171 @@
+# gargor
+
+`gargor`はChefで管理されたサーバのパラメータと負荷試験結果を遺伝的アルゴリズム(genetic algorithm)により機械学習し、最適値を探索します。本ソフトウェアをうまく使うことで今まで勘と経験に頼っていたサーバチューニングをより最適にかつ自動化することができます。
+
+## インストール
+
+Ruby 1.9以降が必要です
+
+    $ gem install gargor
+    
+## どのように動くのか
+
+1. 現在のChefの設定ファイル(JSON)から個体を一つ作ります。
+2. 残りの個体を突然変異により作ります。
+3. 各個体に対し、負荷試験を実施し、適応値を算出します。
+4. 現世代の個体群に対して、エリートと残りを交叉および突然変異により次世代の個体群をつくります。
+5. 次世代の個体群を現世代として`3.`に戻ります。これを指定した世代分実施します。
+6. 最後に最も高い適応値の個体をサーバに配備して終了します
+
+## 使い方
+
+    $ gargor [dsl-file]
+    
+
+    `dsl-file`を省略した場合は、カレントディレクトリの`gargor.rb`を探します。
+
+    dsl-fileの書き方は、添付の`doc/sample.rb`を御覧ください
+    
+
+### 注意
+
+`gargor`は、DSLファイルで指定されたChefのJSONを直接（問答無用で）書き換えます。`git stash`を使うなりしてオリジナルが消えないように配慮ください。
+
+## サンプルレシピ
+
+`Chef`のサンプルレシピをTipsを交えてご紹介します。
+
+### mysql
+
+
+```ruby
+# この時点で起動しようとすると以前の設定ファイルがおかしい場合にエラーが出てしまう
+service "mysqld" do
+  action :nothing
+end
+
+template "/etc/my.cnf" do
+  source "www/my.cnf.erb"
+  notifies :restart,"service[mysqld]"
+end
+
+# ib_logfile0,1はib_logfile1はinnodb_log_file_sizeを変えるとエラーになるので毎回消す
+file "/var/lib/mysql/ib_logfile0" do
+  action :delete
+  notifies :restart,"service[mysqld]"
+end
+file "/var/lib/mysql/ib_logfile1" do
+  action :delete
+  notifies :restart,"service[mysqld]"
+end
+```
+
+このテンプレートは以下のようになっています。`innodb_buffer_pool_size`を探索対象のパラメータにしたいのですが、「`innodb_log_file_size`を`innodb_buffer_pool_size`の25%にすべし」と注意書きがあるので、`innodb_log_file_size`を設定対象にして、`innodb_buffer_pool_size`をその4倍にしています。
+
+```
+[mysqld]
+datadir=/var/lib/mysql
+socket=/var/lib/mysql/mysql.sock
+user=mysql
+# Default to using old password format for compatibility with mysql 3.x
+# clients (those using the mysqlclient10 compatibility package).
+old_passwords=1
+
+# Disabling symbolic-links is recommended to prevent assorted security risks;
+# to do so, uncomment this line:
+# symbolic-links=0
+
+skip-locking
+query_cache_size = <%= node["mysqld"]["query_cache_size"] %>M
+key_buffer = <%= node["mysqld"]["key_buffer"] %>K
+max_allowed_packet = <%= node["mysqld"]["max_allowed_packet"] %>M
+table_cache = <%= node["mysqld"]["table_cache"] %>
+sort_buffer_size = <%= node["mysqld"]["sort_buffer_size"] %>K
+read_buffer_size = <%= node["mysqld"]["read_buffer_size"] %>K
+read_rnd_buffer_size = <%= node["mysqld"]["read_rnd_buffer_size"] %>K
+net_buffer_length = <%= node["mysqld"]["net_buffer_length"] %>K
+thread_stack = <%= node["mysqld"]["thread_stack"] %>K
+
+skip-networking
+server-id	= 1
+
+# Uncomment the following if you want to log updates
+#log-bin=mysql-bin
+
+# Disable Federated by default
+skip-federated
+
+# Uncomment the following if you are NOT using BDB tables
+#skip-bdb
+
+# Uncomment the following if you are using InnoDB tables
+innodb_data_home_dir = /var/lib/mysql/
+innodb_data_file_path = ibdata1:10M:autoextend
+innodb_log_group_home_dir = /var/lib/mysql/
+innodb_log_arch_dir = /var/lib/mysql/
+# You can set .._buffer_pool_size up to 50 - 80 %
+# of RAM but beware of setting memory usage too high
+innodb_buffer_pool_size = <%= node["mysqld"]["innodb_log_file_size"]*4 %>M
+innodb_additional_mem_pool_size = <%= node["mysqld"]["innodb_additional_mem_pool_size"] %>M
+# Set .._log_file_size to 25 % of buffer pool size
+innodb_log_file_size = <%= node["mysqld"]["innodb_log_file_size"] %>M
+innodb_log_buffer_size = <%= node["mysqld"]["innodb_log_buffer_size"] %>M
+innodb_flush_log_at_trx_commit = 1
+innodb_lock_wait_timeout = 50
+
+[mysqld_safe]
+log-error=/var/log/mysqld.log
+pid-file=/var/run/mysqld/mysqld.pid
+```
+
+### httpd
+
+Apacheのパフォーマンスチューニングでは以下の様にしています。レシピには全く工夫はありません。
+
+```
+service "httpd" do
+  action :nothing
+end
+
+template "/etc/httpd/conf/httpd.conf" do
+  source "www/httpd.conf.erb"
+  notifies :restart,"service[httpd]"
+end
+```
+
+ポイントは`MinSpareServers`と`MaxSpareServers`のように上下関係のあるパラメータの決定方法を以下のように`min_spare_servers`と`range_spare_servers`としている点です。また、`ServerLimit`=`MaxClients`としています。
+
+```
+<IfModule prefork.c>
+StartServers     <%= node["httpd"]["start_servers"] %>
+MinSpareServers  <%= node["httpd"]["min_spare_servers"] %>
+MaxSpareServers  <%= node["httpd"]["min_spare_servers"] + node["httpd"]["range_spare_servers"] %>
+ServerLimit      <%= node["httpd"]["max_clients"] %>
+MaxClients       <%= node["httpd"]["max_clients"] %>
+MaxRequestsPerChild  <%= node["httpd"]["max_request_per_child"] %>
+</IfModule>
+```
+
+## 負荷試験ツール
+
+コマンドラインで使えるものであれば、大体使うことができます。サンプルでは、[グリーン破壊](https://github.com/KLab/green-hakai/)を使わせて頂きました。
+
+負荷試験の厳しさによって個体が死滅することがあります。すると以下のように表示され、プログラムが終了します。
+
+    ***** EXTERMINATION ******
+    
+これは、個体の環境が厳しすぎるためで負荷試験を緩めて再度実施してください。    
+
+## FAQ
+
+#####  `gargor`はなんて読むの?
+
+「がるごる」
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/gargor

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__) + '/../lib')
+require 'gargor'
+
+dsl_file=ARGV.last||"gargor.rb"
+
+Gargor.start
+Gargor.load_dsl(dsl_file)
+loop {
+  Gargor.populate.each { |i|
+    if i.fitness == nil
+      i.set_params
+      if i.deploy
+        i.attack
+      end
+    end
+  }
+  break unless Gargor.next_generation
+}
+
+best = Gargor.individuals.max { |a,b| a.fitness <=> b.fitness }
+if best
+  best.set_params
+  best.deploy
+end
+p best

data/gargor.gemspec

@@ -0,0 +1,23 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'gargor/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "gargor"
+  spec.version       = Gargor::VERSION
+  spec.authors       = ["Yoshihiro TAKAHARA"]
+  spec.email         = ["y.takahara@gmail.com"]
+  spec.description   = %q{auto server tunner by generic algorithm}
+  spec.summary       = %q{auto server tunner by generic algorithm using chef}
+  spec.homepage      = "https://github.com/tumf/gargor"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+end

data/lib/gargor.rb

@@ -0,0 +1,154 @@
+# -*- coding: utf-8 -*-
+require "gargor/version"
+require "gargor/individual"
+require "gargor/parameter"
+
+class Gargor
+  GLOBAL_OPTS = ["population","max_generations","target_nodes",
+                 "attack_cmd","elite","mutation","target_cooking_cmd",
+                 "attack_node","fitness_precision","attack_result"]
+
+  GLOBAL_OPTS.each { |name| 
+    define_method(name) { |val|
+      Gargor.class_variable_set("@@#{name}", val)
+    }
+  }
+
+  class << self
+    def start
+      @@fitness_precision = 100000000
+      @@prev_generation = nil
+      @@individuals = []
+      @@param_procs = {}
+      @@population = 10
+      @@max_generations = 1
+      @@generation = 1
+      @@elite = 0
+      @@attack_cmd = "false"
+      @@attack_proc = nil
+      @@evaluate_proc = nil
+    end
+
+    def load_dsl(params_file)
+      contents = File.open(params_file, 'rb'){ |f| f.read }
+      new.instance_eval(contents)
+    end
+
+    def mutation
+      individual = Individual.new
+      @@param_procs.each { |name,proc|
+        param =  Parameter.new(name)
+        param.instance_eval(&proc)
+        individual.params[name] = param
+      }
+      individual
+    end
+
+    # 浮動小数点対応のrand
+    def float_rand(f,p = @@fitness_precision)
+      f *= @@fitness_precision
+      i = f.to_i
+      f = rand(i)
+      f / @@fitness_precision.to_f
+    end
+
+    def crossover a,b
+      return a.clone if a.params == b.params
+      puts "crossover: #{a} #{b}"
+      total = a.fitness + b.fitness
+      c = Individual.new
+      c.params = a.params.clone
+
+      c.params.each { |name,param|
+        cur = float_rand(total)
+        c.params[name] = b.params[name] if b.fitness > cur
+      }
+      puts "#{a.to_s} + #{b.to_s} \n    => #{c.to_s}"
+      c
+    end
+
+    def selection g
+      total = g.inject(0) { |sum,i| sum += i.fitness }
+      cur = float_rand(total)
+      g.each { |i|
+        return i if i.fitness > cur
+        cur -= i.fitness
+      }
+      raise "error selection"
+    end
+
+    def populate
+      @@individuals = []
+      
+      # 第一世代
+      unless @@prev_generation
+        @@individuals << mutation.load_now
+        loop{
+          break if @@individuals.length >= @@population
+          @@individuals << mutation
+        }
+        return @@individuals
+      end
+
+      # fitness > 0 適応している個体
+      prev_count = @@prev_generation.select { |i| i.fitness > 0 }.count
+
+      if prev_count < 2
+        raise "***** EXTERMINATION ******"
+      end
+
+      puts "population: #{@@prev_generation.length}"
+      @@individuals = @@prev_generation.sort{ |a,b| a.fitness<=>b.fitness }.last(@@elite) if @@elite > 0
+      loop{
+        break if @@individuals.length >= @@population
+        if rand <= @@mutation
+          i =  mutation
+          puts "mutation #{i}"
+        else
+          a = selection @@prev_generation
+          b = selection @@prev_generation
+          i = crossover(a,b)
+        end
+
+        #同じのは追加しない
+        if i and !@@individuals.find { |ii| ii.params == i.params }
+          @@individuals << i 
+        end
+      }
+      puts "poulate: #{@@individuals}"
+      @@individuals
+    end
+
+
+    def next_generation
+      puts "<== end generation #{@@generation}"
+      @@generation += 1
+      return false if @@generation > @@max_generations
+
+      puts "==> next generation #{@@generation}"
+      @@prev_generation = @@individuals
+      true
+    end
+
+    def individuals
+      @@individuals
+    end
+
+    def opt name
+      Gargor.class_variable_get("@@#{name}")
+    end
+  end
+
+  def param name,&block
+    @@param_procs[name] = block
+  end
+
+  def attack cmd,&block
+    @@attack_cmd = cmd
+    @@attack_proc = block
+  end
+
+  def evaluate &block
+    @@evaluate_proc = block
+  end
+end

data/lib/gargor/individual.rb

@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+require 'json'
+require 'jsonpath'
+require 'benchmark'
+
+class Gargor
+  class Individual
+    attr_accessor :params,:fitness
+    def initialize
+      @params = {}
+      @fitness = nil
+    end
+
+    def to_s
+      [@params,@fitness].to_s
+    end
+
+    def load_now
+      @params.each { |name,param|
+        json = File.open(param.file).read
+        @params[name].value = JsonPath.on(json,param.path).first
+      }
+      self
+    end
+
+    def set_params
+      puts "==> set params"
+      puts @params
+      @params.each { |name,param|
+        json = File.open(param.file).read
+        json = JSON.pretty_generate(JsonPath.for(json).gsub(param.path) { |v| param.value }.to_hash)
+        # 書き出し
+        File.open(param.file,"w") { |f| f.write(json) }
+        puts "    write #{param.file}"
+      }
+    end
+
+    def deploy
+      ret = true
+      Gargor.opt("target_nodes").each { |node|
+        puts "==> deploy to #{node}"
+        cmd = Gargor.opt("target_cooking_cmd") % [node]
+        puts "    #{cmd}"
+        r = system(cmd)
+        unless r
+          puts "deploy failed"
+          @fitness = 0
+          sleep 1
+        end
+        ret &= r
+      }
+      ret
+    end
+
+
+    def attack
+      ret = nil;out = nil
+      cmd = Gargor.opt('attack_cmd')
+      puts "==> attack"
+      puts "execute: #{cmd}"
+      tms = Benchmark.realtime do
+        out = `#{cmd}`
+        ret = $?
+      end
+
+      @fitness = Gargor.opt('evaluate_proc').call(ret.to_i,out,tms)
+      puts "fitness: #{@fitness}"
+      @fitness
+    end
+  end
+end

data/lib/gargor/parameter.rb

@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+class Gargor
+  class Parameter
+    attr_accessor :file, :path, :value, :name
+    alias :json_file :file=
+    alias :json_path :path=
+    alias :mutation :value=
+    alias_method :eql, :==
+    def initialize name
+      @name = name
+    end
+    def to_s
+      @value.to_s
+    end
+
+    def ==(other)
+      name == other.name and
+        file == other.file and
+        path == other.path and
+        value == other.value
+    end
+
+  end
+end

data/lib/gargor/version.rb

@@ -0,0 +1,3 @@
+class Gargor
+  VERSION = "0.0.1"
+end

metadata

@@ -0,0 +1,90 @@
+--- !ruby/object:Gem::Specification
+name: gargor
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Yoshihiro TAKAHARA
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-06-16 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: auto server tunner by generic algorithm
+email:
+- y.takahara@gmail.com
+executables:
+- gargor
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/gargor
+- gargor.gemspec
+- lib/gargor.rb
+- lib/gargor/individual.rb
+- lib/gargor/parameter.rb
+- lib/gargor/version.rb
+homepage: https://github.com/tumf/gargor
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.25
+signing_key: 
+specification_version: 3
+summary: auto server tunner by generic algorithm using chef
+test_files: []