newral 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 281bfa9ba74cbd19d659513e73eae475f6f1ec7e
+  data.tar.gz: 99a0085f6bc6a1b6a38237dc330110950df04f19
+SHA512:
+  metadata.gz: 63af6de490824297c3561b6e1c77f1681d099e17e38323fbd15080b4447a08abe8fd83c257a8e1c2936e08591260798895cf526af2288dda188ad12490b3e19d
+  data.tar.gz: 786ff86c34e25ad4bbbb4cfd322c0d40efa39308778ab18efd6641bb3126e62e3ad6af02ecc608bc2225fcbb66519d15645e8a603d744e7feb9110d61a8401f1

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in newral.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 ExistsAndIs1
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,278 @@
+# Newral
+
+I recently started to learn about AI. 
+Of course there are great libraries out there but I wanted to have something that makes it easy to test the different concepts to really understand them.
+Also I wanted to have a playground to easily see how good different approaches work for different data sets.
+I chose the name newral as its for newbies trying out neural networks and other AI related concepts
+
+In the implementation I tried to write as little code as possible and used classes trying to avoid "array index hell".
+So the data structures are in no way tuned for efficiency, rather I tried to make clear what actually is going on.
+For every concept there should be at least one test to show it in action.
+
+
+Everything is still quite early stages but there are a lot of things you can do already
+
+* Training Functions 
+  * Hill Climbing
+  * Greedy
+  * Gradient Descent 
+
+* K-Means Clustering
+* K-Nearest Neighbour
+
+* Neural Networks 
+  * Easily define simple ones often used in Tutorials
+  * Backpropagation 
+
+* Graphs 
+  * Tree Search
+  * Cheapest First
+  * A Star 
+
+* Q-Learning
+Learn the computer to play Tic-Tac Toe (or other simple games )
+
+I must say that this is really a total side project for me, so don´t expect lots of updates or bugfixes.
+Whenever I thought about it there are links to the tutorials or websites I used (which will explain the theory much better than I ever could).
+Please check out the tests where there are a few examples of possible use cases.
+
+Stuff still in even earlier stages 
+* everything in genetic folder
+* bayes / probability 
+
+
+So lets do some basic stuff
+
+## Error Calculation
+
+lets assume we have 3 calculated results by our function and 3 expected outputs 
+```ruby
+current =  [1,2,3]
+expected = [2,4,6]
+
+so what´s the error
+
+Newral::ErrorCalculation.mean_square( current, expected  )
+```
+
+same thing for vectors 
+```ruby
+ current = [
+        [1,2,3],
+        [3,9,16]
+        
+      ]
+
+      expected = [
+        [2,4,6],
+        [4,8,9]
+      ]
+
+Newral::ErrorCalculation.mean_square( current, expected  )
+```
+
+## Classifiers
+
+```ruby
+points = [
+  [1,1],[2,2],[4,4],
+  [10,9],[11,12],[13,7]
+].shuffle
+
+n= Newral::Classifier::KMeansCluster.new( points, cluster_labels:[:cows,:elefants] ).process
+n.clusters[:elefants].points
+n.clusters[:cows].points
+
+n=Newral::Classifier::Dendogram.new( points ).process
+n.to_s
+```
+
+## Neural Networks 
+
+### create some neurons 
+
+```ruby
+perceptron = Newral::Networks::Perceptron.new(weights: [-2,-2],bias: 3) # look its a NAND gate
+perceptron.update_with_vector [1,1]
+
+sigmoid = Newral::Networks::Sigmoid.new(weights: [-2,-2],bias: 3) # sigmoids are much cooler 
+sigmoid.update_with_vector [1,1]
+```
+
+### create a basic network 
+```ruby
+      network = Newral::Networks::Network.define do 
+        add_layer "input" do 
+          add_neuron 'a', weights: [-2,-2],bias: 3, type: 'perceptron'
+          add_neuron 'b', weights: [-2,-2],bias: 3, type: 'perceptron'
+        end 
+        add_layer "output" do 
+          add_neuron 'c', weights: [-2,-2],bias: 3, type: 'perceptron'
+        end 
+
+        connect from:'a', to:'c'
+        connect from:'b', to:'c'
+      end
+
+      network.update_with_vector [1,1]
+```
+
+### create a network and perform backpropagation 
+```ruby
+inputs = [
+        [0.05,0.1]
+      ]
+      outputs = [
+        [0.01,0.99]
+      ]
+      network = Newral::Networks::BackpropagationNetwork.new( number_of_hidden: 2, number_of_outputs: 2)
+      network.set_weights_and_bias( layer: 'hidden', weights:[[0.15,0.2],[0.25,0.3]],bias:[0.35,0.35])
+      network.set_weights_and_bias( layer: 'output', weights:[[0.4,0.45],[0.5,0.55]], bias:[0.6,0.6])
+      network.calculate_error( input: inputs, output: outputs ) # stupid network
+      1000.times do 
+        network.train input: inputs , output:outputs
+      end 
+
+      network.calculate_error( input: inputs, output: outputs ) # look it learned
+```
+
+
+
+## Load some data 
+
+load the IRIS data set (Hello World of AI) located in test folder
+```ruby
+data = Newral::Data::Csv.new(file_name:File.expand_path('../test/fixtures/IRIS.csv',__FILE__))
+data.process
+cluster_set = Newral::Classifier::KMeansCluster.new( data.inputs, cluster_labels: data.output_hash.keys ).process
+cluster_set.clusters.length # There are 3 different types 
+```
+
+```ruby
+data = Newral::Data::Csv.new(file_name:File.expand_path('../test/fixtures/IRIS.csv',__FILE__))
+data.process
+
+network = Newral::Networks::BackpropagationNetwork.new( number_of_inputs: data.inputs.first.size, number_of_hidden: data.inputs.first.size, number_of_outputs: data.output_hash.keys.size )
+network.calculate_error( input: data.inputs, output: data.output_as_vector ) # using a network with random weights  
+100.times do
+  network.train( input: data.inputs, output: data.output_as_vector ) # Hard training is the key to success in any neural nets life
+end 
+network.calculate_error( input: data.inputs, output: data.output_as_vector ) # hey it now knows flowers better than me!
+```
+
+Of course we don´t want oversampling so we should train and test on different data sets
+
+```ruby
+data = Newral::Data::Csv.new(file_name:File.expand_path('../test/fixtures/IRIS.csv',__FILE__))
+data.process
+
+network = Newral::Networks::BackpropagationNetwork.new( number_of_inputs: data.inputs.first.size, number_of_hidden: data.inputs.first.size, number_of_outputs: data.output_hash.keys.size )
+network.calculate_error( input: data.sub_set(set: :inputs, category: :validation ), output: data.output_as_vector( category: :testing ) ) 
+
+100.times do
+  network.train( input: data.sub_set(set: :inputs, category: :training ), output: data.output_as_vector( category: :training ) ) 
+end 
+
+network.calculate_error( input: data.sub_set(set: :inputs, category: :validation ), output: data.output_as_vector( category: :testing ) ) 
+
+```
+
+here comes the heavy stuff for this little library, load the MNIST data set (60000 images with 28*28 pixels).
+You can read more about MNIST http://yann.lecun.com/exdb/mnist/
+```ruby
+data = Newral::Data::Idx.new( file_name:'~/Downloads/train-images-idx3-ubyte', label_file_name:'~/Downloads/train-labels-idx1-ubyte')
+data.process
+
+sample_data = data.sample( limit:100 ) 
+sample_data.downsample_input!( width:2,height:2,width_of_line:28 ) # create less resolution pictures
+
+sample_data2 = data.sample( limit:100, offset:100  ) # a 2bd sample
+sample_data2.downsample_input!( width:2,height:2,width_of_line:28 )
+
+
+network = Newral::Networks::BackpropagationNetwork.new( number_of_inputs: sample_data.inputs.first.size, number_of_hidden: sample_data.inputs.first.size, number_of_outputs: sample_data.output_hash.keys.size )
+
+# lets compare the error of a random network vs one trained one
+network.calculate_error( input: sample_data2.inputs, output: sample_data2.output_as_vector )
+
+# use first sample to train
+network.train( input: sample_data.inputs, output: sample_data.output_as_vector )
+
+# now calculate the error of untrained sample
+# it should still go down
+network.calculate_error( input: sample_data2.inputs, output: sample_data2.output_as_vector )
+
+```
+
+
+## use a tree Search to find the fastest path from Arad to Bucharest
+```ruby
+edges,nodes,node_locations = setup_bulgarian_map # find this in the test folder 
+      g = Newral::Graphs::Graph.new 
+      g.add_nodes nodes
+      g.add_edges edges
+      t=Newral::Graphs::CheapestFirst.new( graph: g, start_node: 'Arad', end_node:'Bucharest')
+      path = t.run
+      path.cost
+```
+
+## Use QLearning to play Tic Tac Toe
+
+as we know good players will always reach a draw
+
+```ruby
+require './test/games/tic_tac_toe_game'
+
+  game = TicTacToeGame.new  # ( width: 8, height: 6, in_a_row: 4 )
+      player1 = Newral::QLearning::Base.new( game: game, id: 0 )
+      player2 = Newral::QLearning::Base.new( game: game, id: 1 )
+      # training
+      1000.times do
+        game.run
+        game.reset
+      end
+      game.reset( reset_score: 1 )
+      player1.set_epsilon 1 # stop doing random moves, we know the game
+      player2.set_epsilon 1
+
+   game.run # => its a draw  
+```
+
+## Use Training Algorithms to best approximate data with a function
+Many typical functions suited for such approximations are already there
+```ruby
+f= Newral::Functions::Vector.new vector: [1,6], bias:1 
+f.calculate [4,7] #  4*1+6*7+1 => 47
+ 
+
+Newral::Functions::Polynomial.new factors: [2,5,1] 
+f.calculate 2 # 2*(2**2)+5*2+1 => 19
+```
+
+first lets use a basic polynominal function 
+```ruby
+      input = [2,4,8]
+      output = [4,16,64] # best function is x**2, lets see if our training algorithms find them
+      g=Newral::Training::Greedy.new( input: input, output: output, klass: Newral::Functions::Polynomial )
+      g.process 
+      g.best_function.calculate_error( input: input, output: output )
+            
+      h=Newral::Training::HillClimbing.new( input: input, output: output, klass: Newral::Functions::Polynomial, start_function: g.best_function )
+      h.process 
+      h.best_function.calculate_error( input: input, output: output )
+      
+      # Gradient descent with error gradient approximation function 
+      d=Newral::Training::GradientDescent.new( input: input, output: output, klass: Newral::Functions::Polynomial )
+      d.process  
+      d.best_function.calculate_error( input: input, output: output )
+```
+
+now lets use a Vector
+```ruby
+
+      input = [[1,2],[2,4]]
+      output=[3,7]
+      g=Newral::Training::GradientDescent.new( input: input, output: output, klass: Newral::Functions::Vector )
+      g.process
+      g.best_function.calculate_error( input: input, output: output )
+  ```

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/lib/newral.rb

@@ -0,0 +1,53 @@
+module Newral
+  require "matrix"
+  require "nmatrix"
+  require "newral/tools"
+
+  require "newral/data/base"
+  require "newral/data/csv"
+  require "newral/data/idx"
+  require "newral/data/cluster"
+  require "newral/data/cluster_set"
+
+  require "newral/error_calculation"
+
+  require "newral/networks/perceptron"
+  require "newral/networks/sigmoid"
+  require "newral/networks/layer"
+  require "newral/networks/network"
+  require "newral/networks/backpropagation_network"
+
+#  require "newral/probability" 
+  require "newral/probability_set"
+#  require "newral/bayes"
+
+  require "newral/classifier/node"
+  require "newral/classifier/node_distance"
+  require "newral/classifier/dendogram"
+  require "newral/classifier/k_means_cluster"
+
+  require "newral/functions/base"
+  require "newral/functions/line"
+  require "newral/functions/vector"
+  require "newral/functions/block"
+  require "newral/functions/polynomial"
+  require "newral/functions/gaussian"
+  require "newral/functions/ricker_wavelet"
+  require "newral/functions/radial_basis_function_network"
+  
+  require "newral/training/greedy"
+  require "newral/training/hill_climbing"
+  require "newral/training/linear_regression"
+  require "newral/training/linear_regression_matrix"
+  require "newral/training/gradient_descent"
+  require "newral/q_learning/base"
+
+  require "newral/genetic/tree"
+  require "newral/graphs/node"
+  require "newral/graphs/edge"
+  require "newral/graphs/graph"
+  require "newral/graphs/path"
+  require "newral/graphs/tree_search"
+  require "newral/graphs/cheapest_first"
+  require "newral/graphs/a_star"
+end

data/lib/newral/bayes.rb

@@ -0,0 +1,39 @@
+module Newral
+  
+  class Bayes
+    attr_reader :theorem, :probabilities
+    def initialize( theorem )
+      @theorem = theorem
+      @probabilities = {}
+    end 
+
+    def add_probability(key,probability,apriori: nil)
+      probability = Probability.new(key,probability,apriori: apriori)
+      @probabilities[ probability.key ] = probability 
+    end
+
+    def compute( key )
+      probability = if @probabilities[key]
+        @probabilities[key]
+      elsif key.start_with?("!") && @probabilities[key.sub("!","")]
+        !@probabilities[key.sub("!",'')]
+      elsif key.match('\|')
+        key,apriori=key.split("|")
+        compute("#{apriori}|#{key}")*compute(key)/compute(apriori)
+      else
+        apriori = @probabilities.keys.find{|p| p.split("|")[0]==key && !p.split("|")[1].match('!') }
+        if apriori
+          apriori = apriori.split("|")[1]
+          compute("#{key}|#{apriori}")*compute(apriori)+compute("#{key}|!#{apriori}")*compute("!#{apriori}")
+        else 
+          puts "not found #{key}"
+        end
+      end
+      @probabilities[ probability.key ] = probability
+      probability
+    end
+
+  end
+
+
+end

data/lib/newral/classifier/dendogram.rb

@@ -0,0 +1,68 @@
+module Newral
+
+  module Classifier
+        
+    class Dendogram 
+      attr_reader :nodes, :max_distance, :distances
+      def initialize( points, max_runs: 100, abort_at_distance: 0.5 )
+        @distances = []
+        @abort_at_distance = abort_at_distance
+        @max_runs = max_runs
+        @nodes = points.collect{ |point| Node.new( point, from_point: true ) }
+       end
+
+      def process
+        runs = 0
+        @nodes.each do |node|
+          calculate_distances( node )
+        end
+        @distances.sort!
+        @max_distance = @distances.last.distance 
+        while @distances.size > 2 && @distances.first.distance/@max_distance < @abort_at_distance && runs < @max_runs
+          combine_nodes( @distances.first.node1, @distances.first.node2 )
+          runs = runs+1
+        end
+        self
+      end
+
+      def calculate_distances( node )
+        @nodes.each do |other_node|
+          @distances << NodeDistance.new( node, other_node ) unless node==other_node
+        end
+      end 
+
+      def combine_nodes( node1, node2 )
+        new_node = Node.new([node1,node2])
+        node1.parent_node = new_node
+        node2.parent_node = new_node
+        
+        # remove node1 and node2 
+        @nodes = @nodes.collect do |node| 
+          node unless node == node1 || node == node2
+        end.compact
+        # remove distances for these 2 nodes 
+        @distances = @distances.collect do |distance|
+          distance unless distance.node1 == node1 || distance.node1 == node2 || distance.node2 == node2 || distance.node2 == node1
+        end.compact
+
+        # insert new node
+        @nodes << new_node 
+        # calculate_distances for new node 
+        calculate_distances( new_node )
+        @distances.sort!
+      end 
+
+      def to_s 
+        @nodes.collect do |node|
+          node.to_s
+        end.join(" / ")
+      end
+
+      def to_cluster_set
+        clusters = @nodes.collect{|node| node.to_cluster }
+        Data::ClusterSet.new( clusters: clusters )
+      end
+    end
+
+  end 
+end