backprop 0.0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 30d4ab63e0502df289e6e648ad5c04c5d0ffe4b29a0bd5fce2053809c4879ddd
+  data.tar.gz: 943142da82fb2a4fd4adad13f55a1fe1f1e1713e71128a5c02c48887cc7aa4cb
+SHA512:
+  metadata.gz: fedcb937e83efec000f8cc944e53b8cf8e61eb5e2ea4eccb60d4431ae230e9a7607adf4172a61cc14e32bfa41344538439c3538d5f4f52b2936df6ddf70827c1
+  data.tar.gz: 8d75ed673305ba9213840974a314caef65029156a23f56040dd99035166b8bb357fc44adff9f79a640d343fd84788a98422d7099b49e651016fa18aa48385108

data/README.md

@@ -0,0 +1,214 @@
+# Backward Propagation
+
+This is a reimplementation of Andrej Karpathy's
+[micrograd](https://github.com/karpathy/micrograd) in Ruby.
+It has been further simplified and some liberties have been taken with naming.
+
+# Rationale
+
+This can be used to train neural nets, typically to minimize a loss function.
+An efficient way to do this is via gradient descent.
+Mathematical derivatives and the chain rule from calculus are used to determine
+  inputs with the greatest influence on the output.
+The inputs are manipulated to minimize the output, represented as the loss
+  function.
+That is, the output of the neural net is a prediction.
+The error or loss (prediction compared to the ideal, or known output) is
+  computed for a variety of cases, and the network weights are adjusted to
+  better match the desired output.
+The smallest loss implies the best performance at a given objective.
+
+# Examples
+
+```ruby
+require 'backprop'
+
+include BackProp
+
+# F = ma
+
+mass = Value.new(25, label: 'mass')
+acc = Value.new(10, label: 'acc')
+force = mass * acc
+force.label = 'force'
+p force
+```
+
+```
+force(value=250 gradient=0 *(mass=25, acc=10))
+        mass(value=25 gradient=0)
+        acc(value=10 gradient=0)
+```
+
+Use backward propagation to determine the gradient (derivative with respect
+  to the caller of `#backward`) for each Value:
+
+```ruby
+force.backward
+p force
+```
+
+```
+force(value=250 gradient=1.0 *(mass=25, acc=10))
+        mass(value=25 gradient=10.0)
+        acc(value=10 gradient=25.0)
+```
+
+The gradients have been updated, and the output gradient is 1.0.
+We have a tree structure, where our inputs, mass and acceleration, are
+  leaf nodes, and they combine via multiplication to make a parent node, or
+  root node in this case, force.
+By wrapping our numbers in the Value class, whenever we calculate a result,
+  we have a tree structure representing that expression, and we can easily
+  calculate derivatives for every node in the tree.
+
+# Neural Networks
+
+## Neuron
+
+A neuron has a number of inputs which it combines to yield a single output.
+Traditionally, each input has a weight, and the neuron itself has a bias, or
+  a fixed amount which is added to each input when considering the output.
+Sum each input value times its input weight, add the bias, and apply an
+  *activation function* which "normalizes" the output to a predictable value,
+  typically between -1.0 and 1.0.
+In other words, if you send the right combination of signals, you can get the
+  neuron to "fire".
+
+```ruby
+require 'perceptron'
+
+include BackProp
+
+# create a new neuron with 3 inputs; initial weights and bias are random
+n = Neuron.new(3)
+
+puts n
+#=> N(-0.098, 1.000, 0.064) (0.468 relu)
+
+p n
+#=> -0.098| 0.000         1.000| 0.000    0.064| 0.000    0.468| 0.000
+
+# send 0 to each input
+output = n.apply(0)
+
+puts output
+#=> 0.468
+
+# output is positive due to rectified linear unit (ReLU) activation function
+output.value >= 0 #=> true
+
+# if bias is positive, zero input should result in bias
+(n.bias.value >= 0) ? (output.value == n.bias) : (output.value == 0) #=> true
+```
+
+## Layer
+
+A layer is composed of several neurons.
+Each neuron has the same number of inputs, so the layer has just a single
+  number of inputs.
+Each input is sent to each neuron in the layer.
+If one layer is to feed into another, then the other layer's neurons must have
+  an input count that matches the one layer's neuron count.
+
+```ruby
+require 'perceptron'
+
+include BackProp
+
+# create a new layer of 4 neurons with 3 inputs
+l = Layer.new(3, 4)
+
+puts l
+```
+
+```
+N(0.957, 0.650, 0.995)  (-0.530 relu)
+N(-0.482, 0.272, -0.467)        (0.905 relu)
+N(-0.083, -0.519, -0.921)       (-0.811 relu)
+N(-0.369, -0.688, -0.097)       (0.122 relu)
+```
+
+```ruby
+# send 0 to each input
+output = l.apply(0)
+
+# returns an array of outputs, one for each neuron
+output.size == 4 #=> true
+
+puts output.map(&:value).join(', ')
+#=> 0.0, 0.90522363833711, 0.0, 0.12226124806686789
+```
+
+## Multilayer Perceptron (MLP)
+
+First, define a number of inputs.  Say 5 inputs, like temperature, etc.
+Often we want a single output, which is the simple case.
+Multiple outputs are possible but more complicated.
+A single output could represent the recommended setting on a thermostat.
+We can define multiple layers of neurons for our neural net which will feed
+  on inputs and yield outputs.
+
+```ruby
+require 'perceptron'
+
+include BackProp
+
+# create a network with 3 inputs, 2 layers of 4 neurons, and one output neuron
+n = MLP.new(3, [4, 4, 1])
+
+puts n
+```
+
+```
+N(0.660, 0.250, -0.387) (-0.677 relu)
+N(0.931, 0.202, 0.596)  (0.861 relu)
+N(0.101, 0.611, 0.885)  (-0.295 relu)
+N(-0.858, 0.136, 0.091) (-0.309 relu)
+
+N(-0.594, 0.178, 0.484, -0.208) (0.515 relu)
+N(-0.295, -0.899, 0.437, -0.812)        (-0.200 relu)
+N(-0.478, 0.230, -0.971, 0.897) (-0.858 relu)
+N(0.636, 0.719, -0.857, -0.546) (-0.338 relu)
+
+N(0.962, 0.529, 0.475, -0.837)  (-0.362 relu)
+```
+
+```ruby
+# the first layer has 4 neurons, 3 inputs
+n.layers[0].neurons.size == 4 #=> true
+n.layers[0].neurons[0].weights.size == 3 #=> true
+
+# next layer has 4 neurons, 4 inputs
+n.layers[1].neurons.size == 4 #=> true
+n.layers[1].neurons[0].weights.size == 4 #=> true
+
+# final layer has 1 neuron, 4 inputs
+n.layers[2].neurons.size == 1 #=> true
+n.layers[2].neurons[0].weights.size == 4 #=> true
+
+# send 0 to each input
+output = n.apply(0)
+
+# returns an output value corresponding to the output neuron
+# output is positive to due to ReLU
+output.value >= 0 #=> true
+
+puts output
+#=> 0.045
+```
+
+## Gradient Descent
+
+Loop:
+
+1. Backward propagate the gradients
+   (derivatives for each value with respect to the output value)
+2. Adjust all weights slightly, according to their gradients.
+3. Run the network forward to generate a new output.
+   The loss should be smaller.
+   The new output should be closer to the desired output.
+
+## Further Reading
+
+* [demo/loss.rb](demo/loss.rb)

data/Rakefile

@@ -0,0 +1,22 @@
+require 'rake/testtask'
+
+Rake::TestTask.new :test do |t|
+  t.pattern = "test/*.rb"
+  t.warning = true
+end
+
+#
+# GEM BUILD / PUBLISH
+#
+
+begin
+  require 'buildar'
+
+  Buildar.new do |b|
+    b.gemspec_file = 'backprop.gemspec'
+    b.version_file = 'VERSION'
+    b.use_git = true
+  end
+rescue LoadError
+  warn "buildar tasks unavailable"
+end

data/VERSION

@@ -0,0 +1 @@
+0.0.0.1

data/backprop.gemspec

@@ -0,0 +1,17 @@
+Gem::Specification.new do |s|
+  s.name = 'backprop'
+  s.summary = "WIP"
+  s.description = "WIP"
+  s.authors = ["Rick Hull"]
+  s.homepage = "https://github.com/rickhull/backprop"
+  s.license = "LGPL-3.0"
+
+  s.required_ruby_version = "> 2"
+
+  s.version = File.read(File.join(__dir__, 'VERSION')).chomp
+
+  s.files = %w[backprop.gemspec VERSION README.md Rakefile]
+  s.files += Dir['lib/**/*.rb']
+  s.files += Dir['test/**/*.rb']
+  s.files += Dir['demo/**/*.rb']
+end

data/demo/celsius.rb

@@ -0,0 +1,11 @@
+require 'backprop'
+
+include BackProp
+
+deg_f = 68.5
+
+puts "Convert #{deg_f} to celsius using arithmetic (F - 32 * 5/9):"
+
+f = Value.new(68.5, label: :f)
+c = (f - 32) * 5 / 9; c.label = :c
+p c

data/demo/lol.rb

@@ -0,0 +1,56 @@
+require 'backprop'
+
+include BackProp
+
+a = Value.new(2, label: :a)
+b = Value.new(-3, label: :b)
+c = Value.new(10, label: :c)
+e = a * b; e.label = :e
+d = e + c; d.label = :d
+f = Value.new(-2, label: :f)
+l = d * f; l.label = :L
+
+puts "Setup:"
+p l
+puts
+
+
+puts "Calculate gradient by hand:"
+
+l.gradient = 1.0
+
+# l = d * f; derivative dl/dd = f; dl/df = d
+f.gradient = d.value
+d.gradient = f.value
+
+
+# now c.gradient
+# that is dL/dc
+
+# dL/dd is -2
+# dd/dc is 1
+# by chain rule (multiply): dL/dc is -2 * 1 = -2
+
+c.gradient = d.gradient * l.gradient
+e.gradient = d.gradient * l.gradient
+
+# now b.gradient (and a.gradient)
+# e = a * b
+
+# dL/da = dL/de * de/da
+a.gradient = e.gradient * b.value
+b.gradient = e.gradient * a.value
+
+p l
+puts
+
+puts "Reset gradients"
+l.reset_gradient
+p l
+puts
+
+puts "Calculate gradient via backward:"
+
+l.backward
+
+p l

data/demo/loss.rb

@@ -0,0 +1,64 @@
+require 'perceptron'
+
+include BackProp
+
+num_inputs = 3
+num_examples = 6
+net_structure = [4, 4, 1]
+gradient_step = 0.1
+iterations = 999
+afn = [:tanh, :sigmoid, :relu].sample
+
+# binary classifier; 9 sets of inputs that map to 1 or 0
+inputs = BackProp.rand_inputs(num_inputs, num_examples, (-1.0..1.0))
+outputs = BackProp.rand_outputs(num_examples, 2)
+predictions = []
+
+n = MLP.new(num_inputs, net_structure, activation: afn)
+
+puts "Training Cases:"
+inputs.each.with_index { |input, i|
+  puts format("%s = %s", input.join(', '), outputs[i].value.inspect)
+}
+puts
+
+puts "Neural Net:"
+puts n
+puts
+
+puts "Press Enter to continue"
+gets
+
+999.times { |i|
+  # 1. apply inputs to the net to yield predictions
+  # 2. calculate the loss
+  # 3. backward propagate the gradients
+  # 4. adjust every neuron in the direction of minimizing loss
+
+  # 1. apply inputs
+  predictions = inputs.map { |input| n.apply(input).first }
+
+  # 2. calculate loss
+  loss = BackProp.mean_squared_error(outputs, predictions)
+  puts loss
+
+  # 3. propagate the derivatives (gradients) backwards
+  loss.backward
+
+  # output every so often
+  if i % 100 == 0
+    p outputs.map(&:value)
+    p predictions.map(&:value)
+    puts
+    p n
+    gets
+  end
+
+  # 4. adjust all weights and biases towards minimizing loss function
+  n.descend(gradient_step)
+}
+
+p outputs.map(&:value)
+p predictions.map(&:value)
+puts n
+p n

data/demo/neuron.rb

@@ -0,0 +1,61 @@
+require 'backprop'
+
+include BackProp
+
+# inputs x1, x2
+x1 = Value.new(2, label: :x1)
+x2 = Value.new(0, label: :x2)
+
+# weights w1, w2
+w1 = Value.new(-3, label: :w1)
+w2 = Value.new(1, label: :w2)
+
+# neuron bias
+b = Value.new(6.8813735870195432, label: :b)
+
+xw1 = x1*w1; xw1.label = :xw1
+xw2 = x2*w2; xw2.label = :xw2
+
+sum = xw1 + xw2; sum.label = :sum
+n = sum + b; n.label = :n
+
+o = n.tanh; o.label = :o
+
+puts "Calculate gradient by hand:"
+o.gradient = 1
+
+# do/dn
+# d/dx tanh x = 1 - tanh(x)^2
+
+# 1 - o**2
+
+n.gradient = 1 - o.value ** 2
+
+# n = sum + b
+sum.gradient = n.gradient
+b.gradient = n.gradient
+
+# sum = xw1 + xw2
+xw1.gradient = sum.gradient
+xw2.gradient = sum.gradient
+
+# xw1 = x1 * w1
+x1.gradient = xw1.gradient * w1.value
+w1.gradient = xw1.gradient * x1.value
+
+# xw2 = x2 * w2
+x2.gradient = xw2.gradient * w2.value
+w2.gradient = xw2.gradient * x2.value
+
+p o
+puts
+
+puts "Reset gradient:"
+o.reset_gradient
+p o
+puts
+
+puts "Calculate gradient via backprop:"
+o.backward
+p o
+puts

data/lib/backprop.rb

@@ -0,0 +1,146 @@
+module BackProp
+  class Value
+    def self.wrap(other)
+      other.is_a?(Value) ? other : Value.new(other)
+    end
+
+    attr_reader :children
+    attr_accessor :value, :label, :gradient, :backstep, :op
+
+    def initialize(float, label: '', op: nil, children: [])
+      @value = float.to_f
+      @gradient = 0
+      @children = children
+      if @children.empty?
+        raise "op #{op.inspect} has no children" unless op.nil?
+      else
+        raise "op is required" if op.nil?
+      end
+      @op = op
+      @label = label
+      @backstep = -> {}
+    end
+
+    def to_s
+      @label.empty? ? ("%.3f" % @value) : format("%s=%.3f", @label, @value)
+    end
+
+    def display
+      format("%s(%.3f gradient=%.3f",
+             @label.empty? ? @op || 'Value' : @label, @value, @gradient) +
+        (@op.nil? ? '' :
+           format(" %s(%s)", @op, @children.join(', '))) + ')'
+    end
+
+    def inspect
+      @children.empty? ? self.display :
+        [self.display, @children.map(&:inspect).join("\n\t")].join("\n\t")
+    end
+
+    #
+    # Primary operations; notice every Value.new(op:) also defines a backstep
+    #   The backstep closes over the environment of the method so it can
+    #   refer to values present when the method executes
+    #
+
+    def +(other)
+      other = Value.wrap(other)
+      val = Value.new(@value + other.value, children: [self, other], op: :+)
+      val.backstep = -> {
+        # gradients accumulate to handle a value used multiple times
+        self.gradient += val.gradient
+        other.gradient += val.gradient
+      }
+      val
+    end
+
+    def *(other)
+      other = Value.wrap(other)
+      val = Value.new(@value * other.value, children: [self, other], op: :*)
+      val.backstep = -> {
+        self.gradient += val.gradient * other.value
+        other.gradient += val.gradient * self.value
+      }
+      val
+    end
+
+    # Mostly we are squaring(2) or dividing(-1)
+    def **(other)
+      raise("Value is not supported") if other.is_a? Value
+      val = Value.new(@value ** other, children: [self], op: :**)
+      val.backstep = -> {
+        self.gradient += val.gradient * (other * self.value ** (other - 1))
+      }
+      val
+    end
+
+    def exp
+      val = Value.new(Math.exp(@value), children: [self], op: :exp)
+      val.backstep = -> {
+        self.gradient += val.gradient * val.value
+      }
+      val
+    end
+
+    #
+    # Secondary operations defined in terms of primary
+    #
+
+    def -(other)
+      self + (Value.wrap(other) * Value.new(-1))
+    end
+
+    def /(other)
+      self * (Value.wrap(other) ** -1)
+    end
+
+    #
+    # Activation functions
+    #
+
+    def tanh
+      val = Value.new(Math.tanh(@value), children: [self], op: :tanh)
+      val.backstep = -> {
+        self.gradient += val.gradient * (1 - val.value ** 2)
+      }
+      val
+    end
+
+    # 1 / 1 + e^-x
+    def sigmoid
+      ((self * -1).exp + 1) ** -1
+    end
+
+    # rectified linear unit; not susceptible to vanishing gradient like above
+    def relu
+      neg = @value < 0
+      val = Value.new(neg ? 0 : @value, children: [self], op: :relu)
+      val.backstep = -> {
+        self.gradient += val.gradient * (neg ? 0 : 1)
+      }
+      val
+    end
+
+    #
+    # Backward propagation
+    #
+
+    def backward
+      self.reset_gradient
+      @gradient = 1.0
+      self.backprop
+    end
+
+    def reset_gradient
+      @gradient = 0.0
+      @children.each(&:reset_gradient)
+      self
+    end
+
+    def backprop
+      self.backstep.call
+      @children.each(&:backprop)
+      self
+    end
+  end
+end

data/lib/perceptron.rb

@@ -0,0 +1,119 @@
+require 'backprop'
+
+module BackProp
+  class Neuron
+    # available activation functions for Value objects
+    ACTIVATION = {
+      tanh: :tanh,
+      sigmoid: :sigmoid,
+      relu: :relu,
+    }
+
+    attr_reader :weights, :bias, :activation
+
+    def initialize(input_count, activation: :relu)
+      @weights = Array.new(input_count) { Value.new(rand(-1.0..1.0)) }
+      @bias = Value.new(rand(-1.0..1.0))
+      @activation = ACTIVATION.fetch(activation)
+    end
+
+    def apply(x = 0)
+      x = Array.new(@weights.size) { x } if !x.is_a? Enumerable
+      sum = @weights.map.with_index { |w, i|
+        w * x[i]
+      }.inject(Value.new(0)) { |memo, val| memo + val } + @bias
+      sum.send(@activation)
+    end
+
+    def descend(step_size)
+      (@weights + [@bias]).each { |p|
+        p.value += (-1 * step_size * p.gradient)
+      }
+      self
+    end
+
+    def to_s
+      format("N(%s)\t(%s %s)", @weights.join(', '), @bias, @activation)
+    end
+
+    def inspect
+      fmt = "% .3f|% .3f"
+      @weights.map { |w| format(fmt, w.value, w.gradient) }.join("\t") +
+        "\t" + format(fmt, @bias.value, @bias.gradient)
+    end
+  end
+
+  class Layer
+    attr_reader :neurons
+
+    def initialize(input_count, output_count, activation: :relu)
+      @neurons = Array.new(output_count) {
+        Neuron.new(input_count, activation: activation)
+      }
+    end
+
+    def apply(x = 0)
+      @neurons.map { |n| n.apply(x) }
+    end
+
+    def descend(step_size)
+      @neurons.each { |n| n.descend(step_size) }
+      self
+    end
+
+    def to_s
+      @neurons.join("\n")
+    end
+
+    def inspect
+      @neurons.map(&:inspect).join("\n")
+    end
+  end
+
+  class MLP
+    attr_reader :layers
+
+    # MLP.new(3, [4, 4, 1])
+    def initialize(input_count, output_counts, activation: :relu)
+      flat = [input_count, *output_counts]
+      @layers = output_counts.map.with_index { |oc, i|
+        Layer.new(flat[i], flat[i+1], activation: activation)
+      }
+    end
+
+    def apply(x = 0)
+      @layers.each { |layer| x = layer.apply(x) }
+      # x.size == 1 ? x.first : x
+      x
+    end
+
+    def descend(step_size)
+      @layers.each { |l| l.descend(step_size) }
+      self
+    end
+
+    def to_s
+      @layers.join("\n\n")
+    end
+
+    def inspect
+      @layers.map(&:inspect).join("\n\n")
+    end
+  end
+
+  def self.mean_squared_error(a1, a2)
+    a1.map.with_index { |a, i|
+      (a - a2[i]) ** 2
+    }.inject(Value.new(0)) { |memo, val| memo + val } / a1.size
+  end
+
+  def self.rand_inputs(num_inputs, num_examples, rand_arg)
+    Array.new(num_examples) {
+      Array.new(num_inputs) { Value.new rand(rand_arg) }
+    }
+  end
+
+  def self.rand_outputs(num_examples, rand_arg)
+    Array.new(num_examples) { Value.new rand(rand_arg) }
+  end
+end

data/test/backprop.rb

@@ -0,0 +1,202 @@
+require 'minitest/autorun'
+require 'backprop'
+
+include BackProp
+
+describe Value do
+  describe "basics" do
+    before do
+      @flt = 2.3
+      @v = Value.new(2.3)
+    end
+
+    it "wraps numeric values, primarily floats" do
+      expect(@v).must_be_kind_of Value
+      expect(@v.value).must_be_kind_of Float
+      expect(@v.value).must_equal @flt
+    end
+
+    it "has several string representations" do
+      expect(@v.to_s).must_be_kind_of String
+      expect(@v.display).must_be_kind_of String
+      expect(@v.inspect).must_be_kind_of String
+    end
+
+    it "creates a tree structure when joined by an operator" do
+      expect(@v.children).must_be_empty
+      sum = @v + 3
+      expect(sum).must_be_kind_of Value
+      expect(sum.children).wont_be_empty
+      expect(sum.op).must_equal :+
+      expect(sum.value).must_be_within_epsilon(@flt + 3)
+    end
+
+    it "keeps track of a gradient value, initialized to zero" do
+      expect(@v.gradient).must_equal 0
+    end
+  end
+
+  describe "operations" do
+    it "updates the gradient value when used in a calculation" do
+
+    end
+
+    describe "addition" do
+      before do
+        @a = Value.new(1.0)
+        @b = Value.new(2.0)
+        @sum = @a + @b
+      end
+
+      it "yields a Value" do
+        expect(@sum).must_be_kind_of Value
+        expect(@sum.value).must_be_within_epsilon 3.0
+      end
+
+      it "has a sum parent with _a_ and _b_ as children" do
+        expect(@sum.children).must_include @a
+        expect(@sum.children).must_include @b
+        expect(@sum.op).must_equal :+
+      end
+
+      it "updates child gradients upon back propagation" do
+        expect(@a.gradient).must_equal 0
+        expect(@b.gradient).must_equal 0
+        expect(@sum.gradient).must_equal 0
+
+        @sum.backward
+        expect(@sum.gradient).must_equal 1  # by definition
+        expect(@a.gradient).must_equal 1    # via chain rule for addition
+        expect(@b.gradient).must_equal 1    # via chain rule for addition
+      end
+    end
+
+    describe "multiplication" do
+      before do
+        @a = Value.new(-1)
+        @b = Value.new(2.5)
+        @prod = @a * @b
+      end
+
+      it "yields a Value" do
+        expect(@prod).must_be_kind_of Value
+        expect(@prod.value).must_be_within_epsilon(-2.5)
+      end
+
+      it "has a prod parent with _a_ and _b_ and children" do
+        expect(@prod.children).must_include @a
+        expect(@prod.children).must_include @a
+        expect(@prod.op).must_equal :*
+      end
+
+      it "updates child gradients upon back propagation" do
+        expect(@a.gradient).must_equal 0
+        expect(@b.gradient).must_equal 0
+        expect(@prod.gradient).must_equal 0
+
+        @prod.backward
+        expect(@prod.gradient).must_equal 1
+        expect(@a.gradient).must_equal @b.value # via chain rule
+        expect(@b.gradient).must_equal @a.value # via chain rule
+      end
+    end
+
+    describe "subtraction" do
+      before do
+        @a = Value.new(10)
+        @b = Value.new(4)
+        @diff = @a - @b
+      end
+
+      it "combines addition with multiplication for negation" do
+        # @a + @b * -1
+        expect(@diff.value).must_be_within_epsilon 6.0
+        expect(@diff.op).wont_equal :-
+        expect(@diff.op).must_equal :+
+        expect(@diff.children).must_include @a
+        expect(@diff.children).wont_include @b
+      end
+    end
+
+    describe "pow" do
+      before do
+        @a = Value.new 2
+        @b = 10
+        @pow = @a ** @b
+      end
+
+      it "does not work with right-side Values" do
+        expect { @a ** Value.new(3) }.must_raise
+      end
+
+      it "yields a Value" do
+        expect(@pow).must_be_kind_of Value
+        expect(@pow.value).must_be_within_epsilon 1024.0
+      end
+
+      it "has a pow parent without _b_ in children" do
+        expect(@pow.children).must_include @a
+        expect(@pow.children).wont_include @b
+        expect(@pow.op).must_equal :**
+      end
+
+      it "updates child gradient upon back propagation" do
+        expect(@a.gradient).must_equal 0
+        expect(@pow.gradient).must_equal 0
+
+        @pow.backward
+        expect(@pow.gradient).must_equal 1
+        expect(@a.gradient).must_be_within_epsilon @b * @a.value ** (@b - 1)
+      end
+    end
+
+    describe "division" do
+      before do
+        @a = Value.new 19.1
+        @b = Value.new 2.3
+        @quot = @a / @b
+      end
+
+      it "uses pow(-1)" do
+        # @a * @b ** -1
+        expect(@quot.value).must_be_within_epsilon(19.1 / 2.3)
+        expect(@quot.op).wont_equal :/
+        expect(@quot.op).must_equal :*
+        expect(@quot.children).must_include @a
+        expect(@quot.children).wont_include @b
+      end
+    end
+
+    describe "exp" do
+      before do
+        @a = Value.new 2.4
+        @exp = @a.exp
+      end
+
+      it "yields a Value" do
+        expect(@exp).must_be_kind_of Value
+        expect(@exp.value).must_be_within_epsilon Math.exp(2.4)
+      end
+
+      it "has exp parent with _a_ in children" do
+        expect(@exp.children).must_include @a
+        expect(@exp.op).must_equal :exp
+      end
+
+      it "updates child gradient upon back propagation" do
+        expect(@a.gradient).must_equal 0
+        expect(@exp.gradient).must_equal 0
+
+        @exp.backward
+        expect(@exp.gradient).must_equal 1
+        expect(@a.gradient).must_equal @exp.value # chain rule / derivative
+      end
+    end
+  end
+
+  describe "activation functions" do
+  end
+
+  describe "backward propagation" do
+  end
+end

metadata

@@ -0,0 +1,53 @@
+--- !ruby/object:Gem::Specification
+name: backprop
+version: !ruby/object:Gem::Version
+  version: 0.0.0.1
+platform: ruby
+authors:
+- Rick Hull
+autorequire:
+bindir: bin
+cert_chain: []
+date: 1980-01-01 00:00:00.000000000 Z
+dependencies: []
+description: WIP
+email:
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- Rakefile
+- VERSION
+- backprop.gemspec
+- demo/celsius.rb
+- demo/lol.rb
+- demo/loss.rb
+- demo/neuron.rb
+- lib/backprop.rb
+- lib/perceptron.rb
+- test/backprop.rb
+homepage: https://github.com/rickhull/backprop
+licenses:
+- LGPL-3.0
+metadata: {}
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: '2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.2.26
+signing_key:
+specification_version: 4
+summary: WIP
+test_files: []