substitution_solver 0.5.0

data/README.txt

@@ -0,0 +1,93 @@
+For command line usage see the bottom of this readme.
+
+This archive contains a couple of ruby scripts.  The main script 
+will decode most mono-alphabetic simple substitution ciphers 
+by using a shotgun-hill climbing algorithm.  It works by
+selecting a random key and scoring the result, then making small
+adjustments to the key that improve the score (which is the
+hill climbing part).  If it can't find any obvious adjustments 
+that will improve the score, it will make a small 
+random change to the key and then start hill climbing again.
+The algorithm will do this 10 times after the score plateaus, at
+which point if it can't come up with a better score, the 
+program gives up (assuming that it has either reached 
+a dead end or that it has the answer).  Then the program
+will generate a new, completely random key and start the process
+over from the beginning, (this is the shotgun part of the
+algorithm).  It's a little like trying to find the highest peak
+of a long polynomial expression where you can't plot the line
+ahead of time.  The tactic here would be, pick a random point
+on the line, start climbing the hill until you can't climb any
+higher, then pick another random point on the line and start
+climbing again.  If you have a curve with a small number of 
+relatively uniformly distributed peaks, than this
+is a moderately efficient, albeit uncertain way of
+ascertaining the correct answer.
+
+This program has no way of knowing whether or not it has hit
+upon the correct answer, so it will keep on looking for better
+scores until the user hits CTRL-C to end the program.  It's
+up to the user to decide whether the answer the program has
+come up with makes any sense or not.  The scoring algorithm
+is really the heart of this program.  Basically the program
+has a dictionary of tetra graph frequencies, that is, how
+often do these four letters appear next to one another in the
+English language.  Using this number as a guide, it produces
+the score by taking the log of each of the tetra graph 
+frequencies in the cipher text and adds them all together to 
+produce a score.  adding up the logs of the frequencies to 
+produce the score rather than adding up the scores themselves 
+appears to be important in making the program able to score 
+plain text higher than gibberish.  I didn't realize this the
+first time I wrote this program and didn't get very good
+results.  I should note that I took somebody else's code as
+a guide in producing these ruby scripts.  I've modified the
+strategy slightly from the original author's, and obviously
+I translated it from C to Ruby.  I'm very grateful to the
+original author for sharing his C code with the world.  I
+would have had a hard time getting this program working
+without using his program as a guide, In particular taking
+the logs of the scores would not have occurred to me.
+
+This script runs pretty slowly, owing mostly to the fact that
+I've written it as a ruby script.  I've seen other implementations
+of this idea written in C which run orders of magnatude faster
+than this implementation does.  I plan to retranslate this code
+into several languages to see how well they can run it.
+
+At any rate, included in this archive is the following
+1. the substitution solver script which will over time, 
+(hopefully) recover the plain text.
+2. a sample tetra graph dictionary which was generated using 
+"Harry Potter and the Goblet of Fire" as it's source txt, 
+(yes I own a legal copy of the book)
+3. a script which will generate a tetra graph dictionary from 
+an ascii text file.
+4. this readme.
+5. an inspector script which will show the contents of the
+english.dic file in a readable format.
+
+This program is distributed under the GNU General Public
+License.  If you would like to use this source code in
+your own programs, or rewrite it or whatever, just be
+sure to read the GNU GPL so that you know your rights
+and responsibilities.
+
+examples of usage
+
+to decrypt a cipher text in an ascii text file
+	ruby substitution_solver.rb cipher.txt
+
+to generate an english.dic file using another source use
+	ruby dictionary_builder.rb novel.txt
+
+finally I'm including a script which will print out the english.dic 
+file in a readable form.  It's interesting to note that the 
+frequencies increase exponentially as you go down the list.  
+
+example
+	ruby dictionary_inspector.rb > output.txt
+or
+	ruby dictionary_inspector.rb | less
+or just plain old
+	ruby dictionary_inspector.rb

data/dictionary_builder.rb

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+hash = Hash.new(0)
+
+File::readlines(ARGV[0]).each do |line|
+    line.gsub!(/[^a-zA-Z]/, "").upcase!
+    for x in 0...line.length-4
+        hash[line[x...x+4]] += 1
+    end
+end
+
+#puts hash.to_a.sort {|x, y| x[1] <=> y[1]}
+    
+File.open("english.dic", "w+") do |f|
+    Marshal.dump(hash, f)
+end

data/dictionary_inspector.rb

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+$dictionary = Hash.new(0)                                           # The dictionary of tetragraph frequencies
+
+File.open("english.dic") do |f|                                     # Open the saved
+    $dictionary = Marshal.load(f)                                   # And load this information into our dictionary
+end
+
+array = $dictionary.to_a.sort {|x, y| x[1] <=> y[1]}
+
+x = 0
+array.each do |tetragraph, freq|
+    puts "#{x+=1}, #{tetragraph}, #{freq}"
+end

data/substitution_solver.rb

@@ -0,0 +1,113 @@
+#!/usr/bin/env ruby
+
+$iteration = 0                                                      # To record how many iterations the programs
+                                                                    #  had to churn through
+
+ciphertext = String.new
+
+File::readlines(ARGV[0]).each do |line|                             # Grab the input from the standard input
+    ciphertext << line
+end
+
+ciphertext.gsub!(/[^a-zA-Z]/, "").upcase!                           # get rid of any non-alphabetic characters
+
+key = Hash.new                                                      # Create a hash that will represent the translation key
+    
+$dictionary = Hash.new(0)                                           # The dictionary of tetragraph frequencies
+File.open("english.dic") do |f|                                     # Open the saved tetragraph information
+    $dictionary = Marshal.load(f)                                   # And load this information into our dictionary
+end
+    
+def score(string)                                                   # This function will score a string against the tetragraph statistics
+    $iteration += 1                                                 # Increment the iteration count as this is probably the most fundamental loop to the program
+    tally = 0                                                       # Set a counter to 0
+    0.upto(string.length-4) do |x|                                  # Iterate through the string
+        tally += Math.log(($dictionary[string[x...x+4]].to_i)+1)    # tally up the tetragraph frequencies after applying log to each one (the log is where the magic happens)
+    end
+    return tally                                                    # and return our grand total when we're finished adding it all up
+end
+    
+def small_adj!(key)                                                 # this function makes small random adjustments to the key when we've hill climbed our way into a dead end
+    for i in 0...rand(5)                                            # pick a random number of changes to make
+        j = rand(26)                                                # now pick two random letters in the alphabet to swap
+        k = rand(26)
+        if j != k                                                   # if the random letters aren't equal 
+            temp = key[(j+65).chr]                                  #  then go ahead and swap them
+            key[(j+65).chr] = key[(k+65).chr]
+            key[(k+65).chr] = temp
+        end
+    end
+end
+    
+def plaintext(ciphertext, key)                                      # This function will return the decoded ciphertext using a given key to do the decoding
+    return_string = String.new                                      # create a return string
+
+    for x in 0...ciphertext.length                                  # loop through the ciphertext
+        return_string << key[ciphertext[x].chr]                     # swap the letters out using the key and build up the return string
+    end
+    return return_string                                            # return the answer
+end
+    
+def randomize!(key)                                                 # completely randomize the key, ie start over from scratch
+    array = Array.new                                               # create an array of letters to pick from
+
+    for x in 0...26
+        array[x] = (x+65).chr                                       # populate the array with characters
+    end
+        
+    for x in 0...26                                                 # now loop through the array taking a letter out
+        y = rand(array.length)                                      # one at a time randomely and adding it to the key
+        key[(x+65).chr] = array[y]
+        array.delete_at(y)
+    end
+end
+    
+print "best overall = ", score(ciphertext), " : best score = ", score(ciphertext), "\n" #print the original ciphertext
+puts ciphertext.gsub(/(.....)/, '\1 ')
+    
+randomize!(key)                                                     # randomize the key
+    
+best_score=score(ciphertext);                                       # set the best score to the score of the ciphertext
+best_overall=best_score-1;                                          # set the best overall score to the best score -1
+num_small_adjusts=0;                                                # set the number of small adjustments to 0
+
+loop do                                                             # loop forever
+    best_adj = best_score                                           # set the best adjustment to the current best score
+
+    for i in 0...26                                                 # loop through all possible "trivial" letter replacements
+        for j in i...26                                             # in the key looking for the best swap.  This in effect is
+            test_key = key.dup                                      # the so called "Hill Climbing" part of our program
+            temp = test_key[(i+65).chr]
+            test_key[(i+65).chr] = test_key[(j+65).chr]
+            test_key[(j+65).chr] = temp
+            sc = score(plaintext(ciphertext, test_key))             # score the change we've made
+            if sc > best_adj                                        # if it's better than any so far
+                best_adj=sc                                         # then record the change so we can apply it later if it
+                best_i = i                                          # turns out to be the best one
+                best_j = j
+            end
+        end
+    end
+    
+    if best_adj > best_score                                        # if we found an adjustment that improves the best score
+        temp = key[(best_i+65).chr]                                 # then apply that adjustment to the key
+        key[(best_i+65).chr] = key[(best_j+65).chr]
+        key[(best_j+65).chr] = temp
+        best_score = best_adj
+        if best_score > best_overall                                # if that adjustment is the best overall
+            num_small_adjusts = 0                                   # then reset the number of small adjusts counter
+            best_overall = best_score                               # set this new score as the best overall
+            print "best overall = ", best_overall, " : best score = ", best_score, " : iteration = #{$iteration}\n"
+            puts plaintext(ciphertext, key).gsub(/(.....)/, '\1 ')  # and print our new found best overall value
+        end
+    else                                                            # otherwise none of the adjustments raised are score
+        if num_small_adjusts < 10                                   # so make a small random adjustment to the key
+            small_adj!(key)                                         #  as long as we haven't already made to many small adjustments
+            num_small_adjusts += 1                                  # increment the number of small adjustments
+        else                                                        # otherwise we've made to many small adjustments, we're
+            randomize!(key)                                         #  probably not getting anywhere and need to start looking
+            num_small_adjusts = 0                                   #  somplace else, randomize the key and start climbing the 
+        end                                                         #  hill again
+        best_score=score(plaintext(ciphertext, key))                # set the best score to either the small adjustment value or the new randomized string value depending on what we did above.
+     end
+end

metadata

@@ -0,0 +1,45 @@
+--- !ruby/object:Gem::Specification 
+rubygems_version: 0.8.10
+specification_version: 1
+name: substitution_solver
+version: !ruby/object:Gem::Version 
+  version: 0.5.0
+date: 2005-11-09
+summary: "Program for solving mono-alphabetic simple substitution ciphers, (as in
+  cryptoquotes), without word lengths."
+require_paths: 
+  - lib
+email: pfharlock@yahoo.com
+homepage: 
+rubyforge_project: 
+description: 
+autorequire: 
+default_executable: 
+bindir: "."
+has_rdoc: false
+required_ruby_version: !ruby/object:Gem::Version::Requirement 
+  requirements: 
+    - 
+      - ">"
+      - !ruby/object:Gem::Version 
+        version: 0.0.0
+  version: 
+platform: ruby
+authors: 
+  - Gary Watson
+files: 
+  - substitution_solver.rb
+  - dictionary_builder.rb
+  - dictionary_inspector.rb
+  - README.txt
+test_files: []
+rdoc_options: []
+extra_rdoc_files: 
+  - README.txt
+executables: 
+  - substitution_solver.rb
+  - dictionary_builder.rb
+  - dictionary_inspector.rb
+extensions: []
+requirements: []
+dependencies: []