huffnpuff 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 8d53cb2641733aa17816bf18f3482c9917111bcc
+  data.tar.gz: e4af27561bb0efdd23c19a95ba8e20771c79f007
+SHA512:
+  metadata.gz: 43e413245fcea87c5b5cd2078b6a69ce29470beaeba838e907da7658c0c7742878e0194753f48d17b57a04e6ba598f692b349d915934910f313ddfd832e583d2
+  data.tar.gz: 0da34b39907367109459aa7c2e8b6579a20d6bd806e42cc038b00dfa69f58916d4991ea9ba8545084584a08d9241033eca26a8d66a0122a1def0a6ad8eb28f5a

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.3.0
+before_install: gem install bundler -v 1.11.2

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at bryan@bdlsys.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in huffnpuff.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Bryan Colvin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,158 @@
+# Huffnpuff
+
+`Huffnpuff` implements a general purpose Huffman Coding system that uses a construction string which can implement any system.
+This also includes the `bitfifo` gem which allows variable length bit sequences to address the internal table of the
+`Huffnpuff` object instance.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'huffnpuff'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install huffnpuff
+
+## Usage
+
+Huffman coding systems are widely use on compression engines such as MPEG and PKZIP.
+The theory suggests that symbols that repeat can be represented by fewer bits if we sacrifice rare symbols with larger bit sequences.
+If you look at the game of Scrabble, some letters have higher points than others because they don't appear as often.
+Morse Code uses variable length dits and dahs to represent letters, but is not a valid Huffman system as you need gaps between letters
+to separate them. A valid Huffman system allows for a stream of data to be able to deterministically separate each token (group of bits).
+Below is an example of a valid system:
+
+```
+0       # 0 
+1000    # 1
+1001    # 2
+1010    # 3
+1011    # 4
+11000   # 5
+11001   # 6
+11010   # 7
+11011   # 8
+111     # 9
+```
+
+The above system has an order of 10, meaning it can represent or index 10 items.
+This system can be represented by the the string: `"0:10xx:110xx:111"`.
+The string represents 4 bins where each bin is separated by a colon.
+The bin's preamble is defined as the sequence of bits before an `'x'` is encountered.
+It is the preamble that makes it possible to separate tokens.
+A valid string may only include the following characters: `'0', '1', 'x', ':'`.
+The character `'x'` may only appear after the preamble; also, `'1'` or `'0'` may not appear after `'x'`.
+The character `'x'` represents two possible bit values of `'0'` and `'1'`.
+Consecutive `'xx'` means there are 4 possible values: `'00'`, `'01'`, `'10'`, `'11'`.
+Each consecutive `'x'` doubles the possible values. 
+In order for a system to be valid, a preamble may not overlap another preamble.
+For example, the string `"0:10xxx:101xx"` is invalid because the bit sequence `'10101'`
+would be found in more than one bin.
+
+Coming up with a system can be tricky. The best way to come up with a system is to create a histogram of your data set,
+then sort it in descending order. An evenly distributed histogram means that Huffman encoding will not be effective.
+If there are large differences, then Huffman encoding will be very effective.
+The `Huffnpuff` gem has a utility method that can generate the system string needed to create a system.
+This is best seen by example as follows:
+
+```ruby
+require 'huffnpuff'
+
+hist = [200,50,35,27,22,15,13,12,9,8,8,8,8,8,7,6,5,5,4,4,4,3,3,3,3,3,2,2,2,1,1,1,1,1,1,1,1]
+tight  = Huffnpuff.get_system_string_from_histogram(hist, 0.20)
+medium = Huffnpuff.get_system_string_from_histogram(hist, 0.50) 
+loose  = Huffnpuff.get_system_string_from_histogram(hist, 0.75)
+
+# tight  == "0x:10xxxx:1110xx:110xxxx"
+# medium == "0:10xx:11110:110xx:1110xxxx:111110xx:1111110xxx"
+# loose  == "0:10:110x:1110:1111110:11110xx:111110xxx:11111110xx:111111110xx:1111111110xx:11111111110xxx"
+```
+
+The first parameter is our histogram array sorted in descending order.
+The second parameter is the grouping factor which ranges from 0.1 to 0.9 inclusively.
+The size of the array is equal to the number of elements we wish to represent with a Huffman code.
+The first element of the `hist` array is the most likely to occur token, and is encoded first at the beginning of our system string.
+Note that a system string need not to have its bins numerically sorted.
+The algorithm sorts based on the length of bits needed for the bin.
+A `Huffnpuff` object has an internal lookup table which maps the tokens to an entry in the table.
+This is best seen by example as follows:
+
+```ruby
+bits = Bitfifo.new
+lut  = ['Apple', 'Banana', 'Cherry', 'Dog', 'Elephant', 'Fortune', 'Grape']
+huff = Huffnpuff.new(loose, bits, lut)
+huff.push 'House'  # or lut.push 'House' ... same thing
+
+# prime the pump
+bits.put 0, 1          # bin 0
+bits.put 0b10, 2       # bin 1
+bits.put 0b1100, 4     # bin 2 x==0
+bits.put 0b1101, 4     # bin 2 x==1
+bits.put 0b1110, 4     # bin 3
+bits.put 0b1111110, 7  # bin 4
+bits.put 0b1111000, 7  # bin 5 xx==00
+bits.put 0b1111001, 7  # bin 5 xx==01
+
+bits.size   # 36  ... number of bits in bitfifo
+
+# binary representation of bits in fifo
+bits.to_i.to_s(2)   # "10110011011110111111011110001111001"
+
+# now get the data
+huff.get   #  'Apple'
+huff.get   #  'Banana'
+huff.get   #  'Cherry'
+huff.get   #  'Dog'
+huff.get   #  'Elephant'
+huff.get   #  'Fortune'
+huff.get   #  'Grape'
+huff.get   #  'House'
+huff.get   # nil, >>> no address found in Bitfifo 'bits'
+bits.put 0b1100, 4  # random access
+huff.get   #  'Cherry' 
+```
+
+Huffnpuff has the following methods:
+
+```ruby
+# class methods: 
+# Huffnpuff.histogram_array_valid?(ary)  ... ensures histogram descending integers
+# Huffnpuff.validate_system_string(str)  ... checks for syntax errors on system string
+# Huffnpuff.get_system_string_from_histogram(ary, rat = 0.61) ... creates system string  0.1 <= rat <= 0.9
+# Huffnpuff.new(sys_string, bitfifo, array=[])   ... creates new object
+
+# class constants:
+# Huffnpuff::CMP                         ... proc used to sort bins
+
+# instance methods:
+# huff.min    ... returns minimum number of bits needed to access data
+# huff.max    ... returns maximum number of bits needed to access data
+# huff.size   ... returns maximum number of addresses that may be used to look up data
+# huff.sys    ... returns system string
+# huff.systemize!(sys)  ... recreates system from system string, or replace existing system
+# huff.push   ... adds data to internal lookup table, raises error if no room is available
+# huff.push_each        ... adds elements of array or characters of a string to lookup table
+# huff.get    ... returns data indexed by external bitfifo
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+## Contributing
+
+I need to control this for the time being.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "huffnpuff"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/huffnpuff.gemspec

@@ -0,0 +1,31 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'huffnpuff/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "huffnpuff"
+  spec.version       = Huffnpuff::VERSION
+  spec.authors       = ["Bryan Colvin"]
+  spec.email         = ["bryan@bdlsys.com"]
+
+  spec.summary       = %q{Huffman encoding system}
+  spec.description   = %q{Huffnpuff implements a general purpose Huffman Coding system that uses a construction string which can implement any system.
+This also includes the bitfifo gem which allows variable length bit sequences to address the internal table of the
+Huffnpuff object instance. Huffman coding is essential for compression techniques such as MPEG, and PKZIP.
+}
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+  
+  spec.add_runtime_dependency 'bitfifo', '~> 0.2', '>= 0.2.1'
+
+  spec.add_development_dependency "bundler", "~> 1.11"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency 'byebug', '~> 5.0', '>= 5.0.0'
+  
+end

data/lib/huffnpuff.rb

@@ -0,0 +1,182 @@
+require "huffnpuff/version"
+require "bitfifo"
+
+class Huffnpuff
+  def self.histogram_array_valid?(ary)
+    return false if ary.empty?
+    ary.each do |elm|
+      return false unless elm.kind_of? Integer
+      return false if elm <= 0
+    end
+    return false unless ary == ary.sort.reverse
+    true
+  end
+  
+  def self.validate_system_string(str)
+    ary = str.split(':')
+    raise "singular bin not valid" if 1==ary.count
+    ary.each do |bin|
+      raise "empty bin definition" if bin.empty?
+      raise "preamble missing: must start with 1 or 0" if bin[0]=='x'
+      xmode = false
+      bin.each_char do |ch|
+        if xmode
+          raise "illegal char following 'x': ['#{ch}']" unless 'x'==ch
+        else
+          if ch=='x'
+            xmode = true
+          elsif ch=='0'
+          elsif ch=='1'
+          else
+            raise "invalid symbol #{ch}"
+          end
+        end
+      end
+    end
+    true
+  end
+  
+  CMP = lambda { |a,b| a.length <=> b.length }
+
+  def self.get_system_string_from_histogram(ary, rat = 0.61)
+    raise "histogram array must be reverse sorted with values >= 1" unless histogram_array_valid?(ary)
+    raise "parameter 2 must be between 0.2 and 0.8" unless ((rat >= 0.1) && (rat <= 0.9))
+    sys = []
+    top = (rat * 4096.0).to_i
+    n = ary.count - 1
+    tar = (ary[0] * top) >> 12
+    cnt = 1
+    n.times do |idx|
+      if ary[idx+1] <= tar
+        sys.push cnt
+        cnt = 1
+        tar = (ary[idx+1] * top) >> 12
+      else
+        cnt += 1
+      end
+    end
+    sys.push cnt
+    ary = []
+    borrow = 0
+    sys.each do |num|
+      num -= borrow
+      bin_size = 1 << (num-1).bit_length
+      ary.push bin_size
+      borrow = bin_size - num      
+    end
+    pre = ""
+    if ary.count==1  # don't bother ... all same
+      return "x" * (ary.first.bit_length - 1)
+    end
+    sys = []
+    ary.each do |num|
+      str = pre + "0" + "x" * (num.bit_length-1)
+      pre += "1"
+      sys.push str
+    end
+    sys.sort! &CMP
+    sys.join(':')
+  end  
+end  
+
+
+class Huffnpuff
+  def initialize(sys, fifo, dat = [])
+    @data = dat
+    @fifo = fifo
+    @ary = []
+    systemize!(sys)
+  end
+  def min
+    @min
+  end
+  def max
+    @max
+  end
+  def size
+    @size
+  end
+  
+  def sys()
+    @sys
+  end
+  
+  def systemize!(sys)
+    if sys.kind_of? String
+      @sys = sys
+      ary = sys.split(':')
+    elsif sys.kind_of? Array
+      ary = sys
+      @sys = ary.join(':')
+    else
+      raise "imcompatible type first parameter"
+    end
+    self.class.validate_system_string(@sys)
+    off = 0
+    @min = 2 ** 10000
+    @max = 0
+    ary.each do |ii|
+      pos = ii.index('x')
+      if pos.nil?
+        @ary.push [ii,0, off]
+        off += 1
+        sz = ii.length
+        @min = sz < @min ? sz : @min
+        @max = sz > @max ? sz : @max
+      else
+        xcount = ii.length - pos
+        @ary.push [ii[(0..(pos-1))],xcount,off]
+        off += 2 ** xcount
+        sz = xcount + ii[(0..(pos-1))].length
+        @min = sz < @min ? sz : @min
+        @max = sz > @max ? sz : @max
+      end
+    end
+    @size = off  
+  end
+
+  def push(item)
+    raise "can't excede system size of #{@size}" if @data.size >= @size
+    @data.push item
+  end
+
+  def push_each(item)
+    if item.respond_to? :each
+      item.each do |elm|
+        push elm
+      end
+    elsif item.respond_to? :each_char
+      item.each_char do |ch|
+        push ch
+      end
+    else # punt
+      push item
+    end
+  end  
+  
+  # ok, we have a problem ... the MSB is the preamble
+
+  def get
+    idx = 0
+    @ary.each do |ii|
+      pre = @fifo.get(ii.first.length)  # not enough bits to address data
+      return nil if pre.nil?
+      if(pre==ii.first.to_i(2))
+        if(0==ii[1]) # only one, no x's
+          return @data[ii[2]]
+        else
+          off = @fifo.get(ii[1])
+          if off.nil?
+            @fifo.unget(ii[1]+ii.first.length) # not enough bits to address data
+            return nil
+          end
+          return @data[off + ii[2]]
+        end
+      else # put back to try another system
+        @fifo.unget(ii.first.length)  
+      end
+    end
+    return nil    
+  end
+  
+end

data/lib/huffnpuff/version.rb

@@ -0,0 +1,3 @@
+class Huffnpuff
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,142 @@
+--- !ruby/object:Gem::Specification
+name: huffnpuff
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Bryan Colvin
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-01-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bitfifo
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.2.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.2.1
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.11'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.11'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: byebug
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+description: |
+  Huffnpuff implements a general purpose Huffman Coding system that uses a construction string which can implement any system.
+  This also includes the bitfifo gem which allows variable length bit sequences to address the internal table of the
+  Huffnpuff object instance. Huffman coding is essential for compression techniques such as MPEG, and PKZIP.
+email:
+- bryan@bdlsys.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- huffnpuff.gemspec
+- lib/huffnpuff.rb
+- lib/huffnpuff/version.rb
+homepage: 
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Huffman encoding system
+test_files: []