abacus 0.0.1

data/README.markdown

@@ -0,0 +1,119 @@
+Abacus
+======
+
+Abacus is an xdxf parser and semantic toolset for Ruby.
+
+Installation
+------------
+
+    gem install abacus
+
+Abacus uses a sqllite database to store dictionary data; a config.yml file is used to determine the path of the database file, the default config.yml lies within the gem directory, inside the lib folder. You can point to a different config.yml file by setting the costant ABACUS\_CONFIG\_FILEPATH before requiring the gem. 
+
+The default config.yml file set as database path an ".abacus\_db" folder within the current user home directory, for production environment you may, as stated before, set the ABACUS\_CONFIG\_FILEPATH costant before requiring the gem in order to use a different config.yml file.
+
+Config.yml also store configuration parameters for some tools of the toolset but I'll talk about it later.
+
+After installed Abacus you may want to import some dictionaries in order to start using the semantic tools, the first time you do that you have to execute from command line:
+
+    abacus db:create 
+    
+in order to create the database file. (set ENV[ABACUS\_CONFIG\_FILEPATH] to specify a different config file). 
+
+    ABACUS_CONFIG_FILEPATH=/where/you/want abacus db:create
+
+Then use the import function to load xdxf dictionaries (you can choose from a broad selection from here: http://xdxf.revdanica.com/down/, I used this one: http://downloads.sourceforge.net/xdxf/comn\_sdict05\_eng\_eng\_main.tar.bz2); at the moment the parser is pretty naif and it supports only some tags, but the whole dictionary is already being stored as 'raw_data' during the import so with future releases there might be further improvements also over imported dictionaries. The syntax to import a dict file is:
+
+    abacus db:xdxf:import filaname.xdxf
+    
+The import popolate the db tables with the contents of the dictionary file, multiple dictionaries can be added calling multiple times the above command. 
+
+Navigate the dictionary
+-----------------------
+
+To navigate the dictionary do as following:
+
+    >> require 'abacus'
+    => true
+    >> include Abacus
+    => Object
+    >> Dictionary.all
+    => [#<Abacus::Dictionary id: 1, full_name: "English explanatory dictionary (main)", lang_from: "ENG", lang_to: "ENG", description: nil>]
+    >> Dictionary.first.articles[1000..1002]
+    => [#<Abacus::Article id: 1001, dictionary_id: 1, raw_text: "Camberwell Beauty\nn. a deep purple butterfly, Nymph...">, #<Abacus::Article id: 1002, dictionary_id: 1, raw_text: "Cambodian\nn. & adj. --n. 1 a a native or national o...">, #<Abacus::Article id: 1003, dictionary_id: 1, raw_text: "Cambrian\n\313\210k\303\246mbr\311\252\311\231n adj. & n. --adj. 1 Welsh. 2 ...">]
+    >> Dictionary.first.articles.find(13000).article_keys
+    => [#<Abacus::ArticleKey id: 13000, the_key: "decipher", raw_text: "decipher">]
+    >> ArticleKey.find_by_the_key("ruby").articles.first
+    => #<Abacus::Article id: 34912, dictionary_id: 1, raw_text: "ruby\n\313\210ru:b\311\252 n., adj., & v. --n. (pl. -ies) 1 a ra...">
+    >> ArticleKey.find_by_the_key("ruby").articles.first.raw_text
+    => "ruby\n\313\210ru:b\311\252 n., adj., & v. --n. (pl. -ies) 1 a rare precious stone consisting of corundum with a colour varying from deep crimson or purple to pale rose. 2 a glowing purple-tinged red colour. --adj. of this colour. --v.tr. (-ies, -ied) dye or tinge ruby-colour. \303\270ruby glass glass coloured with oxides of copper, iron, lead, tin, etc. ruby-tail a wasp, Chrysis ignita, with a ruby-coloured hinder part. ruby wedding the fortieth anniversary of a wedding. [ME f. OF rubi f. med.L rubinus (lapis) red (stone), rel. to L rubeus red]"
+    
+    
+There are two main models, Article and ArticleKey, Article is the hub for all the article properties and it contains the raw text (attribute raw_text) taken from xml:
+
+    XML FILE:
+    <ar><k>ironic</k>
+    <tr>aɪˈrɔnɪk</tr> adj. (also ironical) 1 using or displaying irony. 2 in the nature of irony. øøironically adv. [F ironique or LL ironicus f. Gk eironikos dissembling (as IRONY(1))]</ar>
+
+    IRB:
+    >> ArticleKey.find_by_the_key('ironic').articles[0].raw_text
+    => "ironic\na\311\252\313\210r\311\224n\311\252k adj. (also ironical) 1 using or displaying irony. 2 in the nature of irony. \303\270\303\270ironically adv. [F ironique or LL ironicus f. Gk eironikos dissembling (as IRONY(1))]"
+    
+For each article there may be one or more article_keys, which contains the linguistic identifiers of the article itself. Each article key can in turn be related to more than one article but from different dictionaries. 
+
+
+FIRST TOOL: HERIGONE MNEMONIC SYSTEM
+------------------------------------
+
+(detailed explaination of this technique on Wikipedia: http://en.wikipedia.org/wiki/Herigone%27s\_mnemonic_system) Within config.yml you can set a list of association between numbers and letters (the standard one is already written within the standard config file):
+
+Here's a sample config.yml file (this also the default one):
+
+    database:
+      adapter: sqlite3
+      database: <%=File.join(ENV['HOME'] || ENV['USERPROFILE'] || (Abacus::LIB_ROOT + File::SEPARATOR + ".."),'.abacus_db','abacus')%>
+      timeout: 5000
+
+    system:
+      default:
+        0:
+          z,s
+        1:
+          t,d,th
+        2:
+          n
+        3:
+          m
+        4:
+          r
+        5:
+          l
+        6:
+          j,ch,sh,dge
+        7:
+          k
+        8:
+          f,ph,v
+        9:
+          p,b
+
+Following the above instructions you can create your own config file and specify your own system. Multiple systems are supported, simply put the one below the other within config.yml. 
+
+To enhance the existing imported dictionaries with the Hèrigone system you need to launch from commandline:
+ 
+    abacus db:herigone:generate :default [you can change default with your system name]
+    
+Then you can perform some interesting queries as follow:
+    
+    >>  HerigoneNumber.find_by_number(357)
+    => #<Abacus::HerigoneNumber id: 19081, system: "default", number: 357>
+    >>  HerigoneNumber.find_by_number(357).article_keys
+    => [#<Abacus::ArticleKey id: 20159, the_key: "hemlock", raw_text: "hemlock">, #<Abacus::ArticleKey id: 26062, the_key: "milk", raw_text: "milk">, #<Abacus::ArticleKey id: 26068, the_key: "milky", raw_text: "milky">]
+    >>  HerigoneNumber.find_by_number(357).article_keys.map{|a| a.the_key}
+    => ["hemlock", "milk", "milky"]
+    
+    
+CONCLUSIONS
+-----------
+
+If you need more informations please have a look at the source code, or send me an email to sandro dot paganotti at gmail dot com.  

data/bin/abacus

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby 
+
+#= Synopsis 
+#   Abacus helps you search and navigate inside xfxd files, 
+#   use this command line utility to populate the database
+#
+#= Examples
+#   Load a dictionary with this command
+#     abacus load 'path/to/dictionary.xdxf'
+#
+#   Remove a dictionary with
+#     abacus remove 'path/to/dictionary.xdxf'
+#
+#= Usage
+#   abacus [options] {load|remove} source_file
+#
+#= Options
+#   -h --help       print this help page
+#   -v --version    print the application version
+#   no options at the moment
+#
+#= Author
+#   Sandro Paganotti
+#
+#= Copyright
+#   Copyright (c) 2010 Sandro Paganotti. Licensed under the MIT License:
+#   http://www.opensource.org/licenses/mit-license.php
+
+ABACUS_CONFIG_FILEPATH = ENV["ABACUS_CONFIG_FILEPATH"] if !ENV["ABACUS_CONFIG_FILEPATH"].nil?
+require File.join(File.dirname(__FILE__),'..','lib','abacus')
+
+
+case ARGV.shift
+  when 'db:create'
+      # Create database if not exist
+      Abacus::MigrationUtilities.create_database! if !File.exist?(Abacus::Abacus::CONNECTION_STRING['database'])
+    
+  when 'db:xdxf:import'
+      # Import a xdxf file inside the database
+      Abacus::Abacus.import(ARGV.shift, :verbose=>true)
+  
+  when 'db:herigone:generate'
+      # Generate herigone numbers upon a specifc system
+      Abacus::Herigone.new(ARGV.shift).generate_on_db(:verbose=>true)
+
+
+end
+
+
+
+

data/lib/abacus.rb

@@ -0,0 +1,15 @@
+require 'rubygems'
+require 'nokogiri'
+require 'open-uri'
+require 'sqlite3'
+require 'active_record'
+require 'fileutils'
+require 'yaml'
+require 'erb'
+
+module Abacus
+  LIB_ROOT = File.dirname(__FILE__)
+end
+
+require File.join(File.dirname(__FILE__),'abacus','main')
+

data/lib/abacus/apps/herigone.rb

@@ -0,0 +1,45 @@
+module Abacus
+
+  class Herigone
+    
+    attr_reader :name
+    
+    def initialize(name = 'default')
+      @name, @system  = name, Hash[*(CONFIG_FILE['system'][name].to_a.flatten.map{|v| "#{v}"=~/,/ ? v.split(',') : v })]
+      @rev_system = Hash[*(@system.to_a.map{|t| [*t[1]].map{|m|[m,t[0]]}}.flatten)]
+    end
+    
+    def generate_on_db(*args)
+      dictionaries = args.first.is_a?(Hash) ? Dictionary.all : args.shift
+      options      = args.shift  
+       
+      Abacus.transaction do  
+        dictionaries.each do |dict| 
+          dict.articles.each do |art|
+            art.article_keys.each do |ak|
+              if (en = HerigoneNumber.first(:conditions=>{:number=>(n=to_number(ak.the_key)),:system=>@name})).nil?
+                ak.herigone_numbers.create(:number=>n,:system=>@name)
+              else
+                ak.herigone_numbers << en
+              end  
+              puts "<< #{ak.the_key} to #{n}" if options[:verbose]
+            end
+          end
+        end
+      end
+    end
+
+    def to_word(string)
+      raise TypeError.new("Fatal: #{string} is not a number") if (Float(string) rescue false)
+      
+    end
+    
+    def to_number(string)
+      string.downcase.scan(Regexp.compile(@system.values.join('|'))).inject('') do |memo,nr_match|
+         "#{memo}#{@rev_system[nr_match]}"
+      end
+    end
+    
+  end
+  
+end

data/lib/abacus/main.rb

@@ -0,0 +1,23 @@
+# if (Object.const_get("ABACUS_CONFIG_FILEPATH") rescue true)  
+#   ABACUS_CONFIG_FILEPATH = File.join(File.dirname(__FILE__),'..','config.yml') 
+# end
+# puts ABACUS_CONFIG_FILEPATH
+Object.const_get("ABACUS_CONFIG_FILEPATH") rescue begin
+  ABACUS_CONFIG_FILEPATH = File.join(File.dirname(__FILE__),'..','config.yml')
+end
+
+module Abacus
+  CONFIG_FILE = YAML::load(ERB.new(IO.read(ABACUS_CONFIG_FILEPATH)).result)
+end
+
+require File.join(File.dirname(__FILE__), 'models','abacus')
+require File.join(File.dirname(__FILE__), 'models','article')
+require File.join(File.dirname(__FILE__), 'models','article_key')
+require File.join(File.dirname(__FILE__), 'models','article_key_article_join')
+require File.join(File.dirname(__FILE__), 'models','dictionary')
+require File.join(File.dirname(__FILE__), 'models','article_key_herigone_number_join')
+require File.join(File.dirname(__FILE__), 'models','herigone_number')
+
+require File.join(File.dirname(__FILE__), 'apps','herigone')
+
+require File.join(File.dirname(__FILE__), 'utils')

data/lib/abacus/models/abacus.rb

@@ -0,0 +1,42 @@
+module Abacus
+  class Abacus < ActiveRecord::Base
+
+    self.abstract_class = true
+    CONNECTION_STRING = CONFIG_FILE['database']
+      
+    establish_connection CONNECTION_STRING
+  
+    class << self
+    
+      # Import a xdxf file inside the database
+      def import(filename, options = {:verbose=>false})
+        doc = Nokogiri::XML::Document.parse(File.read(filename))
+      
+        Abacus.transaction do 
+       
+          # Info regarding the dict type
+          dic = Dictionary.create!(
+            :full_name => doc.css('full_name').first.content,
+            :lang_from => doc.css('xdxf').first.attributes['lang_from'].value,
+            :lang_to   => doc.css('xdxf').first.attributes['lang_to'].value
+          )
+        
+          # Articles
+          doc.css('ar').each do |ar|
+            (ardb = dic.articles.create!(
+              :raw_text => ar.to_s
+            )).article_keys = ar.css('k').map{ |k|
+              ArticleKey.find_or_create_by_the_key(
+                :the_key  => k.xpath('text()').to_s,
+                :raw_text => k.to_s
+              )
+            }
+            puts "<< #{ardb.article_keys.map{|ak| ak.the_key}.join(",")}"  if options[:verbose]
+          end
+                  
+        end
+      
+      end
+    end
+  end
+end

data/lib/abacus/models/article.rb

@@ -0,0 +1,11 @@
+module Abacus
+  class Article < Abacus
+  
+    validates_presence_of :dictionary_id, :raw_text
+
+    belongs_to :dictionary
+    has_many   :article_key_article_joins, :dependent => :destroy,  :class_name=>"Abacus::ArticleKeyArticleJoin"
+    has_many   :article_keys, :through=>:article_key_article_joins, :class_name=>"Abacus::ArticleKey"
+  
+  end
+end

data/lib/abacus/models/article_key.rb

@@ -0,0 +1,13 @@
+module Abacus
+  class ArticleKey < Abacus
+  
+    validates_presence_of :the_key
+    validates_uniqueness_of :the_key
+
+    has_many   :article_key_article_joins, :dependent => :destroy,              :class_name=>"Abacus::ArticleKeyArticleJoin"
+    has_many   :articles, :through=>:article_key_article_joins,                 :class_name=>"Abacus::Article"
+    has_many   :article_key_herigone_number_joins, :dependent => :destroy,      :class_name=>"Abacus::ArticleKeyHerigoneNumberJoin"
+    has_many   :herigone_numbers, :through=>:article_key_herigone_number_joins, :class_name=>"Abacus::HerigoneNumber"
+  
+  end
+end

data/lib/abacus/models/article_key_article_join.rb

@@ -0,0 +1,8 @@
+module Abacus
+  class ArticleKeyArticleJoin < Abacus
+  
+    belongs_to :article,        :class_name => "Abacus::Article"
+    belongs_to :article_key,    :class_name => "Abacus::ArticleKey"
+  
+  end
+end

data/lib/abacus/models/article_key_herigone_number_join.rb

@@ -0,0 +1,9 @@
+module Abacus
+  class ArticleKeyHerigoneNumberJoin < Abacus
+  
+    belongs_to :herigone_number, :class_name => "Abacus::HerigoneNumber"
+    belongs_to :article_key,     :class_name => "Abacus::ArticleKey"
+  
+  end
+end
+

data/lib/abacus/models/dictionary.rb

@@ -0,0 +1,8 @@
+module Abacus
+  class Dictionary < Abacus
+  
+    validates_presence_of :lang_from, :lang_to, :full_name
+    has_many :articles, :dependent => :destroy, :class_name=>"Abacus::Article"
+  
+  end
+end

data/lib/abacus/models/herigone_number.rb

@@ -0,0 +1,11 @@
+module Abacus
+  class HerigoneNumber < Abacus
+  
+    validates_presence_of   :system, :number
+    validates_uniqueness_of :number, :scope=>:system
+
+    has_many   :article_key_herigone_number_joins, :dependent => :destroy, :class_name=>"Abacus::ArticleKeyHerigoneNumberJoin"
+    has_many   :article_keys, :through=>:article_key_herigone_number_joins,:class_name=>"Abacus::ArticleKey"
+  
+  end
+end

data/lib/abacus/utils.rb

@@ -0,0 +1,57 @@
+
+# TOOLS TO MANAGE Abacus DATABASE
+module Abacus
+
+  class MigrationUtilities
+  
+    class << self    
+    
+      # It creates the database and execute the migrations 
+      def create_database!
+        
+          FileUtils.mkdir_p(File.split(Abacus::CONNECTION_STRING['database']).first)
+        
+          ActiveRecord::Base.establish_connection Abacus::CONNECTION_STRING
+          ActiveRecord::Base.connection
+
+          ActiveRecord::Schema.define do
+            create_table :dictionaries do |t|
+              t.column :full_name, :string
+              t.column :lang_from, :string
+              t.column :lang_to,   :string
+              t.column :description, :text
+            end
+
+            create_table :articles do |t|
+              t.column :dictionary_id, :integer
+              t.column :raw_text,      :text
+            end            
+
+            create_table :article_key_article_joins do |t|
+              t.column :article_id,     :integer
+              t.column :article_key_id, :integer
+            end
+
+            create_table :article_keys do |t|
+              t.column :the_key,     :string
+              t.column :raw_text,    :text
+            end
+            
+            create_table :herigone_numbers do |t|
+              t.column :system,      :string
+              t.column :number,      :integer, :limit=>8
+            end
+
+            create_table :article_key_herigone_number_joins do |t|
+              t.column :article_key_id,             :integer
+              t.column :herigone_number_id,     :integer
+            end
+
+            add_index :herigone_numbers, :number
+            add_index :article_keys,     :the_key
+          end
+      end
+  
+    end
+  end
+end

data/lib/config.yml

@@ -0,0 +1,27 @@
+database:
+  adapter: sqlite3
+  database: <%=File.join(ENV['HOME'] || ENV['USERPROFILE'] || (Abacus::LIB_ROOT + File::SEPARATOR + ".."),'.abacus_db','abacus')%>
+  timeout: 5000
+  
+system:
+  default:
+    0:
+      z,s
+    1:
+      t,d,th
+    2:
+      n
+    3:
+      m
+    4:
+      r
+    5:
+      l
+    6:
+      j,ch,sh,dge
+    7:
+      k
+    8:
+      f,ph,v
+    9:
+      p,b

metadata

@@ -0,0 +1,116 @@
+--- !ruby/object:Gem::Specification 
+name: abacus
+version: !ruby/object:Gem::Version 
+  prerelease: false
+  segments: 
+  - 0
+  - 0
+  - 1
+  version: 0.0.1
+platform: ruby
+authors: 
+- Sandro Paganotti
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-02-28 00:00:00 +01:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: nokogiri
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 1
+        - 3
+        - 3
+        version: 1.3.3
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: sqlite3-ruby
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 1
+        - 2
+        - 5
+        version: 1.2.5
+  type: :runtime
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: activerecord
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 2
+        - 3
+        - 5
+        version: 2.3.5
+  type: :runtime
+  version_requirements: *id003
+description: 
+email: sandro.paganotti@gmail.com
+executables: 
+- abacus
+extensions: []
+
+extra_rdoc_files: 
+- README.markdown
+files: 
+- bin/abacus
+- lib/abacus/apps/herigone.rb
+- lib/abacus/main.rb
+- lib/abacus/models/abacus.rb
+- lib/abacus/models/article.rb
+- lib/abacus/models/article_key.rb
+- lib/abacus/models/article_key_article_join.rb
+- lib/abacus/models/article_key_herigone_number_join.rb
+- lib/abacus/models/dictionary.rb
+- lib/abacus/models/herigone_number.rb
+- lib/abacus/utils.rb
+- lib/abacus.rb
+- lib/config.yml
+- README.markdown
+has_rdoc: true
+homepage: http://github.com/sandropaganotti/Abacus
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.6
+signing_key: 
+specification_version: 3
+summary: Abacus is an xdxf parser and semantic toolset for Ruby.
+test_files: []
+