tokn 0.0.6 → 0.0.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 75d7206c10817a05dbbd9e5ff36b25f20ef5ad18
-  data.tar.gz: d1d84da2be85c05b567b476b53471ac2a7c6a04f
+  metadata.gz: a3803d745ad9198112d78c381a11c18bd7367789
+  data.tar.gz: 8f6ad5c1ce11a76adb2f677b84dc730581c5d82f
 SHA512:
-  metadata.gz: 47a078062419175ef5fd1d66a46bae95b423c1b69cd8d27a28875589b13abdcdd5474c81e70a9b09f11a27b302f9f9038302fd1ad9121bf81d7db42cf20cea96
-  data.tar.gz: 2bd489cb22b1b68d0a365fd70c337e7cc3bfd1872d501a88e735fe23fe3780a77415262be265263cd906eb7cf32afea7f5e1c8cb849c6feef44567ef80b4717b
+  metadata.gz: a9c7a28d748333fd518476dfa3f8f8b859c425b796d76a7df9e4a09a29b784eaab677da7fbcf220bd3d2c16ccc1451c44f86613a97a78225611628eaf4f4d5d3
+  data.tar.gz: 0fbfeb04dc8ff896533193225b4c8715a5abb1f3ab5e2d691721fd17083bc99357500de2b86fc3e1170711d105ad1ab3c184e7b537521b4aabcd4d36ed4005fc

data/CHANGELOG.txt

@@ -0,0 +1,10 @@
+2013-03-07  
+  * Version 0.0.6 released
+
+2013-04-03
+  * Version 0.0.7 released
+  * Added CHANGELOG.txt
+  * Added link to sample_pdf.DFA 
+  
+  * Version 0.0.8
+  * Fixed problem with link URL's

data/README.txt

@@ -1,46 +1,47 @@
-'tokn' : A ruby gem for constructing DFAs and using them to tokenize text files.
+# @markup markdown
+
+tokn 
+=======
+A ruby gem for constructing DFAs and using them to tokenize text files.
 
 Written and (c) by Jeff Sember, March 2013.
-================================================================================
+
 
 
 Description of the problem
-================================================================================
+------
 
 For a simple example, suppose a particular text file is designed to have
 tokens of the following three types:
 
- 			1) 'a' followed by any number of 'a' or 'b'
- 			2) 'b' followed by either 'aa' or zero or more 'b'
- 			3) 'bbb'
- 			
- We will also allow an additional token, one or more spaces, to separate them.
- These four token types can be written using regular expressions as:
+		'a' followed by any number of 'a' or 'b'
+		'b' followed by either 'aa' or zero or more 'b'
+		'bbb'
+      
+We will also allow an additional token, one or more spaces, to separate them.
+These four token types can be written using regular expressions as:
  
-      sep:  \s
-      tku:  a(a|b)*
-      tkv:  b(aa|b*)
-      tkw:  bbb
+		sep:  \s
+		tku:  a(a|b)*
+		tkv:  b(aa|b*)
+		tkw:  bbb
       
 We've given each token definition a name (to the left of the colon).  
  
 Now suppose your program needs to read a text file and interpret the tokens it
-finds there.  This can be done using the DFA (deterministic finite state automaton)
-shown in figures/sample_dfa.pdf.  The token extraction algorithm is as follows:
+finds there.  
+
+This can be done using the DFA (deterministic finite state automaton) found at: <http://www.cs.ubc.ca/~jpsember/sample_dfa.pdf> 
+
+
+The token extraction algorithm has these steps:
  
- 1) Begin at the start state, S0.
- 2) Look at the next character in the source (text) file.  If there is an arrow (edge)
-    labelled with that character, follow it to another state (it may lead to the
-    same state; that's okay), and advance the cursor to the next character in
-    the source file.
- 3) If there's an arrow labelled with a negative number N, don't follow the edge,
-    but instead remember the lowest (i.e., most negative) such N found.
- 4) Continue steps 2 and 3 until no further progress is possible.
- 5) At this point, N indicates the name of the token found.  The cursor should be
- 	  restored to the point it was at when that N was recorded.  The token's text
- 	  consists of the characters from the starting cursor position to that point.
- 6) If no N value was recorded, then the source text doesn't match any of the tokens,
-    which is considered an error.
+1. Begin at the start state, S0.
+1. Look at the next character in the source (text) file.  If there is an arrow (edge) labelled with that character, follow it to another state (it may lead to the same state; that's okay), and advance the cursor to the next character in the source file.
+1.  If there's an arrow labelled with a negative number N, don't follow the edge, but instead remember the lowest (i.e., most negative) such N found.
+1.  Continue steps 2 and 3 until no further progress is possible.
+1.  At this point, N indicates the name of the token found.  The cursor should be restored to the point it was at when that N was recorded.  The token's text consists of the characters from the starting cursor position to that point.
+1.  If no N value was recorded, then the source text doesn't match any of the tokens, which is considered an error.
    
 
 The tokn module provides a simple and efficient way to perform this tokenization process.
@@ -50,7 +51,7 @@ Such DFAs are very useful, and can be used by non-Ruby programs as well.
 
 
 Using the tokn module in a Ruby program
-===================================================================================
+------
 
 There are three object classes of interest: DFA, Tokenizer, and Token.  A DFA is
 compiled once from a script containing token definitions (e.g, "tku:  b(aa|b*) ..."),
@@ -64,31 +65,26 @@ Here's some example Ruby code showing how a text file "source.txt" can be split
 tokens.  We'll assume there's a text file "tokendefs.txt" that contains the
 definitions shown earlier.
 
-	require "Tokenizer"
-	
-	include Tokn   # Avoids having to prefix things with 'Tokn::'
-	
-	dfa = DFA.from_script(readTextFile("tokendefs.txt"))
-	
-	t = Tokenizer.new(dfa, readTextFile("source.txt"))
-	
-	while t.hasNext
-	
-	      k = t.read      # read token
-				
-				if t.typeOf(k) == "sep"   # skip 'whitespace'
-					next
-				end
-				
-				...do something with the token ...
-	end
-	
+		require "Tokenizer"
+		include Tokn 
+		
+		dfa = DFA.from_script(readTextFile("tokendefs.txt"))
+		t = Tokenizer.new(dfa, readTextFile("source.txt"))
+		
+		while t.hasNext
+		  k = t.read                     # read token
+		  next if t.typeOf(k) == "sep"   # skip 'whitespace'
+		  
+		  ...do something with the token ...
+		
+		end
+  
 If later, another file needs to be tokenized, a new Tokenizer object can be
 constructed and given the same dfa object as earlier.
 
 
 Using the tokn command line utilities
-===================================================================================
+------
 
 The module has two utility scripts: tokncompile, and toknprocess.  These can be
 found in the bin/ directory.
@@ -110,62 +106,62 @@ of tokens from the source file to the standard output:
 
 This will produce the following output:
 
-	WS 1 1 // Example source file that can be tokenized 
-	
-	WS 2 1 
-	
-	ID 3 1 speed
-	WS 3 6  
-	ASSIGN 3 7 =
-	WS 3 8  
-	INT 3 9 42
-	WS 3 11    
-	WS 3 14 // speed of object
-	
-	WS 4 1 
-	
-	ID 5 1 gravity
-	WS 5 8  
-	ASSIGN 5 9 =
-	WS 5 10  
-	DBL 5 11 -9.80
-	WS 5 16 
-	
-	
-	ID 7 1 title
-	WS 7 6  
-	ASSIGN 7 7 =
-	WS 7 8  
-	LBL 7 9 'This is a string with \' an escaped delimiter'
-	WS 7 56 
-	
-	
-	IF 9 1 if
-	WS 9 3  
-	ID 9 4 gravity
-	WS 9 11  
-	EQUIV 9 12 ==
-	WS 9 14  
-	INT 9 15 12
-	WS 9 17  
-	BROP 9 18 {
-	WS 9 19 
-	  
-	DO 10 3 do
-	WS 10 5  
-	ID 10 6 something
-	WS 10 15 
-	
-	BRCL 11 1 }
-	WS 11 2 
-	
+		WS 1 1 // Example source file that can be tokenized 
+		
+		WS 2 1 
+		
+		ID 3 1 speed
+		WS 3 6  
+		ASSIGN 3 7 =
+		WS 3 8  
+		INT 3 9 42
+		WS 3 11    
+		WS 3 14 // speed of object
+		
+		WS 4 1 
+		
+		ID 5 1 gravity
+		WS 5 8  
+		ASSIGN 5 9 =
+		WS 5 10  
+		DBL 5 11 -9.80
+		WS 5 16 
+		
+		
+		ID 7 1 title
+		WS 7 6  
+		ASSIGN 7 7 =
+		WS 7 8  
+		LBL 7 9 'This is a string with \' an escaped delimiter'
+		WS 7 56 
+		
+		
+		IF 9 1 if
+		WS 9 3  
+		ID 9 4 gravity
+		WS 9 11  
+		EQUIV 9 12 ==
+		WS 9 14  
+		INT 9 15 12
+		WS 9 17  
+		BROP 9 18 {
+		WS 9 19 
+		  
+		DO 10 3 do
+		WS 10 5  
+		ID 10 6 something
+		WS 10 15 
+		
+		BRCL 11 1 }
+		WS 11 2 
+  
 The extra linefeeds are the result of a token containing a linefeed.
 
 
 FAQ
-===================================================================================
+--------
 
-1) Why can't I just use Ruby's regular expressions for tokenizing text?
+* Why can't I just use Ruby's regular expressions for tokenizing text?
 
 You could construct a regular expression describing each possible token, and use that
 to extract a token from the start of a string; you could then remove that token from the
@@ -178,7 +174,7 @@ implementations actually 'recognize' a richer class of tokens than the ones desc
 here.  This extra power can come at a cost; in some pathological cases, the running time
 can become exponential.
 
-2) Is tokn compatible with Unicode?
+* Is tokn compatible with Unicode?
 
 The tokn tool is capable of extracting tokens made up of characters that have
 codes in the entire Unicode range: 0 through 0x10ffff (hex).  In fact, the labels
@@ -186,8 +182,3 @@ on the DFA edges can be viewed as sets of any nonnegative integers (negative
 values are reserved for the token identifiers).  Note however that the current implementation
 only reads Ruby characters from the input, which I believe are only 8 bits wide.
 
-3) What do I do if I have some ideas for enhancing tokn, or want to point out some 
-    problems with it?
-    
-Well, I can be reached as jpsember at gmail dot com.
- 

data/README2.txt

@@ -0,0 +1,219 @@
+# @markup rdoc
+
+== 'tokn' : A ruby gem for constructing DFAs and using them to tokenize text files.
+
+Written and (c) by Jeff Sember, March 2013.
+---
+
+
+= Description of the problem
+
+For a simple example, suppose a particular text file is designed to have
+tokens of the following three types:
+
+      [] 'a' followed by any number of 'a' or 'b'
+      [] 'b' followed by either 'aa' or zero or more 'b'
+      [] 'bbb'
+      
+We will also allow an additional token, one or more spaces, to separate them.
+These four token types can be written using regular expressions as:
+ 
+      sep:  \s
+      tku:  a(a|b)*
+      tkv:  b(aa|b*)
+      tkw:  bbb
+      
+We've given each token definition a name (to the left of the colon).  
+ 
+Now suppose your program needs to read a text file and interpret the tokens it
+finds there.  {This can be done using the DFA (deterministic finite state automaton)
+shown here.}[link:sample_dfa.pdf] 
+
+The token extraction algorithm has these steps:
+ 
+1. Begin at the start state, S0.
+1. Look at the next character in the source (text) file.  If there is an arrow (edge) labelled with that character, follow it to another state (it may lead to the same state; that's okay), and advance the cursor to the next character in the source file.
+1.  If there's an arrow labelled with a negative number N, don't follow the edge, but instead remember the lowest (i.e., most negative) such N found.
+1.  Continue steps 2 and 3 until no further progress is possible.
+1.  At this point, N indicates the name of the token found.  The cursor should be restored to the point it was at when that N was recorded.  The token's text consists of the characters from the starting cursor position to that point.
+1.  If no N value was recorded, then the source text doesn't match any of the tokens, which is considered an error.
+   
+
+The tokn module provides a simple and efficient way to perform this tokenization process.
+Its major accomplishment is not just performing the above six steps, but rather that
+it also can construct, from a set of token definitions, the DFA to be used in these steps.
+Such DFAs are very useful, and can be used by non-Ruby programs as well.
+
+
+= Using the tokn module in a Ruby program
+
+There are three object classes of interest: DFA, Tokenizer, and Token.  A DFA is
+compiled once from a script containing token definitions (e.g, "tku:  b(aa|b*) ..."),
+and can then be stored (either in memory, or on disk as a JSON string) for later use.
+
+When tokens need to be extracted from a source file (or simple string), a Tokenizer is
+constructed.  It requires both the DFA and the source file as input.  Once this is done,
+individual Token objects can be read from the Tokenizer.
+
+Here's some example Ruby code showing how a text file "source.txt" can be split into 
+tokens.  We'll assume there's a text file "tokendefs.txt" that contains the
+definitions shown earlier.
+
+
+
+
+
+
+
+		  require "Tokenizer"
+		  
+		  include Tokn  
+		  
+		  dfa = DFA.from_script(readTextFile("tokendefs.txt"))
+		  
+		  t = Tokenizer.new(dfa, readTextFile("source.txt"))
+		  
+		  while t.hasNext
+		  
+		  k = t.read    
+		        
+		    if t.typeOf(k) == "sep" 
+		      next
+		    end
+		        
+		        ...do something with the token ...
+		  end
+  
+ 
+
+lorum epson:
+
+
+
+
+
+
+		  require "Tokenizer"
+		  
+		  include Tokn   # Avoids having to prefix things with 'Tokn::'
+		  
+		  dfa = DFA.from_script(readTextFile("tokendefs.txt"))
+		  
+		  t = Tokenizer.new(dfa, readTextFile("source.txt"))
+		  
+		  while t.hasNext
+		  
+		  k = t.read      # read token
+		        
+		    if t.typeOf(k) == "sep"   # skip 'whitespace'
+		      next
+		    end
+		        
+		        ...do something with the token ...
+		  end
+  
+  
+  
+  
+  
+If later, another file needs to be tokenized, a new Tokenizer object can be
+constructed and given the same dfa object as earlier.
+
+
+= Using the tokn command line utilities
+
+The module has two utility scripts: tokncompile, and toknprocess.  These can be
+found in the bin/ directory.
+
+The tokncompile script reads a token definition script from standard input, and
+compiles it to a DFA.  For example, if you are in the tokn/test/data directory, you can 
+type:
+  
+  tokncompile < sampletokens.txt > compileddfa.txt
+  
+It will produce the JSON encoding of the appropriate DFA.  For a description of how
+this JSON string represents the DFA, see Dfa.rb.
+
+The toknprocess script takes two arguments: the name of a file containing a 
+previously compiled DFA, and the name of a source file.  It extracts the sequence
+of tokens from the source file to the standard output:
+
+  toknprocess compileddfa.txt sampletext.txt
+
+This will produce the following output:
+
+  WS 1 1 // Example source file that can be tokenized 
+  
+  WS 2 1 
+  
+  ID 3 1 speed
+  WS 3 6  
+  ASSIGN 3 7 =
+  WS 3 8  
+  INT 3 9 42
+  WS 3 11    
+  WS 3 14 // speed of object
+  
+  WS 4 1 
+  
+  ID 5 1 gravity
+  WS 5 8  
+  ASSIGN 5 9 =
+  WS 5 10  
+  DBL 5 11 -9.80
+  WS 5 16 
+  
+  
+  ID 7 1 title
+  WS 7 6  
+  ASSIGN 7 7 =
+  WS 7 8  
+  LBL 7 9 'This is a string with \' an escaped delimiter'
+  WS 7 56 
+  
+  
+  IF 9 1 if
+  WS 9 3  
+  ID 9 4 gravity
+  WS 9 11  
+  EQUIV 9 12 ==
+  WS 9 14  
+  INT 9 15 12
+  WS 9 17  
+  BROP 9 18 {
+  WS 9 19 
+    
+  DO 10 3 do
+  WS 10 5  
+  ID 10 6 something
+  WS 10 15 
+  
+  BRCL 11 1 }
+  WS 11 2 
+  
+The extra linefeeds are the result of a token containing a linefeed.
+
+
+= FAQ
+
+1. Why can't I just use Ruby's regular expressions for tokenizing text?
+
+You could construct a regular expression describing each possible token, and use that
+to extract a token from the start of a string; you could then remove that token from the
+string, and repeat.  The trouble is that the regular expression has no easy way to indicate
+which individual token's expression was matched.  You would then (presumably) have to match 
+the returned token with each individual regular expression to identify the token type.
+
+Another reason why standard regular expressions can be troublesome is that their 
+implementations actually 'recognize' a richer class of tokens than the ones described 
+here.  This extra power can come at a cost; in some pathological cases, the running time
+can become exponential.
+
+1. Is tokn compatible with Unicode?
+
+The tokn tool is capable of extracting tokens made up of characters that have
+codes in the entire Unicode range: 0 through 0x10ffff (hex).  In fact, the labels
+on the DFA edges can be viewed as sets of any nonnegative integers (negative
+values are reserved for the token identifiers).  Note however that the current implementation
+only reads Ruby characters from the input, which I believe are only 8 bits wide.
+

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tokn
 version: !ruby/object:Gem::Version
-  version: 0.0.6
+  version: 0.0.8
 platform: ruby
 authors:
 - Jeff Sember
@@ -10,9 +10,9 @@ bindir: bin
 cert_chain: []
 date: 2013-03-07 00:00:00.000000000 Z
 dependencies: []
-description: Given a script containing token descriptions (each a regular expression),
-  tokn compiles an automaton which it can then use to efficiently convert a text file
-  to a sequence of those tokens.
+description: "Given a script containing token descriptions (each a regular expression),
+  \ntokn compiles an automaton which it can then use to efficiently convert a \ntext
+  file to a sequence of those tokens.\n"
 email: jpsember@gmail.com
 executables:
 - tokncompile
@@ -33,16 +33,18 @@ files:
 - lib/tokn/tools.rb
 - bin/tokncompile
 - bin/toknprocess
+- CHANGELOG.txt
 - README.txt
+- README2.txt
 - test/Example1.rb
 - test/data/compileddfa.txt
 - test/data/sampletext.txt
 - test/data/sampletokens.txt
 - test/test.rb
 - test/testcmds
-- figures/sample_dfa.pdf
 homepage: http://www.cs.ubc.ca/~jpsember/
-licenses: []
+licenses:
+- mit
 metadata: {}
 post_install_message: 
 rdoc_options: []