sm-transcript 0.0.3

data/LICENSE.txt

@@ -0,0 +1,23 @@
+# $Id: LICENSE.txt 192 2010-03-27 01:24:26Z pwilkins $
+
+Copyright (c) 2010 Massachusetts Institute of Technology
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+

data/README.txt

@@ -0,0 +1,140 @@
+$Id: README.txt 187 2010-03-22 19:32:07Z pwilkins $
+
+sm-transcript reads results of SLS processing and produces transcripts for
+the SpokenMedia browser.  For each file in the source folder whose extension 
+matches the source type, a file of destination type is created in the 
+destination folder.  All of these parameters have default values.  
+
+Requirements:
+	sm-transcript is written in Ruby and packaged as a RubyGem.  Since Ruby is
+	not a compiled language, you will need to have Ruby installed on your 
+	machine to run sm-transcript.  You can determine if Ruby is installed by 
+	typing "ruby -v" at a terminal prompt.  It should return the version of 
+	Ruby that is installed.  If Ruby is not installed on your machine, contact
+	me (or your local Ruby wizard) for assistance.
+	
+Installation:
+	You can get sm-transcript as either a RubyGem or as source from svn.
+	
+	The preferred way to install this package is as a Rubygem.  You can 
+	download the gem from this page:
+	
+	http://TBD
+	
+	and install it with this command:  
+	
+	sudo gem install [--verbose]  <path to gem file>sm-transcript-n.n.n.gem
+	
+	You must use "sudo" to properly install the gem.  If you execute "gem 
+	install" (omitting the "sudo") the gem is installed in your home gem 
+	repository and it isn't in your path without additional configuration.
+	
+	Note: You need sudo privileges to run the command as written.  If you 
+	can't sudo, then you can install it locally and will need some additional
+	configuration.  Contact me (or your local Ruby wizard) for assistance. 
+	 
+	The executable is now in your path.
+	
+	You can cleanly uninstall the gem with this command:
+	
+	sudo gem uninstall sm-transcript	
+	
+	If you have access to our svn repository, you are welcome to check out the 
+	code.  Be warned that the trunk tip is not necessarily stable.  It changes 
+	frequently as enhancements (and bug fixes) are added. (note that the
+	'smb_transcript' in the command line below is not a typo. )
+
+	svn co svn+ssh://svn.mit.edu/oeit-tsa/SMB/smb_transcript/trunk sm_transcript
+	
+	build the gem by running this command from the directory you installed the 
+	source.
+	
+	rake gem
+	
+	The gem will be built and put in ./pkg   You can now use the gem 
+	installation instructions above.
+	
+
+Using the App:
+	Run with no command line parameters, the app reads *.wrd files out of 
+	./results and writes *t1.html files to ./transcripts.  These directories
+	are relative to where sm_transcript is called.
+	
+	Note: destination files are overwritten without a warning prompt.  If you 
+	want to preserve an existing output file, rename it before running the app
+	again.
+	
+	For example, run the app by navigating to the bin folder and running 
+
+		projects/sm_transcript/bin felix$ sm_transcript
+	
+	This command run from this folder will read *.wrd files from bin/results
+	and write *-t1.html to bin/transcripts.
+	
+	Usage: sm_transcript [options]  
+     --srcdir PATH         Read files from this folder (Default: ./results)
+     --destdir PATH        Write files to this folder (Default: ./transcripts)
+     --srctype wrd | seg   Kind of file to process (Default: wrd)
+     --desttype html | ttml   Kind of file to output (Default: html)
+     -h, --help            Show this message	
+
+
+Troubleshooting:
+	sm-transcript requires additional gems to operate.  The RubyGem 
+	installation instructions should install dependencies automatically, but
+	they may not.  If you get an error that includes 
+	
+	... no such file to load -- builder (LoadError)
+	
+	in the first few lines when you run sm-transcript, the problem is a 
+	missing dependent gem.  (the error above indicates that the Builder 
+	gem is missing.)  Try installing the missing gem.  For the error above,
+	command looks like this:
+	
+	sudo gem install builder
+	
+	See "Required Gems" below for more information.
+	
+		
+Upgrading:
+	You can easily upgrade by simply executing the same command you used to 
+	install the gem.  Running install again will add the newer version and make
+	it active.  By default the most recent version is used, but older versions
+	are still available, simply inactive.
+	
+	If are using svn, you should already know what to do.
+	
+	
+Required Gems:
+	builder    - create structured data, such as XML
+	extensions - added for the 'require_relative' command.  (To get this
+	             command in Ruby 1.8 you need to install this gem, for Ruby 1.9
+	             the command is already part of the core.)
+	htmlentities - html parsing
+	json       - create JSON structured data
+	optparse   - option parsing
+	ostruct    - open data structures
+	ppcommand  - pp is a pretty printer.  It is used only for debugging
+	rake       - make for Ruby
+	rubygems   - support for gems
+	shoulda    - enhancement for Test::Unit
+		
+	This command installs gems on OSX and Linux:
+	felix$ sudo gem install <gem name>
+	
+Unit Tests:
+	You may run all unit tests by navigating to the test folder and running 
+	rake with no parameters (the default rake task runs all tests):
+
+	projects/sm_transcript/test felix$ rake 
+
+
+Release Notes:
+	Initial Version - runs under Ruby 1.8. 
+
+To Do:
+	update code to run under Ruby 1.9
+
+	Make this a rubygem, making it available from an OEIT server, rather than
+	from a public gem repository like RubyForge.
+	

data/Rakefile

@@ -0,0 +1,31 @@
+# $Id: Rakefile 190 2010-03-26 22:00:32Z pwilkins $
+
+require 'rake/gempackagetask'
+require 'rake'
+
+spec = Gem::Specification.new do |s| 
+  s.name       = "sm-transcript"
+  s.summary    = "Convert word lists to transcripts"
+  s.description= File.read(File.join(File.dirname(__FILE__), 'README.txt'))
+  s.requirements = [ 'TBD' ]
+  s.version     = "0.0.3"
+  s.author      = "Peter Wilkins"
+  s.email       = "pwilkins@mit.edu"
+  s.homepage    = "http://spokenmedia.mit.edu"
+  s.platform    = Gem::Platform::RUBY
+  s.required_ruby_version = '>=1.8'
+  s.files       = Dir['lib/**/**'] +
+                  Dir['bin/sm-transcript'] + 
+                  Dir['bin/results/PLACEHOLDER.txt'] + 
+                  Dir['bin/transcripts/PLACEHOLDER.txt'] + 
+                  Dir['test/**/**'] +
+                  Dir['README.txt'] +
+                  Dir['LICENSE.txt'] +
+                  Dir['Rakefile'] 
+  s.files.reject! { |fn| fn.include? "process_" }                 
+  s.executables = [ 'sm-transcript' ]
+  s.test_files  = Dir["test/test*.rb"]
+  s.has_rdoc    = false
+end
+ 
+Rake::GemPackageTask.new(spec).define

data/bin/results/PLACEHOLDER.txt

@@ -0,0 +1,8 @@
+# $Id: PLACEHOLDER.txt 186 2010-03-20 14:21:04Z pwilkins $
+
+The bin/results directory provides a default location for input files.  The 
+directory would be empty, but Subversion and Rubygems won't version an empty 
+directory.
+
+This file serves no purpose other than tricking Subversion and Rubygems into 
+creating what might otherwise be an empty directory.  

data/bin/sm-transcript

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby -KU
+# $Id: sm-transcript 192 2010-03-27 01:24:26Z pwilkins $
+# Copyright (c) 2010 Massachusetts Institute of Technology
+# see LICENSE.txt for license text
+
+
+require 'rubygems'
+require 'extensions/kernel'
+require_relative '../lib/sm_transcript/runner'
+
+runner = SmTranscript::Runner.new(ARGV)
+runner.run

data/bin/transcripts/PLACEHOLDER.txt

@@ -0,0 +1,8 @@
+# $Id: PLACEHOLDER.txt 186 2010-03-20 14:21:04Z pwilkins $
+
+The bin/transcripts directory exists to receive transcript files created as
+a result of processing.  The directory would be empty, but Subversion and
+Rubygems won't version an empty directory.
+
+This file serves no purpose other than tricking Subversion and Rubygems into 
+creating what might otherwise be an empty directory.  

data/lib/sm_transcript/LICENSE.txt

@@ -0,0 +1,23 @@
+# $Id$
+
+Copyright (c) 2010 Massachusetts Institute of Technology
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+

data/lib/sm_transcript/metadata.rb

@@ -0,0 +1,69 @@
+# $Id: metadata.rb 183 2010-03-15 19:07:50Z pwilkins $
+# Copyright (c) 2010 Massachusetts Institute of Technology
+# see LICENSE.txt for license text
+
+require "rexml/document"
+require 'extensions/kernel'
+require 'json/ext'
+require_relative 'word'
+
+module SmTranscript
+  class Metadata
+
+    def initialize(metadata)
+      @metadata = metadata
+    end
+    
+    # While the format this method writes is JSON, it writes a 
+    # data.js-like file for compatibility with the IIHS demo player.
+    # There is also a write_json() method which will write a richer 
+    # content set. 
+    def write_datajs(dest_file)
+      # TODO: Do we want to notify user when overwriting existing file?
+      # if File.exists?(dest_file)
+      #   p "overwriting existing destination file"
+      # end
+      return if @metadata.empty?
+      
+      id_val = @metadata['name'].chomp + "-#{id_from_time()}" unless @metadata['name'].nil?
+      title_val = @metadata['title'].chomp unless @metadata['title'].nil?
+      speaker_val = @metadata['speaker'].chomp unless @metadata['speaker'].nil?
+      video_val = @metadata['video'].chomp unless @metadata['video'].nil?
+      
+      File.open(dest_file, "w") do |f|
+        # f.puts 'var SMData = {'
+        tmp = JSON.pretty_generate( 
+          { 'videos' => [
+              'id' => id_val, 
+              'title' => title_val,
+              'speaker' => speaker_val,
+              'video' => video_val,
+              'width' => 320,
+              'height' => 240,
+              'duration' => 523,
+              'preview' => 'preview.png',
+              'transcripts' => {'en-US' => 'transcript.trn'},
+              'audio' => {'en-US' => '../audio/'},
+              'defaultLocale' => 'en-US' 
+            ],
+            'locales' => {'en-US' => 'English (US)'} 
+          }
+        )
+        # p tmp
+        f.puts tmp
+        f.flush
+      end
+    
+    end  # write_datajs()
+    
+    def boilerplate()
+      JSON
+    end
+    
+    def id_from_time()
+      t = Time.now
+      t.strftime("%y%m%d%H%M")
+    end
+    
+  end
+end

data/lib/sm_transcript/metadata_reader.rb

@@ -0,0 +1,56 @@
+# $Id: metadata_reader.rb 182 2010-03-12 22:07:34Z pwilkins $
+# Copyright (c) 2010 Massachusetts Institute of Technology
+# see LICENSE.txt for license text
+
+require 'rubygems'
+require 'extensions/kernel'
+require_relative 'word'
+
+module SmTranscript
+  class MetadataReader
+    attr_reader :metadata
+    attr_reader :words
+
+    def self.from_file(file_name)
+      # p File.expand_path(file_name)
+      new(File.open(file_name))
+    end
+
+    def initialize(src_file)
+      @metadata = {}
+      @words = []
+      parse_metadata(src_file)
+#      parse_words(src_file)
+    end
+
+    def parse_metadata(src_file)
+      # don't process PLACEHOLDER.txt
+      reg = Regexp.new('.*PLACEHOLDER.*')
+      
+      src_file.each do |ln|
+        # break if src_file.lineno > 6  # we're only interested in the 1st 6 lines
+        case src_file.lineno
+        when 1
+          return if reg.match(ln)  
+          @metadata['name'] = ln 
+        when 2
+          @metadata['email'] = ln
+        when 3
+          @metadata['org'] = ln
+        when 4
+          @metadata['title'] = ln
+        when 5
+          @metadata['speaker'] = ln
+        when 6
+          @metadata['video'] = ln
+        else
+          break
+        end
+      end      
+    end
+
+    def parse_words(src_file)
+      # there are currently no timed text in metadata files
+    end
+  end
+end

data/lib/sm_transcript/options.rb

@@ -0,0 +1,89 @@
+# $Id: options.rb 183 2010-03-15 19:07:50Z pwilkins $
+# Copyright (c) 2010 Massachusetts Institute of Technology
+# see LICENSE.txt for license text
+
+require "optparse"
+require 'ostruct'
+
+module SmTranscript
+  class Options
+    
+    SEG_SRC_TYPE = 'seg'
+    WRD_SRC_TYPE = 'wrd'
+    TXT_SRC_TYPE = 'txt'
+    TTML_DEST_TYPE = 'ttml'
+    HTML_DEST_TYPE = 'html'
+    DATAJS_DEST_TYPE = 'datajs'
+    DEFAULT_SRC_DIR   = "./results"
+    DEFAULT_DEST_DIR  = "./transcripts"
+    DEFAULT_SRC_TYPE  = WRD_SRC_TYPE
+    DEFAULT_DEST_TYPE = HTML_DEST_TYPE
+    
+    attr_reader :srcdir
+    attr_reader :destdir
+    attr_reader :srctype
+    attr_reader :desttype
+    attr_reader :options
+    
+    def initialize(argv)
+      @options = OpenStruct.new
+      @srcdir  = DEFAULT_SRC_DIR
+      @destdir = DEFAULT_DEST_DIR
+      @srctype = DEFAULT_SRC_TYPE
+      @desttype = DEFAULT_DEST_TYPE
+      parse(argv)
+    end
+
+    def parse(argv)
+      @options.srcdir  = @srcdir
+      @options.destdir = @destdir
+      @options.srctype = @srctype
+      @options.desttype = @desttype
+      
+      opts = OptionParser.new do |opts|
+        opts.banner = "Usage: sm_transcript [options]"
+        opts.separator " "
+        opts.separator "Specific options:"
+
+        opts.on(
+        "--srcdir PATH", 
+        "Read files from this folder (Default: ./results)") do |sdir|
+          @options.srcdir = @srcdir = sdir 
+        end
+        
+        opts.on(
+        "--destdir PATH", 
+        String, 
+        "Write files to this folder (Default: ./transcripts)") do |ddir|
+          @options.destdir = @destdir = ddir
+        end
+        
+        opts.on("--srctype seg | wrd | txt",
+        "Kind of file to process (Default: seg)") do |stype|
+          @options.srctype = @srctype = stype
+        end
+        
+        opts.on("--desttype html | ttml | datajs",
+        "Kind of format to output (Default: html)") do |dtype|
+          @options.desttype = @desttype = dtype
+        end
+        
+        opts.on("-h", "--help", "Show this message") do
+          puts "\n#{opts}"
+          return
+        end
+
+        begin
+          argv = ["-h"] if argv.empty?
+          opts.parse!(argv)
+        rescue OptionParser::ParseError => e
+          STDERR.puts e.message, "\n", opts
+          exit(-1)
+        end
+      end            
+    end
+  end
+end
+
+
+