img_scripts 0.0.1 → 0.0.3

data/README.md

@@ -0,0 +1,88 @@
+# img_scripts
+
+A collection of scripts dealing with local stores of IMG data.
+
+## ```img_metadata_scanner.rb```
+Scan through a taxon metadata file, filtering by particular fields. Only certain fields are extracted.
+
+Print out the genus and species of each archaeon:
+```sh
+$ img_metadata_scanner.rb Domain=Archaea --output-fields "Genus,Species" |head
+Thermococcus	gammatolerans
+Methanolobus	
+Pyrobaculum	
+Methanobacterium	sp.
+Sulfolobus	islandicus
+Desulfurococcus	mucosus
+Haladaptatus	paucihalophilus
+Methanothermobacter	thermautotrophicus
+Methanosarcina	barkeri
+Methanobrevibacter	smithii
+```
+
+Print out the headers available for filtering or reporting
+```sh
+$ img_metadata_scanner.rb -l |less
+taxon_oid
+Domain
+Status
+Proposal Name
+ etc.
+ etc.
+```
+
+Randomly sample each species in the genus Shigella:
+```sh
+$ img_metadata_scanner.rb --output-fields "taxon_oid,Genus,Species" Genus=Shigella --sample Species
+649989998 Shigella	dysenteriae
+637000261	Shigella	boydii
+637000265	Shigella	flexneri
+640427143	Shigella	sonnei
+645058835	Shigella	sp.
+```
+In contrast, without sampling:
+```sh
+$ img_metadata_scanner.rb --output-fields "taxon_oid,Genus,Species" Genus=Shigella
+649989998  Shigella	dysenteriae
+641522650	Shigella	boydii
+637000263	Shigella	flexneri
+637000265	Shigella	flexneri
+638341196	Shigella	dysenteriae
+637000264	Shigella	flexneri
+637000261	Shigella	boydii
+640427143	Shigella	sonnei
+646862341	Shigella	flexneri
+640427142	Shigella	dysenteriae
+645058835	Shigella	sp.
+```
+
+
+## The metadata file
+The data comes from a metadata file, which is obtained through the instructions below. You can specify the location of
+this file to the script using the ```--img-metadata-file``` flag, or you can set the ```IMG_METADATA_FILE``` environment
+variable if you are too lazy to type it in each time.
+
+Go to IMG > Genome Browser: http://img.jgi.doe.gov/cgi-bin/w/main.cgi?section=TaxonList&page=taxonListAlpha
+
+In the Table Configuration section:
+* Genome Field     > Click All
+* Project Metadata > Click All
+* Data Statistics  > Click All
+
+Click Display Genomes Again. In the Genome Browser section > Click Select All. Finally, click the Export button.
+
+PS/ Don't trust the IMG metadata _too much_. There are some big mistakes, e.g. in the 16S copy number
+
+PS2/ What have I done to create the FIXED metadata?
+* I have deleted two occurences of "\r" (^M) by ""
+* taxonoid 2515154013 has two extra fields: remove the two cells containing "Human wound, cranian"
+* Replace cells containing "-1" by ""
+* Replaced 'Marine archaeal group 1 BG20 (Nitrosoarchaeum limnia BG20)' by 'Nitrosoarchaeum limnia BG20'
+
+(Download instructions kindly contributed by @fangly / Florent Angly)
+
+## Copyright
+
+Copyright (c) 2013 Ben J. Woodcroft. See LICENSE.txt for
+further details.
+

data/VERSION

@@ -1 +1 @@
-0.0.1
+0.0.3

data/bin/img_metadata_scanner.rb

@@ -4,139 +4,163 @@ require 'optparse'
 require 'bio-logger'
 require 'bio-img_metadata'
 
-if __FILE__ == $0 #needs to be removed if this script is distributed as part of a rubygem
-  SCRIPT_NAME = File.basename(__FILE__); LOG_NAME = SCRIPT_NAME.gsub('.rb','')
-  IMG_METADATA_FILE_ENV_VARIABLE_NAME='IMG_METADATA_FILE'
-  
-  # Parse command line options into the options hash
-  options = {
-    :logger => 'stderr',
-    :output_fields => ['taxon_oid'],
-    :listing_mode => false,
-  }
-  o = OptionParser.new do |opts|
-    opts.banner = "
+SCRIPT_NAME = File.basename(__FILE__); LOG_NAME = SCRIPT_NAME.gsub('.rb','')
+IMG_METADATA_FILE_ENV_VARIABLE_NAME='IMG_METADATA_FILE'
+
+# Parse command line options into the options hash
+options = {
+  :logger => 'stderr',
+  :log_level => 'info',
+  :output_fields => ['taxon_oid'],
+  :listing_mode => false,
+}
+o = OptionParser.new do |opts|
+  opts.banner = "
       Usage: #{SCRIPT_NAME} [-i <img_metadata_file>] <key=value>
-      
+
       Runs through an IMG metadata file, and prints out the IMG identifiers of those genomes that match the key=value criteria. E.g. Kingdom=Archaea to grep for all archaeons.
 
-      Example: 
+      Example:
+
 
 $ img_metadata_scanner.rb Domain=Archaea --output-fields \"Genus,Species\" |head
+
 DEBUG img_metadata_scanner: Using environment variable IMG_METADATA_FILE to define path to IMG metadata file /srv/whitlam/bio/db/img/4.0/metadata/img_metadata_4_0_FIXED.csv
  INFO img_metadata_scanner: Using 1 filters.
  INFO img_metadata_scanner: Found 4700 taxons in the IMG metadata file
 Thermococcus	gammatolerans
-Methanolobus	
-Pyrobaculum	
+Methanolobus
+Pyrobaculum
 Methanobacterium	sp.
 Sulfolobus	islandicus
 Desulfurococcus	mucosus
 Haladaptatus	paucihalophilus
 Methanothermobacter	thermautotrophicus
 Methanosarcina	barkeri
-Methanobrevibacter	smithii
-
-\n\n"
-
-    opts.separator "\nOptions:\n\n"
-    opts.on("-i", "--img-metadata-file PATH", "Path to IMG metadata file [required]. This is not necessary if there is a valid environment variable #{IMG_METADATA_FILE_ENV_VARIABLE_NAME} available.") do |arg|
-      options[:img_metadata_file] = arg
-    end
-    opts.on("-o", "--output-fields FIELDS", "List of output fields, comma separated [default: #{options[:output_fields].join(',')}]") do |arg|
-      options[:output_fields] = arg.split ','
-    end
-    opts.on("-l", "--list", "Instead of filtering print a list of the fields in the output file, newline separated [default: #{options[:listing_mode]}]") do |arg|
-      options[:listing_mode] = true
-    end
-
-    # logger options
-    opts.separator "\nVerbosity:\n\n"
-    opts.on("-q", "--quiet", "Run quietly, set logging to ERROR level [default INFO]") {Bio::Log::CLI.trace('error')}
-    opts.on("--logger filename",String,"Log to file [default #{options[:logger]}]") { |name| options[:logger] = name}
-    opts.on("--trace options",String,"Set log level [default INFO]. e.g. '--trace debug' to set logging level to DEBUG"){|s| Bio::Log::CLI.trace(s)}
-  end; o.parse!
+Methanobrevibacter	smithii\n\n"
 
-  # Setup logging. bio-logger defaults to STDERR not STDOUT, I disagree
-  Bio::Log::CLI.logger(options[:logger]); log = Bio::Log::LoggerPlus.new(LOG_NAME); Bio::Log::CLI.configure(LOG_NAME)
-  
-  # Read in the system-specific ENV variable for the path to the img metadata file unless a path has already been specified 
-  if options[:img_metadata_file].nil?
-    if ENV[IMG_METADATA_FILE_ENV_VARIABLE_NAME].nil?
-      $stderr.puts "img-metadata-file not found as either a command line option or environment variable, failing."
-      $stderr.puts o
-      exit 1
-    else
-      options[:img_metadata_file] = ENV[IMG_METADATA_FILE_ENV_VARIABLE_NAME]
-      log.debug "Using environment variable #{IMG_METADATA_FILE_ENV_VARIABLE_NAME} to define path to IMG metadata file #{options[:img_metadata_file]}"
-    end
+  opts.separator "\nOptions:\n\n"
+  opts.on("-o", "--output-fields FIELDS", "List of output fields, comma separated [default: #{options[:output_fields].join(',')}]") do |arg|
+    options[:output_fields] = arg.split ','
   end
-  unless File.exist?(options[:img_metadata_file])
-    $stderr.puts "IMG metadata file #{options[:img_metadata_file]} not found - was it specified correctly?"
-    exit 2
+  opts.on("-l", "--list", "Instead of filtering print a list of the fields in the output file, newline separated [default: #{options[:listing_mode]}]") do |arg|
+    options[:listing_mode] = true
   end
-  
-
-  
-  # If listing mode is set, don't do any filtering, just list the variables
-  if options[:listing_mode]
-    # Read the IMG metadata
-    img = Bio::IMG::TaxonomyDefinitionFile.read(options[:img_metadata_file])
-    log.info "Found #{img.length} taxons in the IMG metadata file"
-    
-    img[0].attributes.keys.each do |key|
-      puts key
-    end
-    
-    exit 0
+  opts.on("--sample FIELD", "Only return 1 row for each entry type of that column, chosen randomly. E.g. one row per genus: '--sample Genus'. Only 1 column can be sampled at a time, currently. Any filtering is done _before_ sampling. Any rows that are empty in the sampling column are filtered out.[default: no sampling]") do |arg|
+    options[:sampling_column] = arg
   end
-    
-  # Parse the key/value pairs
-  filter_hash = {}
-  unless ARGV[0].nil?
-    ARGV[0].split(',').each do |split|
-      splits2 = split.split('=')
-      unless splits2.length == 2
-        log.error "Badly parsed key/value pair '#{split}', expected exactly 1 = sign, found #{splits2.length}"
-        exit 1
-      end
-      
-      key = splits2[0]
-      unless filter_hash[key].nil?
-        log.error "Duplicate filter key found: #{key}, failing"
-        exit 1
-      end
-      filter_hash[key] = splits2[1]
-    end
+  opts.on("-i", "--img-metadata-file PATH", "Path to IMG metadata file [required]. This is not necessary if there is a valid environment variable #{IMG_METADATA_FILE_ENV_VARIABLE_NAME} available.") do |arg|
+    options[:img_metadata_file] = arg
   end
-  log.info "Using #{filter_hash.length} filters."
   
+  # logger options
+  opts.separator "\nVerbosity:\n\n"
+  opts.on("-q", "--quiet", "Run quietly, set logging to ERROR level [default INFO]") {options[:log_level] = 'error'}
+  opts.on("--logger filename",String,"Log to file [default #{options[:logger]}]") { |name| options[:logger] = name}
+  opts.on("--trace options",String,"Set log level [default INFO]. e.g. '--trace debug' to set logging level to DEBUG"){|s| options[:log_level] = s}
+end; o.parse!
+
+# Setup logging. bio-logger defaults to STDERR not STDOUT, I disagree
+Bio::Log::CLI.logger(options[:logger]); Bio::Log::CLI.trace(options[:log_level]); log = Bio::Log::LoggerPlus.new(LOG_NAME); Bio::Log::CLI.configure(LOG_NAME)
+
+# Read in the system-specific ENV variable for the path to the img metadata file unless a path has already been specified
+if options[:img_metadata_file].nil?
+  if ENV[IMG_METADATA_FILE_ENV_VARIABLE_NAME].nil?
+    $stderr.puts "img-metadata-file not found as either a command line option or environment variable, failing."
+    $stderr.puts o
+    exit 1
+  else
+    options[:img_metadata_file] = ENV[IMG_METADATA_FILE_ENV_VARIABLE_NAME]
+    log.debug "Using environment variable #{IMG_METADATA_FILE_ENV_VARIABLE_NAME} to define path to IMG metadata file #{options[:img_metadata_file]}"
+  end
+end
+unless File.exist?(options[:img_metadata_file])
+  $stderr.puts "IMG metadata file #{options[:img_metadata_file]} not found - was it specified correctly?"
+  exit 2
+end
+
+# If listing mode is set, don't do any filtering, just list the variables
+if options[:listing_mode]
   # Read the IMG metadata
   img = Bio::IMG::TaxonomyDefinitionFile.read(options[:img_metadata_file])
   log.info "Found #{img.length} taxons in the IMG metadata file"
-  
-  # Expect that each of the filters are found in the list of headers available, otherwise filtering will be ineffectual
-  # Ditto for the output names
-  [filter_hash.keys, options[:output_fields]].flatten.each do |key|
-    unless img[0].attributes.keys.include?(key)
-      log.warn "Unable to find column named #{key} in the IMG metadata file - typo perhaps?"
+
+  img[0].attributes.keys.each do |key|
+    puts key
+  end
+
+  exit 0
+end
+
+# Parse the key/value pairs
+filter_hash = {}
+unless ARGV[0].nil?
+  ARGV[0].split(',').each do |split|
+    splits2 = split.split('=')
+    unless splits2.length == 2
+      log.error "Badly parsed key/value pair '#{split}', expected exactly 1 = sign, found #{splits2.length}"
       exit 1
     end
-  end
-  
-  # Go through each row, printing the outputs if they pass the filter
-  img.each do |taxon|
-    passed = true
-    filter_hash.each do |key, value|
-      if taxon.attributes[key] != value
-        passed = false
-        break
-      end
+
+    key = splits2[0]
+    unless filter_hash[key].nil?
+      log.error "Duplicate filter key found: #{key}, failing"
+      exit 1
     end
-    
-    if passed
-      puts options[:output_fields].collect{|field| taxon.attributes[field]}.join("\t")
+    filter_hash[key] = splits2[1]
+  end
+end
+log.info "Using #{filter_hash.length} filters."
+
+# Read the IMG metadata
+img = Bio::IMG::TaxonomyDefinitionFile.read(options[:img_metadata_file])
+log.info "Found #{img.length} taxons in the IMG metadata file"
+
+# Check that the sampling column is a real column
+unless options[:sampling_column].nil? or img[0].attributes.key?(options[:sampling_column])
+  log.error "You asked to sample the column '#{options[:sampling_column]}', but this doesn't appear to be a column in the metadata file"
+end
+
+# Expect that each of the filters are found in the list of headers available, otherwise filtering will be ineffectual
+# Ditto for the output names
+[filter_hash.keys, options[:output_fields]].flatten.each do |key|
+  unless img[0].attributes.keys.include?(key)
+    log.warn "Unable to find column named #{key} in the IMG metadata file - typo perhaps?"
+    exit 1
+  end
+end
+
+# Filter the rows
+filtered = img.select do |taxon|
+  passed = true
+  filter_hash.each do |key, value|
+    if taxon.attributes[key] != value
+      passed = false
+      break
     end
   end
+  passed
+end
+log.debug "After filtering, there were #{filtered.length} taxons"
+
+# Sample from the rows
+if options[:sampling_column]
+  samples = {}
+  filtered.each do |taxon|
+    type = taxon.attributes[options[:sampling_column]]
+    samples[type] ||= []
+    samples[type].push taxon
+  end
+  filtered = samples.values.collect do |group|
+    group.sample
+  end
   
-end #end if running as a script
+  log.debug "After sampling, there were #{filtered.length} taxons"
+end
+
+# Go through each row, printing the outputs
+log.info "Printing out outputs for #{filtered.length} taxons"
+filtered.each do |taxon|
+  puts options[:output_fields].collect{|field| taxon.attributes[field]}.join("\t")
+end
+

data/img_scripts.gemspec

@@ -0,0 +1,63 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = "img_scripts"
+  s.version = "0.0.3"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Ben J. Woodcroft"]
+  s.date = "2013-01-15"
+  s.description = "Scripts related to the IMG (Integrated Microbial Genomes) database"
+  s.email = "donttrustben near gmail.com"
+  s.executables = ["img_metadata_scanner.rb"]
+  s.extra_rdoc_files = [
+    "LICENSE.txt",
+    "README.md"
+  ]
+  s.files = [
+    ".document",
+    "Gemfile",
+    "LICENSE.txt",
+    "README.md",
+    "Rakefile",
+    "VERSION",
+    "bin/img_metadata_scanner.rb",
+    "img_scripts.gemspec",
+    "lib/img_scripts.rb",
+    "test/helper.rb",
+    "test/test_img_scripts.rb"
+  ]
+  s.homepage = "http://github.com/wwood/img_scripts"
+  s.licenses = ["MIT"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = "1.8.15"
+  s.summary = "Scripts related to the IMG (Integrated Microbial Genomes) database"
+
+  if s.respond_to? :specification_version then
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<bio-img_metadata>, [">= 0"])
+      s.add_development_dependency(%q<shoulda>, [">= 0"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_development_dependency(%q<bundler>, [">= 1.0.0"])
+      s.add_development_dependency(%q<jeweler>, [">= 1.8.4"])
+    else
+      s.add_dependency(%q<bio-img_metadata>, [">= 0"])
+      s.add_dependency(%q<shoulda>, [">= 0"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_dependency(%q<bundler>, [">= 1.0.0"])
+      s.add_dependency(%q<jeweler>, [">= 1.8.4"])
+    end
+  else
+    s.add_dependency(%q<bio-img_metadata>, [">= 0"])
+    s.add_dependency(%q<shoulda>, [">= 0"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
+    s.add_dependency(%q<bundler>, [">= 1.0.0"])
+    s.add_dependency(%q<jeweler>, [">= 1.8.4"])
+  end
+end
+

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: img_scripts
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-11 00:00:00.000000000 Z
+date: 2013-01-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bio-img_metadata
-  requirement: &77007880 !ruby/object:Gem::Requirement
+  requirement: &15221740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *77007880
+  version_requirements: *15221740
 - !ruby/object:Gem::Dependency
   name: shoulda
-  requirement: &77007610 !ruby/object:Gem::Requirement
+  requirement: &15221120 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *77007610
+  version_requirements: *15221120
 - !ruby/object:Gem::Dependency
   name: rdoc
-  requirement: &77007320 !ruby/object:Gem::Requirement
+  requirement: &15220340 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: '3.12'
   type: :development
   prerelease: false
-  version_requirements: *77007320
+  version_requirements: *15220340
 - !ruby/object:Gem::Dependency
   name: bundler
-  requirement: &77006960 !ruby/object:Gem::Requirement
+  requirement: &15219580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: 1.0.0
   type: :development
   prerelease: false
-  version_requirements: *77006960
+  version_requirements: *15219580
 - !ruby/object:Gem::Dependency
   name: jeweler
-  requirement: &77006640 !ruby/object:Gem::Requirement
+  requirement: &15218680 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,7 +65,7 @@ dependencies:
         version: 1.8.4
   type: :development
   prerelease: false
-  version_requirements: *77006640
+  version_requirements: *15218680
 description: Scripts related to the IMG (Integrated Microbial Genomes) database
 email: donttrustben near gmail.com
 executables:
@@ -73,15 +73,16 @@ executables:
 extensions: []
 extra_rdoc_files:
 - LICENSE.txt
-- README.rdoc
+- README.md
 files:
 - .document
 - Gemfile
 - LICENSE.txt
-- README.rdoc
+- README.md
 - Rakefile
 - VERSION
 - bin/img_metadata_scanner.rb
+- img_scripts.gemspec
 - lib/img_scripts.rb
 - test/helper.rb
 - test/test_img_scripts.rb
@@ -100,7 +101,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1064213905
+      hash: 1386709472173417088
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -109,7 +110,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.17
+rubygems_version: 1.8.15
 signing_key: 
 specification_version: 3
 summary: Scripts related to the IMG (Integrated Microbial Genomes) database

data/README.rdoc

@@ -1,19 +0,0 @@
-= img_scripts
-
-Description goes here.
-
-== Contributing to img_scripts
- 
-* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
-* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
-* Fork the project.
-* Start a feature/bugfix branch.
-* Commit and push until you are happy with your contribution.
-* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
-* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
-
-== Copyright
-
-Copyright (c) 2013 Ben J. Woodcroft. See LICENSE.txt for
-further details.
-