full_lengther_next 0.0.1

data/History.txt

@@ -0,0 +1,4 @@
+=== 0.0.1 2012-01-30
+
+* 1 major enhancement:
+  * Initial release

data/Manifest.txt

@@ -0,0 +1,27 @@
+bin/download_fln_dbs.rb
+bin/make_user_db.rb
+bin/full_lengther_next
+History.txt
+lib/full_lengther_next/classes/common_functions.rb
+lib/full_lengther_next/classes/fl2_stats.rb
+lib/full_lengther_next/classes/fl_analysis.rb
+lib/full_lengther_next/classes/fl_string_utils.rb
+lib/full_lengther_next/classes/lcs.rb
+lib/full_lengther_next/classes/my_worker.rb
+lib/full_lengther_next/classes/my_worker_manager.rb
+lib/full_lengther_next/classes/orf.rb
+lib/full_lengther_next/classes/sequence.rb
+lib/full_lengther_next/classes/test_code.rb
+lib/full_lengther_next/classes/une_los_hit.rb
+lib/full_lengther_next.rb
+Manifest.txt
+PostInstall.txt
+Rakefile
+README.rdoc
+script
+script/console
+script/destroy
+script/generate
+test
+test/test_full_lengther_next.rb
+test/test_helper.rb

data/PostInstall.txt

@@ -0,0 +1,6 @@
+For more information on full_lengther_next, see * http://www.scbi.uma.es/downloads
+
+Now it is time to install databases using the following command:
+
+$> algo
+

data/README.rdoc

@@ -0,0 +1,147 @@
+= full_lengther_next
+
+* http://www.scbi.uma.es/downloads
+
+== DESCRIPTION:
+
+FULL-LENGTHERNEXT is a tool adapted to NGS technologies, able to work in parallel and in a distributed way to minimise computing time. It is able to classify unigenes to full-length, 5’-end, 3’-end and internal, suggesting which unknown genes are coding or not. It will be also shown that FULL-LENGTHERNEXT fixes frame shifts, one of the main mistake found in wrong entries of full-length sequences databases, and it is a fast tool to compare different transcriptome assemblies.
+
+== FEATURES/PROBLEMS:
+
+* FULL-LENGTHERNEXT uses scbi_mapreduce and thus is able to exploit all the benefits of a cluster environment. It also works in multi-core machines big shared-memory servers.
+
+* It is able to classify unigenes to full-length, 5’-end, 3’-end and internal.
+
+* FULL-LENGTHERNEXT fixes frame shifts.
+
+* It returns the translated protein sequence for the complete genes and the nucleotide sequence with frame shift fixed and highlighting the start and end codon for an easier finding of the gene and the UTR regions.
+
+* FULL-LENGTHERNEXT suggests putative new genes analysing what of the genes classified as unknown are probably coding.
+
+* It produces a stats file useful for assemblies comparison.
+
+== SYNOPSIS:
+
+FULL-LENGTHERNEXT must be fed with a multifasta file containing all unigenes to analyse and which group belongs the organism under study among fungi, human, invertebrates, mammals, plants, rodents or vertebrates, to use the most appropriate databases. Furthermore, it is possible parametrizing the number of cpus to be used (workers), the minimum identity percent (default = 45%) and minimum e value (default = 1e-25) thresholds, the maximum distance between query and subject gene limits (default = 15 amino acids) and a user database of complete proteins if desired.
+
+full_lengther_next -f input.fasta -g [fungi|human|invertebrates|mammals|plants|rodents|vertebrates] -d user_db [options]
+
+
+=== PBS Submission script
+
+$> cat sample_work.sh 
+
+# 12 distributed workers and 1 GB memory per worker:
+#PBS -l select=12:ncpus=1:mpiprocs=1:mem=1gb
+# request 10 hours of walltime:
+#PBS -l walltime=10:00:00
+# cd to working directory (from where job was submitted)
+cd $PBS_O_WORKDIR
+
+# create workers file with assigned node names
+
+cat ${PBS_NODEFILE} > workers
+
+# init seqtrimnext
+source ~seqtrimnext/init_env
+
+time seqtrimnext -t paired_ends.txt -Q fastq -w workers -s 10.0.0
+Once this submission script is created, you only need to launch it with:
+
+qsub sample_work.sh
+
+== REQUIREMENTS:
+
+Ruby 1.9.2
+
+Blast plus 2.24 or greater (prior versions have bugs that produces bad results)
+
+== INSTALL:
+
+=== Installing Blast
+
+*Download the latest version of Blast+ from ftp.ncbi.nlm.nih.gov/blast/executables/release/LATEST/ 
+*You can also use a precompiled version if you like 
+*To install from source, decompress the downloaded file, cd to the decompressed folder, and issue the following commands:
+
+./configure
+make
+sudo make install
+
+=== Installing Ruby 1.9
+
+*You can use RVM to install ruby:
+
+Download latest certificates (maybe you don’t need them):
+
+$ curl -O http://curl.haxx.se/ca/cacert.pem 
+$ export CURL_CA_BUNDLE=`pwd`/cacert.pem # add this to your .bashrc or
+equivalent
+
+Install RVM:
+
+$ bash < <(curl -k https://rvm.beginrescueend.com/install/rvm)
+Setup environment:
+
+$ echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bash_profile
+Install ruby 1.9.2 (this can take a while):
+
+$ rvm install 1.9.2
+Set it as the default:
+
+$ rvm use 1.9.2 --default
+
+
+=== Install Full-LengtherNEXT
+
+Full-LengtherNEXT is very easy to install. It is distributed as a ruby gem. The next command will install Full-LengtherNEXT and all the required gems:
+
+gem install full_lengther_next
+
+
+=== Install and rebuild Full-LengthNEXT databases
+
+Full-LengthNEXT needs some databases to work. You can use the BLASTDB environment variable to to change the default database location. To install them, execute:
+
+$ download_fln_dbs.rb
+
+==== User database
+
+In addition, Full-LengthNEXT is able to use a customised database. It can be created executing:
+
+$ make_user_db.rb
+
+This script only needs two parameters, a database division among fungi, human, invertebrates, mammals, plants, rodents or vertebrates, and a ‘taxon’ corresponds to a specific taxonomic group such as genus, family, or order.
+For example, if our organism under study is a pine, the database division will be ‘plants’ and the taxon may be Pinus, Pinaceae or even Coniferales order, which includes pines and the other of conifers. Therefore, the command line will be:
+
+$ mk_user_db.rb plants Coniferales.
+
+Otherwise, this database must contain only with complete proteins, and formatted with the BLAST command:
+
+makeblastdb -in sequences.fasta -dbtype 'prot' -parse_seqids
+
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2012 - Noe Fernandez & Dario Guerrero, noeisneo@gmail.com & dariogf@gmail.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,37 @@
+require 'rubygems'
+gem 'hoe', '>= 2.1.0'
+require 'hoe'
+require 'fileutils'
+require './lib/full_lengther_next'
+
+Hoe.plugin :newgem
+# Hoe.plugin :website
+# Hoe.plugin :cucumberfeatures
+
+# Generate all the Rake tasks
+# Run 'rake -T' to see list of generated tasks (from gem root directory)
+$hoe = Hoe.spec 'full_lengther_next' do
+  self.developer 'Noe Fernandez & Dario Guerrero', 'noeisneo@gmail.com & dariogf@gmail.com'
+  self.post_install_message = 'PostInstall.txt' # TODO remove if post-install message not required
+  self.rubyforge_name       = self.name # TODO this is default value
+  # self.extra_deps         = [['activesupport','>= 2.0.2']]
+  self.extra_deps         = []
+  # self.extra_deps << ['narray','>=0']
+  # self.extra_deps << ['gnuplot','>=0']
+  # self.extra_deps << ['term-ansicolor','>=1.0.5']
+  self.extra_deps << ['xml-simple','>=1.0.12']
+  self.extra_deps << ['scbi_blast','>=0.0.32']
+  self.extra_deps << ['scbi_mapreduce','>=0.0.29']
+  self.extra_deps << ['scbi_fasta','>=0.1.7']
+  # self.extra_deps << ['scbi_fastq','>=0.0.13']
+  self.extra_deps << ['scbi_plot','>=0.0.6']
+  # self.extra_deps << ['scbi_math','>=0.0.1']
+
+end
+
+require 'newgem/tasks'
+Dir['tasks/**/*.rake'].each { |t| load t }
+
+# TODO - want other tests/tasks run by default? Add them to the list
+# remove_task :default
+# task :default => [:spec, :features]

data/bin/download_fln_dbs.rb

@@ -0,0 +1,197 @@
+#!/usr/bin/env ruby
+
+# 15-2-2011 Noe Fernandez-Pozo
+# Script to download Full-LengtherNext databases.
+# Once in UniProtKB/Swiss-Prot, a protein entry is removed from UniProtKB/TrEMBL.
+
+require 'net/ftp'
+
+################################################### Functions
+
+def conecta_uniprot(my_array, formatted_db_path)
+	
+	$ftp = Net::FTP.new()
+	
+	if !File.exists?('blast_dbs')
+		Dir.mkdir('blast_dbs')
+	end
+	
+	$ftp.connect('ftp.uniprot.org')
+	
+	$ftp.login
+	
+	puts "connected to UniProt"
+	
+	my_array.each do |db_group|
+		puts "Downloading #{db_group}"
+		download_uniprot(db_group, formatted_db_path)
+	end
+	
+	$ftp.chdir("/pub/databases/uniprot/current_release/knowledgebase/complete")
+	$ftp.getbinaryfile("uniprot_sprot_varsplic.fasta.gz", "#{formatted_db_path}/uniprot_sprot_varsplic.fasta.gz")
+	
+	puts "isoform files downloaded"
+	
+	$ftp.close
+	
+end
+
+def download_uniprot(uniprot_group, formatted_db_path)
+	
+	$ftp.chdir("/pub/databases/uniprot/current_release/knowledgebase/taxonomic_divisions")
+	$ftp.getbinaryfile("uniprot_sprot_#{uniprot_group}.dat.gz", "#{formatted_db_path}/uniprot_sprot_#{uniprot_group}.dat.gz")
+	$ftp.getbinaryfile("uniprot_trembl_#{uniprot_group}.dat.gz", "#{formatted_db_path}/uniprot_trembl_#{uniprot_group}.dat.gz")
+	
+	puts "#{uniprot_group} files downloaded"
+	
+end
+
+def filter_incomplete_seqs(file_name, isoform_hash, formatted_db_path)
+	
+	puts "filtering sequences from #{file_name}"
+	
+	# UniProtKB fragments with FT NON_CONS and FT NON_TER features.
+	# 
+	#     * FT NON_TER: The residue at an extremity of the sequence is not the terminal residue. If applied to position 1, this signifies that the first position is not the N-terminus of the complete molecule. If applied to the last position, it means that this position is not the C-terminus of the complete molecule. There is no description field for this key. Examples of NON_TER key feature lines:
+	#       FT NON_TER 1 1
+	#       FT NON_TER 29 29
+	#     * FT NON_CONS: Non-consecutive residues. Indicates that two residues in a sequence are not consecutive and that there are a number of unreported or missing residues between them. Example of a NON_CONS key feature line:
+	#       FT NON_CONS 1683 1684
+	# 
+	# NON_CONS fragments are not indicated as non-consecutive in InterPro and being non-consecutive the match to methods may be incorrect if the method spans the 'break'.
+	
+	newseq=false
+	print_seq=true
+	id=''
+	description = ''
+	organism_name = ''
+	seq = ''
+	organelle = ''
+	
+	file_name =~ /uniprot_([a-z]+)_([a-z]+).dat/
+	db_name = $1
+	output_name = $2
+	db_name.sub!('sprot','sp')
+	db_name.sub!('trembl','tr')
+	
+	if !File.exists?("#{formatted_db_path}/#{db_name}_#{output_name}")
+		Dir.mkdir("#{formatted_db_path}/#{db_name}_#{output_name}")
+	end
+	
+	output_file = File.new("#{formatted_db_path}/#{db_name}_#{output_name}/#{db_name}_#{output_name}.fasta", "w")
+	
+	File.open(file_name).each_line do |line|
+		if (newseq == false)
+			if (line =~ /^AC\s+(\w+);/)
+				id=$1
+				newseq = true
+				description = ''
+				organism_name = ''
+				seq = ''
+				print_seq = true
+				organelle = ''
+			end
+		else
+			if (line =~ /^DE\s+(.+)\;*/)
+				if (description == '')
+					description = $1
+					description.sub!(/RecName: Full=/,'sp=')
+					description.sub!(/SubName: Full=/,'tr=')
+				end
+				if (line =~ /Flags: Fragment/)
+					# puts "#{id} #{line}"
+					print_seq=false
+				end
+			elsif (line =~ /^OS\s+(.+)/)
+				organism_name = $1
+			elsif (line =~ /^OG\s+(.+)/)
+				organelle = $1
+			elsif (line =~ /^FT\s+NON_TER\s+/)
+				print_seq=false
+				# puts "#{id}   NON_TER"
+			elsif (line =~ /^FT\s+NON_CONS\s+(\d+)\s+/)
+				print_seq=false
+				# puts "#{id}   NON_CONS"
+			elsif (line =~ /^\s+([\w\s]+)/)
+				seq += $1
+			elsif (line =~ /^\/\//)
+				seq.gsub!(/\s*/,'')
+				if (seq !~ /^M/i)
+					print_seq=false
+				end
+				newseq = false
+				
+				if (print_seq)
+					output_file.puts ">#{id} #{description} #{organism_name} #{organelle}\n#{seq}"
+					if (!isoform_hash[id].nil?)
+						output_file.puts isoform_hash[id]
+					end
+				end
+			end
+		end
+	end
+	output_file.close
+end
+
+def load_isoform_hash(file)
+	
+	isoform_hash = {}
+	my_fasta = ''
+	acc = ''
+	File.open(file).each do |line|
+		line.chomp!
+		if (line =~ /(^>\w+\|(\w+)\-\d\|.+)/)
+			if (isoform_hash[acc].nil?)
+				isoform_hash[acc]= "#{my_fasta}\n"
+			else
+				isoform_hash[acc]+= "#{my_fasta}\n"
+			end
+			my_fasta = "#{$1}\n"
+			acc = $2
+		else
+			my_fasta += line
+		end
+	end
+
+	# if (isoform_hash[acc].nil?)
+	# 	isoform_hash[acc]= "#{my_fasta}\n"
+	# else
+	# 	isoform_hash[acc]+= "#{my_fasta}\n"
+	# end
+	
+	return isoform_hash
+end
+################################################### MAIN
+
+ROOT_PATH=File.dirname(__FILE__)
+
+if ENV['BLASTDB'] && File.exists?(ENV['BLASTDB'])
+  formatted_db_path = ENV['BLASTDB']
+else # otherwise use ROOTPATH + DB
+  formatted_db_path = File.expand_path(File.join(ROOT_PATH, "blast_dbs"))
+end
+
+ENV['BLASTDB']=formatted_db_path
+puts "Databases will be downloaded at: #{ENV['BLASTDB']}"
+
+
+my_array = ["human","fungi","invertebrates","mammals","plants","rodents","vertebrates"]
+# my_array = ["plants","invertebrates"] # used for a shoter test
+
+conecta_uniprot(my_array, formatted_db_path)
+`gunzip #{formatted_db_path}/*gz`
+
+isoform_hash = {}
+isoform_hash = load_isoform_hash("#{formatted_db_path}/uniprot_sprot_varsplic.fasta")
+
+my_array.each do |db_group|
+
+	filter_incomplete_seqs("#{formatted_db_path}/uniprot_sprot_#{db_group}.dat", isoform_hash, formatted_db_path)
+	filter_incomplete_seqs("#{formatted_db_path}/uniprot_trembl_#{db_group}.dat", isoform_hash, formatted_db_path)
+	
+	`makeblastdb -in #{formatted_db_path}/sp_#{db_group}/sp_#{db_group}.fasta -dbtype 'prot' -parse_seqids`
+	`makeblastdb -in #{formatted_db_path}/tr_#{db_group}/tr_#{db_group}.fasta -dbtype 'prot' -parse_seqids`
+	
+end
+
+puts "download_fln_dbs.rb has finished"

data/bin/full_lengther_next

@@ -0,0 +1,173 @@
+#!/usr/bin/env ruby
+
+# 12-2-2011 Noe Fernandez Pozo.
+# Full-Lengther2 predicts if your sequences are complete, showing you the nucleotide sequences and the translated protein
+
+#------------------------------------------------------------------ parameters entry
+require 'optparse'
+require 'socket'
+
+options = {}
+
+if !File.exists?('logs')
+	Dir.mkdir('logs')
+end
+
+optparse = OptionParser.new do |opts|
+	
+	options[:fasta] = nil
+	opts.on( '-f', '--fasta FILE', 'Fasta input file' ) do |file|
+		options[:fasta] = file
+	end
+	
+	options[:tax_group] = nil
+	opts.on( '-g', '--taxon_group GROUP', "Taxon group, required to use the best databases:\n\t\t\t\t\tfungi\n\t\t\t\t\thuman\n\t\t\t\t\tinvertebrates\n\t\t\t\t\tmammals\n\t\t\t\t\tplants\n\t\t\t\t\trodents\n\t\t\t\t\tvertebrates\n\n" ) do |tax_name|
+		options[:tax_group] = tax_name
+	end
+	
+	options[:user_db] = nil
+	opts.on( '-d', '--blast_db DB_NAME', 'User blast plus database' ) do |db|
+		options[:user_db] = db
+	end
+	
+	# options[:verbose] = nil
+	# opts.on( '-v', '--verbose_mode', "verbose mode\n\n" ) do |verbose|
+	# 	options[:verbose] = verbose
+	# end
+
+	options[:evalue] = 1.0e-25
+	opts.on( '-e', '--evalue EVALUE', 'e value threshold to consider as reliable the orthologue sequence. Default=1.0e-25' ) do |evalue|
+		options[:evalue] = evalue.to_f
+	end
+
+	options[:ident] = 45.00
+	opts.on( '-i', '--identity_percent IDENTITY', 'identity percent threshold to consider as reliable the sequence similarity. Default=45.00' ) do |ident|
+		options[:ident] = ident.to_f
+	end
+
+	options[:distance] = 15
+	opts.on( '-a', '--aas_distance DISTANCE', "distance threshold in aminoacids used for some calculations, the less distance the more strict. Default=15\n\n" ) do |distance|
+		options[:distance] = distance.to_i
+	end
+	
+	options[:workers] = 2
+	opts.on( '-w', '--workers INTEGER/FILE', 'Number of CPUs, or a file containing machine names to launch workers with ssh' ) do |workers|
+    if File.exists?(workers)
+       # use workers file
+       options[:workers] = File.read(workers).split("\n").map{|w| w.chomp}
+       options[:workers].shift
+    elsif (workers.to_i > 0)
+       options[:workers] = workers.to_i
+		else
+			options[:workers] = 2
+    end
+  end
+
+	options[:chunk_size] = 200
+	opts.on( '-c', '--chunk_size SIZE', "Number of sequences processed in each block when parallelization is used. Default=200" ) do |s|
+		options[:chunk_size] = s.to_i
+	end
+	
+	options[:server_ip] = '0.0.0.0'
+  opts.on( '-s', '--server IP', 'Server ip. Can use a partial ip to select the apropriate interface' ) do |server_ip|
+
+    # get list of available ips
+    ip_list = Socket.ip_address_list.select{|e| e.ipv4?}.map{|e| e.ip_address}
+    
+    ip=ip_list.select{|ip| ip.index(server_ip)==0}.first
+    
+    if !ip
+      ip='0.0.0.0'
+      # $LOG.info("No available ip matching #{server_ip}")
+    end
+    # $ .info("Using ip #{ip}")
+    options[:server_ip] = ip
+  end
+
+  options[:port] = 0 #50000
+  opts.on( '-p', '--port PORT', "Server port\n\n" ) do |port|
+    options[:port] = port.to_i
+  end
+  
+
+	# Set a banner, displayed at the top of the help screen.
+	opts.banner = "Usage: full_lengther_2 -f input.fasta -g [fungi|human|invertebrates|mammals|plants|rodents|vertebrates] [options]\n\n"
+
+	# This displays the help screen
+	opts.on( '-h', '--help', 'Display this screen' ) do
+		puts opts
+		exit
+	end
+   
+end
+
+# parse options and remove from ARGV
+optparse.parse!
+# @verbose = options[:verbose]
+
+# if (!@verbose.nil?)
+# 	puts "You have chosen the verbose mode:\n\nInput File:\t#{options[:fasta]}\nTaxon Group:\t#{options[:tax_group]}\nOwn Database:\t#{options[:user_db]}\nCPU Number:\t#{options[:workers]}"
+# end
+
+#----------------------------------------------------------------------- testing errors in parameters entry
+if (options[:fasta].nil?) || (options[:tax_group].nil?)
+		puts "incorrect number of arguments, you need a fasta file and a taxonomical group:\n\n\t"
+		puts optparse.help
+		exit
+end
+#----------------------------------------------------------------------- loading classes and gems
+ROOT_PATH=File.dirname(__FILE__)
+
+# $: << File.expand_path(File.join(ROOT_PATH, "classes"))
+
+# load gem path, only to test locally
+# $: << File.expand_path('~/progs/ruby/gems/full_lengther_next/lib')
+
+require 'full_lengther_next'
+
+if ENV['BLASTDB'] && File.exists?(ENV['BLASTDB'])
+  formatted_db_path = ENV['BLASTDB']
+else # otherwise use ROOTPATH + DB
+  formatted_db_path = File.expand_path(File.join(ROOT_PATH, "blast_dbs"))
+end
+
+ENV['BLASTDB']=formatted_db_path
+puts "Using databases at: #{ENV['BLASTDB']}"
+
+if !File.exists?("#{ENV['BLASTDB']}/sp_#{options[:tax_group]}/sp_#{options[:tax_group]}.fasta.psq")
+  puts "DB File #{ENV['BLASTDB']}/sp_#{options[:tax_group]}/sp_#{options[:tax_group]}.fasta.psq doesn't exists, or"
+	puts "incorrect taxon group name: #{options[:tax_group]} choose:"
+	puts optparse.help
+	exit
+end
+
+require 'scbi_blast' # is a gem
+require 'scbi_mapreduce'
+# puts $:
+require 'fl_string_utils'
+require "une_los_hit"
+require "lcs" # like the class simliar of seqtrim, return the longest common sequence
+require "test_code"
+
+##########################################################  MAIN  #################################################################
+
+require 'my_worker_manager'
+
+$LOG = Logger.new(STDOUT)
+$LOG.datetime_format = "%Y-%m-%d %H:%M:%S"
+
+custom_worker_file = File.join(ROOT_PATH,'classes','my_worker.rb')
+
+	$LOG.info 'Starting server'
+	# initialize work manager (open files, etc)
+	MyWorkerManager.init_work_manager(options, options[:chunk_size])
+	
+	# Create server
+	server = ScbiMapreduce::Manager.new(options[:server_ip],options[:port], options[:workers], MyWorkerManager,custom_worker_file, STDOUT,File.join(ROOT_PATH,'init_env'))
+	server.chunk_size=options[:chunk_size]
+	# launch server
+	server.start_server
+	
+	$LOG.info 'Closing server'
+
+	puts "\nGracias por utilizar Full-LengtherNEXT"