protk 1.2.0 → 1.2.1

data/README.md

@@ -14,70 +14,103 @@ Protk is a wrapper for various proteomics tools. It aims to present a consistent
  
 Protk depends on ruby 1.9.  The recommended way to install ruby and manage ruby gems is with rvm. Install rvm using this command.
     
-    curl -L https://get.rvm.io | bash -s stable
+```sh
+curl -L https://get.rvm.io | bash -s stable
+```
 
 Next install ruby and protk's dependencies
 
 On OSX
 
-    rvm install 1.9.3 --with-gcc=clang
-    rvm use 1.9.3
-    gem install protk
-    protk_setup.rb package_manager
-    protk_setup.rb system_packages
-    protk_setup.rb all
-
+```sh
+rvm install 1.9.3 --with-gcc=clang
+rvm use 1.9.3
+gem install protk
+protk_setup.rb package_manager
+protk_setup.rb system_packages
+protk_setup.rb all
+```
 On Linux
-    
-    rvm install 1.9.3
-    rvm use 1.9.3
-    gem install protk
-    sudo protk_setup.rb system_packages
-    protk_setup all
 
+```sh    
+rvm install 1.9.3
+rvm use 1.9.3
+gem install protk
+sudo ~/.rvm/bin/rvm 1.9.3 do protk_setup.rb system_packages
+protk_setup all
+```
+
+Instead off using protk_setup.rb all it might be preferable to only install some of the protk tool dependencies.  'all' is just an alias for the following full target list, any of which can be omitted with the consequence that tools depending on that component will not be available.  
+
+```sh
+protk_setup.rb tpp omssa blast msgfplus pwiz openms galaxyenv
+```
 
 ## Sequence databases
 
 After running the setup.sh script you should run manage_db.rb to install specific sequence databases for use by the search engines. Protk comes with several predefined database configurations. For example, to install a database consisting of human entries from Swissprot plus known contaminants use the following commands;
 
-    manage_db.rb add crap
-    manage_db.rb add sphuman
+```sh
+manage_db.rb add --predefined crap
+manage_db.rb add --predefined sphuman
+manage_db.rb update crap
+manage_db.rb update sphuman
+```
+
+You should now be able to run database searches, specifying this database by using the -d sphuman flag.  Every month or so swissprot will release a new database version. You can keep your database up to date using the manage_db.rb update command. This will update the database only if any of its source files (or ftp release notes) have changed. The manage_db.rb tool also allows completely custom databases to be configured. Setup requires adding quite a few command-line options but once setup, databases can easily be updated without further config. The example below shows the commandline arguments required to manually configure the sphuman database.
 
-You should now be able to run database searches, specifying this database by using the -d sphuman flag.  Every month or so swissprot will release a new database version. You can keep your database up to date using;
+```sh
+manage_db.rb add --ftp-source 'ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/reldate.txt' --include-filters '/OS=Homo\ssapiens/' --id-regex 'sp\|.*\|(.*?)\s' --add-decoys --make-blast-index --archive-old sphuman
+```
 
-    manage_db.rb update sphuman
+## Annotation databases
 
-This will update the database only if any of its source files (or ftp release notes) have changed. The manage_db.rb tool also allows completely custom databases to be configured. Setup requires adding quite a few command-line options but once setup databases can easily be updated without further config. The example below shows the commandline arguments required to manually configure the sphuman database.
+The manage_db.rb script also deals with annotation databases.  Actually at this stage the only useful annotation database it has support for is the Swissprot Uniprot database which contains detailed information on protein entries.  The following commands download and index this database.
 
-    manage_db.rb add --ftp-source 'ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/reldate.txt' --include-filters '/OS=Homo\ssapiens/' --id-regex 'sp\|.*\|(.*?)\s' --add-decoys --make-blast-index --archive-old sphuman
+```sh
+manage_db.rb add --predefined swissprot_annotation
+manage_db.rb update swissprot_annotation
+```
 
+Once this step is complete you should be able to use annotation tools such as the uniprot_annotation.rb tool
 
 ## Galaxy integration
 
 Although all the protk tools can be run directly from the command-line a nicer way to run them (and visualise outputs) is to use the galaxy web application.
 
+The preferred method for use of protk with galaxy is to install via the [galaxy toolshed](http://toolshed.g2.bx.psu.edu).  You can find protk tools under the Proteomics heading.  You will still need to install and setup galaxy itself (step 1 below) and you should familiarise yourself with the admin functions of galaxy including tool installation.  Lots of instructions on this are available on the [toolshed wiki](http://wiki.galaxyproject.org/Tool%20Shed).  One final note is that although protk based tools are configured to automatically install their dependencies via the toolshed you should take careful note of the system packages that must be installed before proceeding.  In addition it is also a good idea to install low-level tools (eg the galaxy_protk repository) before installing dependent tools (eg TPP Prophets).
+
 1. Check out and install the latest stable galaxy [see the official galaxy wiki for more detailed setup instructions](http://wiki.g2.bx.psu.edu/Admin/Get%20Galaxy,"galaxy wiki")
 
-        hg clone https://bitbucket.org/galaxy/galaxy-dist 
-		cd galaxy-dist
-		sh run.sh
+    ```sh
+    hg clone https://bitbucket.org/galaxy/galaxy-dist 
+    cd galaxy-dist
+    sh run.sh
+    ```
+
+
+2. Make the protk tools available to galaxy. (Legacy. This step is not needed when installing via the toolshed)
 
-2. Make the protk tools available to galaxy. 
     - Create a directory for galaxy tool dependencies. It's best if this directory is outside the galaxy-dist directory. I usually create a directory called `tool_depends` alongside `galaxy-dist`.
     - Open the file `universe_wsgi.ini` in the `galaxy-dist` directory and set the configuration option `tool_dependency_dir` to point to the directory you just created
     - Create a protkgem directory inside `<tool_dependency_dir>`. 
 
-            cd <tool_dependency_dir>
-            mkdir protkgem
-			cd protkgem
-            mkdir rvm193
-            ln -s rvm193 default
-            cd default
-            ln -s ~/.protk/galaxy/env.sh env.sh
-
-3. Install any of the Proteomics tools that depend on protk from the galaxy toolshed
-
-4. After installing the protk wrapper tools from the toolshed it will be necessary to tell those tools about databases you have installed. Use the manage_db.rb tool to do this. 
-
+        ```sh
+        cd <tool_dependency_dir>
+        mkdir protkgem
+		cd protkgem
+        mkdir rvm193
+        ln -s rvm193 default
+        cd default
+        ln -s ~/.protk/galaxy/env.sh env.sh
+        ```
+
+3. After installing the protk wrapper tools from the toolshed it will be necessary to tell those tools about databases you have installed. Use the manage_db.rb tool to do this.  In particular the manage_db.rb tool has a -G option to automatically tell galaxy about the location of its databases.  To use this though you will need to tell protk about the location of your galaxy install.  To do this
+    - Create a file named `config.yml` inside your .protk directory
+    - Add the line `galaxy_root: /home/galaxy/galaxy-dist` to config.yml substituting the actual path to the root directory of your galaxy installation
+
+        ```sh 
+        echo 'galaxy_root: /home/galaxy/galaxy-dist' > ~/.protk/config.yml
+        ```
 
 

data/bin/gffmerge.rb

@@ -82,12 +82,22 @@ protxml_parser=XML::Parser.file(tool.protxml)
 protxml_doc=protxml_parser.parse
 proteins = protxml_doc.find('.//protxml:protein','protxml:http://regis-web.systemsbiology.net/protXML')
 
-p "Indexing sixframe translations"
-db_filename = Pathname.new(tool.sixframe).realpath.to_s
 
-if tool.skip_fasta_indexing 
+db_filename = nil
+case
+when Pathname.new(tool.sixframe).exist? # It's an explicitly named db  
+  db_filename = Pathname.new(tool.sixframe).realpath.to_s
+else
+  db_filename=Constants.new.current_database_for_name(tool.sixframe)
+end
+
+db_indexfilename = "#{db_filename}.pin"
+
+if File.exist?(db_indexfilename)
+  p "Using existing indexed translations"
   orf_lookup = FastaDB.new(db_filename)
 else
+  p "Indexing sixframe translations"
   orf_lookup = FastaDB.create(db_filename,db_filename,'prot')
 end
 
@@ -114,6 +124,7 @@ for prot in proteins
       p "Looking up #{protein_name}"
       orf = orf_lookup.get_by_id protein_name
       if ( orf == nil)
+        p "Failed lookup for #{protein_name}"
         raise KeyError
       end
 
@@ -121,11 +132,12 @@ for prot in proteins
       position = orf.identifiers.description.split('|').collect { |pos| pos.to_i }
 
       if ( position.length != 2 )
+        p "Badly formatted entry #{orf}"
         raise EncodingError
       end
       orf_name = orf.entry_id.scan(/lcl\|(.*)/)[0][0]
       frame=orf_name.scan(/frame_(\d)/)[0][0]
-      scaffold_name = orf_name.scan(/(scaffold_\d+)/)[0][0]
+      scaffold_name = orf_name.scan(/(scaffold_?\d+)_/)[0][0]
 
       strand = (frame.to_i > 3) ? '-' : '+'
 #      strand = +1
@@ -177,7 +189,6 @@ for prot in proteins
 
     rescue KeyError,EncodingError
       skipped+=0
-      p "Lookup failed for #{protein_name}"
     end
 
     # p orf_name

data/bin/mascot_search.rb

@@ -13,6 +13,130 @@ require 'protk/command_runner'
 require 'protk/search_tool'
 require 'rest_client'
 
+def login(mascot_cgi,username,password)
+
+  authdict={}
+  authdict[:username]=username
+  authdict[:password]=password
+  authdict[:action]="login"
+  authdict[:savecookie]="1"
+
+  p "Logging in to #{mascot_cgi}/login.pl"
+  p authdict
+  response = RestClient.post "#{mascot_cgi}/login.pl", authdict
+
+  cookie = response.cookies
+  cookie
+end
+
+def export_results(mascot_cgi,session_cookie,results_path,format)
+  export_dict={}
+  export_dict[:generate_file]=1
+  export_dict[:pep_query]=1
+  export_dict[:file]=results_path #"../data/20130423/F208623.dat"
+  export_dict[:export_format]=format
+  export_dict[:protein_master]=1
+  export_dict[:prot_hit_num]=1
+  export_dict[:show_unassigned]=1
+  export_dict[:query_title]=1
+  export_dict[:pep_expect]=1
+  export_dict[:pep_rank]=1
+  export_dict[:search_master]=1
+  export_dict[:pep_var_mod]=1
+  export_dict[:pep_isbold]=1
+  export_dict[:report]=0
+  export_dict[:show_queries]=1
+  export_dict[:pep_exp_mz]=1
+  export_dict[:pep_exp_z]=1
+  export_dict[:query_master]=0
+  export_dict[:pep_scan_title]=1
+  export_dict[:query_qualifiers]=1
+  export_dict[:_showsubsets]=1
+  export_dict[:_sigthreshold]=0.99
+  export_dict[:pep_isunique]=1
+  export_dict[:show_header]=1
+  export_dict[:pep_ident]=1
+  export_dict[:query_peaks]=1
+  export_dict[:pep_seq]=1
+  export_dict[:query_raw]=1
+  export_dict[:pep_score]=1
+  export_dict[:show_same_sets]=1
+  export_dict[:do_export]=1
+  export_dict[:peptide_master]=1
+  export_dict[:prot_score]=1
+  export_dict[:prot_acc]=1
+  export_dict[:show_params]=1
+  export_dict[:pep_homol]=1
+  export_dict[:show_mods]=1
+
+  # RestClient.add_before_execution_proc do |req, params|
+  #   require 'debugger'; debugger
+  #   p req
+  #   p params
+  # end
+
+
+  export_url="#{mascot_cgi}/export_dat_2.pl"
+
+  begin
+    RestClient.post(export_url , export_dict , {:cookies=>session_cookie}){ |response, request, result, &block|
+      # require 'debugger'; debugger
+      if ( response.code==303)
+        sleep(5)
+      end
+      response.return!(request, result, &block)
+    }
+#    response = RestClient.post export_url , export_dict , {:cookies=>session_cookie}
+  rescue
+    p "Ignoring exception"
+      # require 'debugger'; debugger
+  end
+
+  begin
+    p response.to_s[0,1000]
+  rescue
+  end
+
+  fout = File.new("results.xml",  "w+")
+
+  fout.write response
+end
+
+def search_params_dictionary(search_tool,input_file)
+  var_mods = search_tool.var_mods.split(",").collect { |mod| mod.lstrip.rstrip }.reject {|e| e.empty? }.join(",")
+  fix_mods = search_tool.fix_mods.split(",").collect { |mod| mod.lstrip.rstrip }.reject { |e| e.empty? }.join(",")
+
+  # None is given by a an empty galaxy multi-select list and we need to turn it into an empty string
+  #
+  var_mods=""  if var_mods=="None"
+  fix_mods="" if fix_mods=="None"
+
+  postdict={}
+  postdict[:SEARCH]="MIS" 
+  postdict[:CHARGE]=search_tool.allowed_charges
+  postdict[:CLE]=search_tool.enzyme
+  postdict[:PFA]=search_tool.missed_cleavages
+  postdict[:COM]="Protk"
+  postdict[:DB]=search_tool.database
+  postdict[:INSTRUMENT]=search_tool.instrument
+  postdict[:IT_MODS]=var_mods
+  postdict[:ITOL]=search_tool.fragment_tol
+  postdict[:ITOLU]=search_tool.fragment_tolu
+  postdict[:MASS]=search_tool.precursor_search_type
+  postdict[:MODS]=fix_mods
+  postdict[:REPORT]="AUTO"
+  postdict[:TAXONOMY]="All entries"
+  postdict[:TOL]=search_tool.precursor_tol
+  postdict[:TOLU]=search_tool.precursor_tolu
+  postdict[:USEREMAIL]=search_tool.email
+  postdict[:USERNAME]=search_tool.username
+  postdict[:FILE]=File.new(input_file)
+  postdict[:FORMVER]='1.01'
+  postdict[:INTERMEDIATE]=''
+
+  postdict
+end
+
 # Environment with global constants
 #
 genv=Constants.new
@@ -20,138 +144,81 @@ genv=Constants.new
 # Setup specific command-line options for this tool. Other options are inherited from SearchTool
 #
 search_tool=SearchTool.new({:msms_search=>true,:background=>false,:database=>true,:explicit_output=>true,:over_write=>true,:msms_search_detailed_options=>true})
+
 search_tool.jobid_prefix="o"
 
 search_tool.option_parser.banner = "Run a Mascot msms search on a set of mgf input files.\n\nUsage: mascot_search.rb [options] file1.mgf file2.mgf ..."
 search_tool.options.output_suffix="_mascot"
 
-search_tool.options.mascot_server="#{genv.default_mascot_server}/mascot/cgi/"
-#search_tool.option_parser.on( '-P', '--http-proxy url', 'The url to a proxy server' ) do |url| 
-#  search_tool.options.mascot_server=url
-#end
-
-#search_tool.options.http_proxy="http://squid.latrobe.edu.au:8080"
-#search_tool.option_parser.on( '-P', '--http-proxy url', 'The url to a proxy server' ) do |url| 
-#  search_tool.options.http_proxy=url
-#end
+search_tool.options.mascot_server="#{genv.default_mascot_server}/mascot/cgi"
 
-search_tool.option_parser.parse!
+search_tool.options.httpproxy=""
+search_tool.option_parser.on( '--proxy url', 'The url to a proxy server' ) do |urll| 
+   search_tool.options.httpproxy=urll
+end
 
+search_tool.options.mascot_password=""
+search_tool.option_parser.on( '--password psswd', 'Password to use when Mascot security is enabled' ) do |psswd| 
+  search_tool.options.mascot_password=psswd
+end
 
+search_tool.options.use_security=FALSE
+search_tool.option_parser.on( '--use-security', 'When Mascot security is enabled this is required' ) do  
+  search_tool.options.use_security=TRUE
+end
 
+search_tool.option_parser.parse!
 
 
-# Set search engine specific parameters on the SearchTool object
-#
 fragment_tol = search_tool.fragment_tol
 precursor_tol = search_tool.precursor_tol
 
 
 
-mascot_cgi=search_tool.mascot_server
+mascot_cgi=search_tool.mascot_server.chomp('/')
 
-unless ( mascot_cgi =~ /^http:\/\//)
+unless ( mascot_cgi =~ /^http[s]?:\/\//)
   mascot_cgi  = "http://#{mascot_cgi}"
 end
 
-#
-RestClient.proxy=search_tool.http_proxy
-
-genv.log("Var mods #{search_tool.var_mods} and fixed #{search_tool.fix_mods}",:info)
-
-var_mods = search_tool.var_mods.split(",").collect { |mod| mod.lstrip.rstrip }.reject {|e| e.empty? }.join(",")
-fix_mods = search_tool.fix_mods.split(",").collect { |mod| mod.lstrip.rstrip }.reject { |e| e.empty? }.join(",")
-
-# None is given by a an empty galaxy multi-select list and we need to turn it into an empty string
-#
-var_mods=""  if var_mods=="None"
-fix_mods="" if fix_mods=="None"
-
-postdict={}
-
-# CHARGE
-#
-postdict[:CHARGE]=search_tool.allowed_charges
-
-# CLE
-#
-postdict[:CLE]=search_tool.enzyme
-
-# PFA
-#
-postdict[:PFA]=search_tool.missed_cleavages
-
-# COM (Search title)
-# 
-postdict[:COM]="Protk"
-
-# DB (Database)
-#
-postdict[:DB]=search_tool.database
+mascot_xcgi = "#{mascot_cgi.chomp('cgi')}x-cgi"
 
-# INSTRUMENT
 #
-postdict[:INSTRUMENT]=search_tool.instrument
+RestClient.proxy=search_tool.httpproxy
 
-# IT_MODS (Variable Modifications)
-#
-postdict[:IT_MODS]=var_mods
-
-# ITOL (Fragment ion tolerance)
-#
-postdict[:ITOL]=search_tool.fragment_tol
-
-# ITOLU (Fragment ion tolerance units)
-#
-postdict[:ITOLU]=search_tool.fragment_tolu
-
-# MASS (Monoisotopic and Average)
-#
-postdict[:MASS]=search_tool.precursor_search_type
+genv.log("Var mods #{search_tool.var_mods} and fixed #{search_tool.fix_mods}",:info)
 
-# MODS (Fixed modifications)
-#
-postdict[:MODS]=fix_mods
 
-# REPORT (What to include in the search report. For command-line searches this is pretty much irrelevant because we retrieve the entire results file anyway)
-#
-postdict[:REPORT]="AUTO"
 
-# TAXONOMY (Blank because we don't allow taxonomy)
-#
-postdict[:TAXONOMY]="All entries"
+cookie=""
+openurlcookie=""
 
-# TOL (Precursor ion tolerance (Unit dependent))
-#
-postdict[:TOL]=search_tool.precursor_tol
+if ( search_tool.use_security)
+  # Login
+  #
+  genv.log("Logging in",:info)
 
-# TOLU (Tolerance Units)
-#
-postdict[:TOLU]=search_tool.precursor_tolu
+  # authdict={}
+  # authdict[:username]=search_tool.username
+  # authdict[:password]=search_tool.mascot_password
+  # authdict[:action]="login"
+  # authdict[:savecookie]="1"
 
-# Email
-#
-postdict[:USEREMAIL]=search_tool.email
+  # response = RestClient.post "#{mascot_cgi}/login.pl", authdict
 
-# Username
-#
-postdict[:USERNAME]=search_tool.username
+  cookie = login(mascot_cgi,search_tool.username,search_tool.mascot_password)
 
+  #cookie = response.cookies
+  openurlcookie = "MASCOT_SESSION=#{cookie['MASCOT_SESSION']}; MASCOT_USERID=#{cookie['MASCOT_USERID']}; MASCOT_USERNAME=#{cookie['MASCOT_USERNAME']}"
+end
 
-# FILE
-#
-postdict[:FILE]=File.new(ARGV[0])
+postdict = search_params_dictionary search_tool, ARGV[0]
 
-postdict[:FORMVER]='1.01'
-postdict[:INTERMEDIATE]=''
 
 genv.log("Sending #{postdict}",:info)
 
-postdict.each do |kv| 
-  p "#{kv}|\n"
-end
 
-search_response=RestClient.post "#{mascot_cgi}/nph-mascot.exe?1",  postdict
+search_response=RestClient.post "#{mascot_cgi}/nph-mascot.exe?1",  postdict, {:cookies=>cookie}
 
 genv.log("Mascot search response was #{search_response}",:info)
 
@@ -168,8 +235,14 @@ else
   results_date=results[1]
   results_file=results[2]
 
+#  results=/master_results_?2?\.pl\?file=(\.*\/data\/.*\/.+\.dat)/.match(search_response)
+#  results_file = results[1]
+#  export_results mascot_cgi,cookie,results_file,"XML"
+
 
-  get_url= "#{mascot_cgi}/../x-cgi/ms-status.exe?Autorefresh=false&Show=RESULTFILE&DateDir=#{results_date}&ResJob=#{results_file}" 
+  get_url= "#{mascot_xcgi}/ms-status.exe?Autorefresh=false&Show=RESULTFILE&DateDir=#{results_date}&ResJob=#{results_file}"
+
+  genv.log("Getting results file at #{get_url}",:info)
 
   if ( search_tool.explicit_output!=nil)
     output_path=search_tool.explicit_output
@@ -181,7 +254,11 @@ else
   #
   require 'open-uri'
   open("#{output_path}", 'wb') do |file|
-    file << open("#{get_url}").read
+    file << open("#{get_url}","Cookie"=>openurlcookie).read
   end
+
+
+#open("F1.dat", 'wb') do |file| file << open("#{get_url}","Cookie" => cookie).read end
+
 end