ncbi-blast-dbs 0.0.4 → 0.0.5

data/README.md

@@ -1,14 +1,15 @@
 ## Fast download BLAST databases from NCBI
 
 Database files (volumes) are downloaded in parallel: number of threads to use
-is determined automatically. MD5 checksum is verified and the database files
-extracted upon download. Database volumes are not downloaded in a particular
-order. Downloads are incremental, that is it will only download new data,
-provided the `.tar.gz` files have not been deleted.
+is determined automatically. MD5 checksum is verified and the database volume
+extracted upon download.  Database volumes are not downloaded in a particular
+order. The volumes are updated if a newer version is available on the server,
+or re-downloaded if corrupt. Aborted downloads are safely resumed.
 
-It is faster than NCBI's `update_blastdb.pl`. But unlike `update_blastdb.pl`,
-which is a pure Perl script, it delegates download and checksum verification
-to `wget` and `md5sum` and is thus not as universal.
+`ncbi-blast-dbs` is faster than NCBI's `update_blastdb.pl`. But unlike
+`update_blastdb.pl`, which is a pure Perl script, `ncbi-blast-dbs` delegates
+download and checksum verification to `wget` and `md5sum` and is thus not as
+universal.
 
 ### Installation
 
@@ -22,7 +23,12 @@ to `wget` and `md5sum` and is thus not as universal.
 
 #### Download all volumes of a BLAST database
 
-    ncbi-blast-dbs nr
+    ncbi-blast-dbs nt nr
+
+Databases are downloaded one after the other while volumes of each database
+are downloaded in parallel.
+
+    ncbi-blast-dbs nt nr
 
 NCBI expects users to submit their email address when downloading data from
 their FTP server. To comply with that, download as:

data/lib/ncbi-blast-dbs.rake

@@ -4,15 +4,16 @@ require 'net/ftp'
 # local copy is older than at the given URL, or if the local copy is corrupt.
 def download(url)
   file = File.basename(url)
-  # Download tarball if the local copy is older than at the given URL or fetch
-  # it for the first time.
+  # Resume an interrupted download or fetch the file for the first time. If
+  # the file on the server is newer, then it is downloaded from start.
+  sh "wget -Nc #{url}"
+  # If the local copy is already fully retrieved, then the previous command
+  # ignores the timestamp. So we check with the server again if the file on
+  # the server is newer and if so download the new copy.
   sh "wget -N #{url}"
-  # Resume aborted download. Do nothing if the file is already fully retrieved
-  # (at the cost is a round trip to server).
-  sh "wget -c #{url}"
 
-  # Always download md5 and verify the tarball. Re-download tarball if corrupt;
-  # extract otherwise.
+  # Immediately download md5 and verify the tarball. Re-download tarball if
+  # corrupt; extract otherwise.
   sh "wget #{url}.md5 && md5sum -c #{file}.md5" do |matched, _|
     if !matched
       sh "rm #{file} #{file}.md5"; download(url)
@@ -22,6 +23,11 @@ def download(url)
   end
 end
 
+# Connects to NCBI's FTP server, gets the URL of all database volumes and
+# returns them grouped by database name:
+#
+#     {'nr' => ['ftp://...', ...], 'nt' => [...], ...}
+#
 def databases
   host, dir = 'ftp.ncbi.nlm.nih.gov', 'blast/db'
   usr, pswd = 'anonymous', ENV['email']
@@ -35,10 +41,14 @@ def databases
   end
 end
 
+# Create user-facing task for each database to drive the download of its
+# volumes in parallel.
 databases.each do |name, files|
   multitask(name => files.map { |file| task(file) { download(file) } })
 end
 
+# List name of all databases that can be downloaded if executed without
+# any arguments.
 task :default do
   puts databases.keys.join(', ')
 end

data/ncbi-blast-dbs.gemspec

@@ -2,7 +2,7 @@ Gem::Specification.new do |s|
   s.authors     = ['Anurag Priyam']
   s.email       = ['anurag08priyam@gmail.com']
   s.name        = 'ncbi-blast-dbs'
-  s.version     = '0.0.4'
+  s.version     = '0.0.5'
   s.summary     = 'Fast download BLAST databases from NCBI.'
   s.description = <<DESC
 Downloads BLAST databases from NCBI. Database files (volumes) are downloaded in

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ncbi-blast-dbs
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-02-16 00:00:00.000000000 Z
+date: 2016-02-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake