bioroebe 0.10.80

data/doc/blosum.md

@@ -0,0 +1,5 @@
+This file describes the references towards BLOSUM.
+
+There are different BLOSUM matrices out there and they can
+be found in either blosum/ or the yaml/ subdirectory of
+the bioroebe project.

data/doc/compatibility/BIO_PHP.md

@@ -0,0 +1,37 @@
+One goal of the Bioroebe project is to offer everything that BioPHP
+offers as well, in ruby.
+
+BioPHP can be found here:
+
+  http://www.biophp.org/
+
+This file here (BIO_PHP.md) keeps track of this ongoing endeavour.
+
+Not all functionality will be ported, though. For example,
+the "PCR Amplification" functionality of biophp is utterly
+useless.
+
+First, some more relevant links:
+
+  http://www.biophp.org/resources.php?mode=functions
+
+Status of the rewrites:
+
+  Restriction digest of DNA                       |                           | http://www.biophp.org/minitools/restriction_digest/demo.php
+  Find Palindromic Sequences                      | [IMPLEMENTED FULLY]
+  Protein to DNA                                  | [IMPLEMENTED FULLY]
+  Sequence manipulation and data                  | [not implemented yet]
+  Melting Temperature (Tm) Calculator             | [not implemented yet]
+  PCR Amplification                               | [will not be implemented] | http://www.biophp.org/minitools/pcr_amplification/
+  Microsatellite Repeats Finder                   | [not implemented yet]
+  Alignment of DNA/Protein sequences              | http://www.biophp.org/functions/alignment_of_dna/
+  DNA to protein                                  | [not implemented yet]
+  Microarray analysis: adaptive quantification    | [not implemented yet]
+  Protein sequence information                    | [not implemented yet]
+  Reduced alphabets for proteins                  | [not implemented yet]
+  Chaos Game Representation                       | [not implemented yet]
+  GC-, AT-, KETO- and oligo-skews generator       | [not implemented yet]
+  Oligonucleotide Frequency                       | [not implemented yet]
+  Oligonucleotides for distance among sequences   | [not implemented yet]
+  Random DNA                                      | http://www.biophp.org/functions/random_dna/
+  Useful formulas                                 |

data/doc/compatibility/README.md

@@ -0,0 +1,3 @@
+This directory may contain documentation in regards to other
+software suites - in particular bio-php and the Emboss software
+suite.

data/doc/compatibility/emboss.md

@@ -0,0 +1,56 @@
+This file will keep track of which functionality of EMBOSS is either
+fully implemented, partially implemented or not yet implemented.
+
+Implemented here means "available as part of the Bioroebe" project -
+either via some webinterface or as its own standalone file.
+
+Overview:
+
+  aligncopy            |
+  aligncopypair        |
+  biosed               |
+  cgplot               | https://www.bioinformatics.nl/cgi-bin/emboss/cpgplot
+  codcopy              |
+  cutseq               | [90% IMPLEMENTED; missing output format possibilities]
+  degapseq             |
+  descseq              |
+  entret               |
+  extractalig          |
+  extractfeat          |
+  extractseq           |
+  featcopy             |
+  featmerge            |
+  featreport           |
+  feattext             |
+  listor               | https://www.bioinformatics.nl/cgi-bin/emboss/help/listor
+  makenucseq           |
+  makeprotseq          |
+  maskambignuc         |
+  maskambigprot        |
+  maskfeat             |
+  maskseq              |
+  newseq               |
+  nohtml               |
+  noreturn             | [Will not be re-implemented, as it is not necessary.]
+  nospace              |
+  notab                |
+  notseq               |
+  nthseq               |
+  nthseqset            |
+  pasteseq             |
+  revseq               |
+  seqcount             | [Will not be re-implemented, as it is not necessary.]
+  seqret               |
+  seqretsetall         |
+  seqretsplit          |
+  sizeseq              |
+  skipredundant        |
+  skipseq              | https://www.bioinformatics.nl/cgi-bin/emboss/skipseq
+  splitsource          |
+  splitter             |
+  trimest              |
+  trimseq              | Implemented in the file bioroebe/utility_scripts/compacter.rb.
+  trimspace            |
+  union                |
+  vectorstrip          |
+  yank                 |

data/doc/extensive_usage_example.md

@@ -0,0 +1,35 @@
+This file here attempts to note down what should be possible and doable 
+within the Bioroebe main sequence class.
+
+It is closely modelled towards this file here from the bioruby project:
+
+    https://raw.githubusercontent.com/bioruby/bioruby/master/doc/Tutorial.rd
+
+There will not be a lot of explanation going on here in this file - the
+whole purpose of this file is to simply show what is possible with the
+Bioroebe sequence, in a succinct manner.
+
+For a more extensive explanation, either consider looking at the online
+documentation for the various methods/classes, or simply have a look at
+the .md file (README.md) that is distributed with the bioroebe-gem
+as such.
+
+    require 'bioroebe'
+
+    seq = Bioroebe::Sequence.new("ATGCATGCAAAA")
+    seq.complement                 # => "TTTTGCATGCAT"
+    seq.subseq(3,8)                # => "GCATGC"
+    seq.composition                # => {"A"=>6, "T"=>2, "G"=>2, "C"=>2}
+    seq.gc_percent                 # => 33
+    seq.translate                  # => "MHAK"
+    seq.translate(2)               # => "CMQ" # This translates from frame 2
+    seq.translate(:frame_two)      # => "CMQ" # Alias to the above.
+    seq.translate(:from_frame_two) # => "CMQ" # Alias to the above.
+    seq.translate.names            # => ["methionine", "histidine", "alanine", "lysine"]
+    seq.translate.codes            # => ["Met", "His", "Ala", "Lys"]
+    seq.translate(1,11)            # codon table 11
+    seq.translate(2)               # translate from frame 2 ==> "CMQ"
+    seq.translate(1,11)            # codon table 11 ==> "MHAK"
+    seq.translate.composition      # => {"K"=>1, "A"=>1, "M"=>1, "H"=>1}
+    seq.translate.molecular_weight # => 485.605
+    seq.complement.translate       # => "FCMH"

data/doc/german_names_for_the_aminoacids.md

@@ -0,0 +1,27 @@
+# =========================================================================== #
+# === German names for the aminoacids
+#
+# This file just carries the german names for the specific aminoacid
+# residues.
+# =========================================================================== #
+
+Alanin:         -CH3
+Arginin:        -CH2CH2CH2NH-C(NH)NH2
+Asparagin:      -CH2CONH2
+Asparaginsäure: -CH2COOH
+Cystein:        -CH2SH
+Glutamin:       -CH2CH2CONH2
+Glutaminsäure:  -CH2CH2COOH
+Glycin:         -H
+Histidin:       -CH2(C3H3N2)
+Isoleucin:      -CH(CH3)CH2CH3
+Leucin:         -CH2CH(CH3)2
+Lysin:          -CH2CH2CH2CH2NH2
+Methionin:      -CH2CH2SCH3
+Phenylalanin:   -CH2(C6H5)
+Prolin:         -CH2CH2CH2-
+Serin:          -CH2OH
+Threonin:       -CH(OH)CH3
+Tryptophan:     -CH2(C8H6N)
+Tyrosin:        -CH2(C6H4)OH
+Valin:          -CH(CH3)2

data/doc/instructions_for_the_taxonomy_subproject.md

@@ -0,0 +1,504 @@
+=== ABOUT
+
+This document refers to the taxonomy/ subdirectory of the bioroebe
+project.
+
+The Taxonomy module is written in ruby and loosely built around the
+official NCBI Taxonomy Database, which is available via ftp at
+the following URL:
+
+    ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
+
+=== INSTALLATION
+
+The taxonomy-related code is distributed with the bioroebe project,
+so if you install the latter then you will also have the Taxonomy
+submodule available.
+
+The archetypical way to install bioroebe is through this way:
+
+    gem install bioroebe
+
+Sometimes you may not have server-wide access or it may be unwanted
+to install it globally.
+
+In this case, you can install bioroebe into the home directory by
+using the --user-install comandline switch.
+
+Example:
+
+    gem install --user-install bioroebe
+
+If you have a local .gem then you can also install this, e. g.
+via:
+
+    gem install --user-install ./*.gem
+
+The Taxonomy module itself can be used primarily for the following
+three main tasks:
+
+(1)
+
+- Update the remote NCBI Taxonomy database into a local database:
+
+  The URL that will be used for this main task can be found here:
+
+    ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
+
+  This URL is assigned to the constant called REMOTE_URL, in 
+  the file constants/constants.rb of the Taxonomy module.
+
+  You can obtain the file path of this file inside the interactive
+  module by issuing "shared_code?" as a command, without the quotes,
+  or from the very commandline itself, via:
+
+    taxonomy shared_code?
+    taxonomy scode?       # Shortcuts are also possible.
+
+  To update the NCBI database (which will download the remote
+  taxonomy database), issue this command:
+
+    update ncbi_database
+
+  Also see the rest of the file for a more thorough explanation.
+
+(2)
+
+- Generate and Populate a table that holds information from(2)
+  the /data/curated/* directory subset, and subsequently,
+  from localome/ as well.
+
+  This is the dataset that derives from local datasets.
+  The directory that will fetch this dataset is derived from 
+  incoming/.
+
+(3)
+
+- Query data from the dataset interactively, via the file 
+  interactive.rb, but also from the commandline directly.
+
+  Note Bene: all commands that work from the file interactive.rb
+  will also work from the commandline.
+
+The following examples will demonstrate how all of this can be
+done - some more options are available, though not all of them
+are extensively documented and commented; see the case menu in
+the file interactive.rb for a really comprehensive and complete
+list.
+
+Without further ado, let's start:
+
+(1) In order to update the (local mirror of the) NCBI Taxonomy,
+    'update ncbi' as instruction to use should suffice:
+
+      update ncbi
+      update_ncbi # <-- This works as well, if you dislike the ' ' 
+
+    This instruction will attempt to download the Taxonomy database,
+    extract the archive, generate the necessary .sql files and
+    then connect to postgresql, so that it can then add this
+    information into the postgresql database.
+
+    For this to work, a connection to the database should be 
+    established prior to running this command.
+
+    Note that the login information is stored in the file
+    postgresql_login_command.rb, so if at any moment in time
+    the information how to access the postgreSQL database gets
+    changed, that file can be changed as well.
+
+    The location of that file can be queried via the instruction:
+      'login_information?'
+    so:
+
+      taxonomy login_information?
+
+    And if you want to modify it in vim, simply do: 'edit login'.
+
+      taxonomy edit login
+      taxonomy edit_login
+
+    At the time of writing this file, the default location mirror
+    for the download of the NCBI Taxonomy database was:
+
+      /tmp/robert/Downloads
+
+    This of course can be changed, also in shared/shared.rb,
+    see the constant DOWNLOAD_DIR. Simply set it to another directory
+    if you wish to change the download location.
+
+    Also note that generating the .sql files and populating the 
+    postgresql table from that information, may take about an
+    hour. This could possibly be optimized by using COPY rather
+    than INSERT INTO statements, but it seemed better to not 
+    spend too much time into researching how to optimize and
+    instead see that more features are made possible in general.
+
+(2) Generating and populating the second table, the one that holds all
+    the information from the "localomes", can most conveniently
+    be done by issuing either 'default' or simply by doing
+    'update localome'.
+
+    This requires a working postgresql database at present as well.
+
+    * Note that the above two commands can be combined by issuing
+     'update all' or 'update everything', but this will also 
+     update (overwrite) the localome table, so 'update ncbi'
+     is usually better. (If we want to build up the localome
+     table anew anyway, we can safely use "update localome", which
+     will take about 24 minutes.)
+
+(3) The third part, the query subsection, is the largest part of the
+    Taxonomy module. There are many aliases to the main instructions
+    to have - for instance, rather than having to type 'update', 'upd'
+    would also suffice, and so it is for many other commands as well;
+    most of them have shorter aliases.
+
+    To start the interactive shell, you can either invoke the file
+    interactive.rb directly, or you just call the 'taxonomy'
+    file like so:
+
+      taxonomy interactive
+
+    Of course you can also use 'taxonomy' without having to use the
+    interactive component, i.e. from the commandline directly.
+
+    Once inside the interactive shell, typing 'help' will give 
+    some information about the subcommands available.
+
+    Now let's look at a few specific examples:
+
+      taxid 33 # Obtain the Organism that has taxID 33, which is
+               # Myxococcus fulvus. This queries the NCBI dataset.
+
+      scientific_name 33 # This will attempt to fetch the scientific
+                         # name of an organism. "sname" is a shorter
+                         # alias.
+
+      parent_id       33 # In order to obtain the ParentID, use
+                         # this command. In our case, it gives us
+                         # back 32, so we now know that the parentID
+                         # of 33 is 32, so 33 -> 32.
+
+      last_update? # This tells us when we last updated the NCI
+                   # database. The constant SHALL_WE_LOG_LAST_UPDATE
+                   # determines whether we will log - set it to false
+                   # if you don't need logging.
+
+    The update() or lupdate() method can be used to specifically
+    update a database. For instance:
+
+      update ncbi      # Update the NCBI database.
+      update localomes # Update the database from localomes.
+      update lineage   # Update the lineage - this is only required after 
+                       # the NCBI database was downloaded (at the weekend).
+
+      update all       # This will perform both updates, first ncbi,
+                       # then localomes. Also note that an update
+                       # in this way will drop all tables, which
+                       # is ok for the NCBI database, but possibly
+                       # unwanted for the localomes database. In general,
+                       # it is recommended to do only "update ncbi".
+
+      available?       # Query which species are available in the localomes table,
+                       # in total. This is quite similar to doing this:
+                       # name,taxid,path
+
+      size?            # This instruction queries the size of the 2 NCBI
+                       # tables, names and nodes, and the localomes (should
+                       # be less than 200 MB in total). "table_size?" is
+                       # a more explicit alias to this. In order for this
+                       # to correctly work, these entries must exist of
+                       # course.
+
+    Helpful queries can be done like so:
+
+      query localome   # Queries how many entries are in the localome.
+                       # "localome?" or just "loc?" is an alias to this.
+
+    More specific queries can be simplified by separating with a ',' character,
+    and will default to the localome table, such as inputting the following
+    as examples:
+
+      taxid,name # Show taxid,name relation from the localome table.
+      name,taxid # As above, but in reverse order - name first, then taxid.
+      name,taxid,path # As above, but also includes the local path.
+      name,path,taxid # As above but a different order.
+      taxid,name,path,filesize # Similar to the above but also gives us the filesize.
+      taxid,comment_field # Show the taxid and the comment_field
+      name,taxid,comment_field
+      name,taxid,comment_field,filesize
+      taxid,lineage_ids # Display taxid and lineage_ids.
+      taxid,lineage_ids,path
+
+    There are ten entries in the localome table, so you can combine these
+    ten in any way to yield whatever information you desire.
+
+    To get all comments, do:
+
+      comments?
+
+    Searching can be done like this:
+
+      search_by_name  Blastocystis hominis # This would search for Blastocystis
+                                           # hominis in the names table.
+      search_in_localomes Blastocystis hominis # As above, but we search
+                                               # in localomes rather than
+                                               # in NCBI. "slocalomes" is a 
+                                               # shorter alias to this.
+
+    To sort the localome table by modtime, we can do this:
+
+      modtime
+
+    To feedback the paths stored in the localome table, do this:
+
+      paths?
+
+    If you want to modify the path manually, you can use either set_path
+    or path (both are the same). We require the ID to update the path.
+
+    For example:
+
+      set_path TAXONOMY_ID_HERE new_path_goes_here
+      set_path 45157 /data/curated/sequences/localome/proteome/Cyanidioschyzon_merolae_pep.fa
+
+    Keep in mind that this can all be done through the commandline as well:
+
+      taxonomy set_path 45157 /data/curated/sequences/localome/proteome/Cyanidioschyzon_merolae_pep.fa
+
+    If we do not want to see any colours, we can disable them via:
+
+      nocolours # There may be some warnings about redefines taking place,
+                # but they are harmless. They happen because some methods
+                # get redefined.
+
+    If we don't want to see the help hint after every hit to the return-key,
+    do:
+
+      nohelp
+
+    If we wish to see the main entries in the nodes and names tables
+    from the NCBI Taxonomy, we can do this:
+
+      nodes?
+      names?
+ 
+    The above two commands will fetch 10 random entries from the
+    two tables, which should aid a little bit in ensuring that the
+    dataset is correct.
+
+    If we wish to find out which Eukaryota and which Prokaryota are
+    available in the localome table (the name of that table is "fasta"
+    at the time of writing this file here), we can do this:
+
+      eukarya
+      prokarya
+
+    Note that this, by default, won't give us the path, so in order
+    to find out the local path as well, we need to issue this command:
+
+      eukarya path
+      prokarya path # There were no entries from prokarya at the time 
+                    # of writing this file here though.
+
+    If we wish to change the path of a localome entry, we must know
+    the TaxID, then we can do something like this here:
+
+      path 1257118 /resources/seqdata/curated/sequences/localome/proteomes/Acanthamoeba_castellanii_Neff_pep.fa
+
+    The first argument is the mandatory TaxID, the second entry is the
+    new path that we wish to assign. Note that if you need to modify
+    several paths, you can also load up a file instead:
+
+      path name_of_file_here
+
+    This file should be in CSV format with two entries - to the left
+    the TaxID, to the right the new path, separated by | or \t (tabulator),
+    for example:
+
+      1257118 | /new/path/Acanthamoeba_castellanii_Neff_pep.fa
+      8128    | /new/path/Bla_ble.fa
+
+    To eliminate an entry from the localomes, do this:
+
+      remove 1257118
+
+    The argument to this must be the Taxonomic ID for now. Obviously
+    you should be sure that you want to remove this entry.
+
+    If you wish to load a dataset into the localome database, we
+    need the .INFO file. Then, do this:
+
+      load_from_info foo.INFO
+
+    The argument must be the path to the .INFO file in question.
+    Alternatively, simply provide the .INFO file here. If as 
+    part of the location we can find the string 'incoming/', we
+    will assume that you want to load it into the table.
+
+    To show the lineage directly from the localome table, we can do this:
+
+       show_lineage_from_localome_table 2762
+
+    * Note that the TaxID must exist before we can query it (remember,
+    you can do: "taxid,names" to quickly fetch the table data).
+
+    If you wish to find out the amount of .INFO files available, do
+    this here:
+
+      n_info?
+
+    This will look at the directory /data/curated/sequences/INFO/ and
+    the directory /resources/seqdata/curated/sequences/localome/incoming.
+
+    You can parse an info file by doing:
+
+      info foo.INFO
+
+    or by simply putting the full location of the .INFO file 
+    in the interactive shell.
+
+    Note that the code that handles parsing of .INFO files will also
+    attempt to locate a corresponding .fa entry. It will first check
+    for the local directory before checking the directory
+      /data/curated/sequences/aa/ or /data/curated/sequences/nt/
+
+    To parse a fasta file, we can do this:
+ 
+      pfasta foo.fa
+
+    or by simply putting the full location of the .fa file in the interactive
+    shell. The argument to this should be the fasta file in question of course.
+ 
+    If you quickly wish to obtain the taxtree in a tree display-like
+    fashion, try these two ways:
+ 
+      taxtree  77166
+      taxtree 106583
+
+    If we wish to update the lineage IDs and lineage scientific names in
+    the localome table, we can do this by issuing the following command:
+
+      update lineage
+
+    This will only update these two entries.
+
+    If you need to show the password, do:
+
+      password?
+
+    If you need to modify the password, try:
+
+      edit password
+
+    This will open vim at the file position.
+
+    The password is also defined in the postgresql_login_command.rb file,
+    which you can simply open via: "edit login"
+
+    If you need to display the port we will use, do any of these:
+
+      show_port
+      port?
+      --port
+
+== Commandline Usage (no interactive shell)
+
+In principle, all that can be done in the interactive shell, can also
+be done from the commandline alone. The main "binary" (it is just a
+text file of course, but resides in the bin/ subdirectory) to 
+invoke is the file 'taxonomy':
+
+    bin/taxonomy
+    taxonomy
+
+Here are some example for how to use it in cron scripts, with arguments -
+of course all of the above commands should also work as well, but we
+assume that in the interactive mode we need to feedback something
+to the user, whereas in a cron-like task we don't need to give any output
+to the user at all:
+
+  taxonomy update ncbi    # Download the remote NCBI database and generate
+                          # a new nodes and names table.
+
+  taxonomy update lineage # Just as the above, we will update the lineage
+                          # in the localome table (the fasta table).
+
+== Aliases
+
+In general, for most available actions in the main menu of the Taxonomy 
+module, aliases exist. So the subsection "query", for instance, can be
+shortened to "qu". In case of ambiguity, we try to decide which component
+is more important from the point of view of the Taxonomy module, so there
+should always be a clear "winner" in these cases.
+
+== Readline support
+
+If available, the Taxonomy module will make use of GNU Readline, which
+basically means that the <TAB> key can be used for autocompletion and
+that the cursor arrow keys (up and down) will work as well for the
+input history, just as it works in shells like bash or zsh.
+
+== Dependencies
+
+The Taxonomy module depends on a few ruby gems (addons), which can
+be queried in the interactive taxonomy shell via:
+
+  dependencies?
+
+== Packaging, Licensing and Redistribution
+
+If you need to package the taxonomy module for any reason, you can do
+so via the interactive menu by issuing:
+
+  gem
+
+or
+
+  make_gem
+
+This will try to build a .gem file and tell you the full location
+to this file as well. This .gem file can be used and redistributed.
+(A .gem file is Ruby's way to distribute addons, similar to perl's
+CPAN.)
+
+Permission is explicitly granted to allow the I.M.P. team to
+modify all code parts that were produced specifically for the 
+Taxonomy module in any way they see fit - in other words, the full
+content of this .gem archive is licensed to the I.M.P, in particular
+to Alex Schleiffer. This should allow modifications to everything 
+that belongs to the Taxonomy module at any moment of time.
+
+The .gem file would allow installation by doing something like this:
+
+  gem install taxonomy.gem # global installation
+
+or into the user home directory via:
+
+  gem install --user-install taxonomy.gem
+
+== Shebang line
+
+At the ray server, the Shebang line to ruby appears to be this line:
+
+#!/biosw/debian5-x86_64/ruby/1.9.3/bin/ruby -w
+
+On another server this may have to be adjusted, or one may have to call
+the "env" binary to find the proper location for ruby. But this of course
+works the same as if one would have Shebang point to perl - it all comes
+down to the real location of the binary in question. (On localomics,
+another ruby version was installed by default by Petair, so another
+Shebang would have to be used - also, /biosw seems to not exist on
+localomics; I assume this is a completely different machine with virtual
+and different slices assigned to particular tasks).
+
+== TEMP_DIR: the temporary directory
+
+Downloads will happen to the temporary directory, stored in the constant
+TEMP_DIR, which defaults to /tmp/robert at the time of writing this file.
+
+This is defined in the file shared/shared.rb. If that directory does not
+exist, we will attempt to create it. This is important because we will
+download data and generate files into that directory, so it must exist
+for the module to work.

data/doc/legacy_paths.md

@@ -0,0 +1,9 @@
+This small file here just is a reminder of the paths that were in
+use during February 2014 for the Taxonomy project when it was still
+a standalone project:
+
+  /biosw/debian5_x86_64/
+  /biosw/debian5_x86_64/postgresql/8.4.16/bin
+
+One day this file here may be removed, but for the time being it will
+remain as it is.