oswitch 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f17f0c2bd70cf5a969f60d6a000ebacaba1940dc
+  data.tar.gz: 4fc93cdb2580315e7a49032c2ff32797566144a1
+SHA512:
+  metadata.gz: 08362f1f00b5d02531c2df28c59ccb9107f13e9ffcce5efef32cf2378e922f83c8db512dd098af6c081237b6db7920bf64c1d258c0f916762de5bd5a3eb0e883
+  data.tar.gz: 73b363b026815f51c84b110dd59a857a168fc9c8e9aaee6b22b14cb96341b4d83c7eba06b02a92d9fd18ef2cbb4d4f2fd91307cf1ba52f27aa4b6909da0f8e3e

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/README.mkd

@@ -0,0 +1,295 @@
+# OSwitch
+
+One-line access to other operating systems.
+
+* [Usage](#example-usage)
+* [Installation](#installation)
+* [FAQ](#faq)
+* [Roadmap](#roadmap)
+* [Contributors & Funding](#contributors-&-funding)
+
+## Background
+
+Genomic analyses require using many bioinformatics tools starting from assessing
+the quality of sequenced data and assembly, to annotation, comparison and
+analysis. The data types are young, thus the tools are too and many
+genomicists lack computational training. Thus tools are frequently
+updated yet often challenging to install. Furthermore, software updates often
+involve changes in algorithms or input/output formats, making analyses
+difficult to reproduce. To make matters worse, genomicists often lack
+the skills necessary to setup complex bioinformatics software, and
+systems administrators can be overwhelmed by large numbers of software
+installation requests & the challenges of managing multiple versions.
+
+## Aim & Features
+
+We are developing `oswitch` to enable **seamless switching
+from one operating system to another** - providing access to diverse
+ranges of tools. This project grew from our own need to rapidly access
+software distributed as part of
+[BioLinux](http://environmentalomics.org/bio-linux/) on our MacBooks
+and our university HPC system.
+
+For this we take advantage of the [docker](http://docker.io/)
+technology. Docker works by creating "to the specification" image from
+a Dockerfile, which is then run in an isolated
+"container". Dockerfiles or the resulting images can persist forever,
+are easily [shared or published](https://hub.docker.com/), and make
+it possible for anybody to recreate the exact same setup at
+any point of time in the future. This is similar to using virtual
+machine images - but much more
+[flexible and light-weight](http://stackoverflow.com/questions/16047306/how-is-docker-io-different-from-a-normal-virtual-machine).
+
+`oswitch` is thus a wrapper facilitating access to docker images (without the
+need for ssh-ing). Importantly, when switching operating systems inside a
+shell, most things remain unchanged:
+
+* Current working directory is maintained
+* User name, uid and gid are maintained
+* Login shell (bash/zsh/fish) is maintained
+* Home directory is maintained (thus all .dotfiles and config files
+  are maintained).
+* read/write permissions are maintained
+* Paths are maintained whenever possible. Thus volumes (external drives,
+NAS) mounted on the host are available in the container at the same
+path.
+
+## Example Usage
+
+There are two broad usage scenarios: interactive use & non-interactive
+use.
+
+##### Use a package interactively in a normal command-line
+
+Minimalist example:
+
+```shell
+Yannick@n56-169 ~/g/oswitch> uname -a
+Darwin n56-169.sbcs.qmul.ac.uk 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64
+Yannick@n56-169 ~/g/oswitch> oswitch yeban/biolinux:8
+### You are now running: biolinux_8, in container: biolinux_8-27182. ###
+Yannick@biolinux_8-27182 ~/g/oswitch> uname -a
+Linux biolinux_8-27182 3.16.4-tinycore64 #1 SMP Thu Oct 23 16:14:24 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
+```
+
+Biologically relevant example:
+
+```shell
+# Trying to run blast.
+pixel:~/test/ $ ls 
+mygene.fasta
+pixel:~/test/ $ cat mygene.fa
+>myfavoritegene isthisone
+MNTLWLSLWDYPGKLPLNFMVFDTKDDLQAAYWRDPYSIPLAVIFEDPQPISQRLIYEIR
+TNPSYTLPPPPTKLYSAPISCRKNKTGHWMDDILSIKTGESCPVNNYLHSGFLALQMITD
+ITKIKLENSDVTIPDIKLIMFPKEPYTADWMLAFRVVIPLYMVLALSQFITYLLILIVGE
+KENKIKEGMKMMGLNDSVF
+pixel:~/test/ $ blastp -query mygene.fa -remote -db nr -outfmt 7 > mygene_blastp_nr.tab
+zsh: command not found: blastp
+# Indeed... blastp is not installed on my MacBook. 
+
+# Switch to BioLinux and run blastp.
+pixel:~/test/ $ oswitch biolinux
+###### You are now running: biolinux in container biolinux-7187. ######
+biolinux-7187:~/test/ $ blastp -query mygene.fa -remote -db nr -outfmt 7 >  mygene_blastp_nr.tab
+# BioLinux includes blastp, thus the command ran smoothly.
+
+# View the result.
+biolinux-7187:~/test/ $ head mygene_blastp_nr.tab
+# BLASTP 2.2.28+
+# Query: myfavoritegene isthisone
+# RID: BJAHAHU9015
+# Database: nr
+# Fields: query id, subject id, % identity, alignment length, mismatches, gap opens, q. start, q. end, s. start, s. end, evalue, bit score
+# 501 hits found
+myfavoritegene	gi|322796550|gb|EFZ19024.1|	100.00	199	0	0	1	199	1	199	2e-142	 407
+myfavoritegene	gi|307183032|gb|EFN69988.1|	86.07	201	25	2	1	199	80	279	6e-115	 361
+myfavoritegene	gi|572260155|ref|XP_006608402.1|	80.60	201	36	2	1	199	95	294	4e-108	 350
+myfavoritegene	gi|328778864|ref|XP_397465.4|	80.60	201	36	2	1	199	95	294	5e-108	 350
+
+
+# [... potentially run other analyses that require biolinux things...]
+
+# Return to normal operating system
+biolinux-7187:~/test/ $ exit
+pixel:~/test/ $ ls 
+mygene.fasta mygene_blastp_nr.txt
+```
+
+##### Use a package non-interactively
+
+Alternatively, single commands can be run directly in a container
+(e.g. BioLinux) without entering it interactively. This can
+be useful to test new tools, or to run a single piece of
+not-locally-installed software as part of a single command. The
+container terminates automatically once the command has been
+executed, output is printed to the terminal and can be redirected, and
+the exit status of the command run within container is returned.
+
+```shell
+# Run command directly in BioLinux and view results if success.
+pixel:~/test/ $ oswitch biolinux blastp -remote -query mygene.fa -db nr > mygene_blastp_nr.txt
+```
+
+##### Listing available operating system containers
+
+OSwitch can pull any image from docker hub. You can see the images you pulled
+from docker hub using oswitch as:
+
+```shell
+pixel:~ $ oswitch -l
+yeban/biolinux:8
+ubuntu:14.04
+```
+	
+##### Availability
+
+OSwitch has been tested on:
+
+* Mac OS X Yosemite
+* Ubuntu 14.04.1
+* CentOS 7
+
+##### Caveats
+
+* Works only for Debian, Ubuntu, CentOS based docker images.
+* Host directories/volumes with paths conflicting with container paths are
+  skipped.
+* SELinux must be disabled on CentOS for mounting volumes to work.
+
+## Installation
+
+OSwitch first requires a working docker install.
+
+#### Install and setup docker
+
+##### Mac OS X
+
+Installing docker - https://docs.docker.com/installation/mac/
+
+##### Ubuntu
+
+Installing docker - https://docs.docker.com/installation/ubuntulinux/
+
+Add yourself to docker group so you can run docker client without sudo:
+
+```shell
+    $ sudo usermod -aG docker `whoami`
+    
+    # then logout and login again for the above command to take effect
+```
+	
+##### CentOS
+
+Installing docker - https://docs.docker.com/installation/centos/
+
+Add yourself to docker group so you can run docker client without sudo:
+
+```shell
+    $ sudo usermod -aG docker `whoami`
+    
+    # then logout and login again for the above command to take effect
+```
+	
+Disable SELinux as it gets in the way of mounting volumes within the container:
+
+```shell
+    $ sed -i .bak 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
+
+    # then reboot your system
+```
+	
+The above command backs up the original file to `/etc/selinux/config.bak`. If
+you are concerned about disabling SELinux, do note that we are trying to work
+out a better solution.
+
+#### Test that docker is correctly installed
+
+The following should give an encouraging message: 
+
+    $ docker run hello-world
+
+#### Install oswitch
+
+Requirements: Ruby 2.0 or higher.
+
+    $ gem install oswitch
+
+#### Testing oswitch
+
+    $ oswitch ubuntu:14.04
+
+## FAQ
+
+##### Q. Directories mounted within container on Mac host are empty.
+The problem is, on Mac `boot2docker` is the _real_ host, not OS X. `oswitch`
+can mount only what's available to it from `boot2docker`. For example,
+`/Applications`.
+
+Run `boot2docker ssh ls /Applications` and you will find it empty as well.
+
+The workaround is to correctly mount the directories you want in `boot2docker`
+first.
+
+```
+boot2docker down
+VBoxManage sharedfolder remove boot2docker-vm --name Applications
+VBoxManage sharedfolder add boot2docker-vm --name Applications --hostpath /Applications
+boot2docker up
+boot2docker ssh "sudo mkdir -p /Applications && sudo mount -t vboxsf -o uid=1000,gid=50 Applications /Applications"
+```
+
+##### Q. cwd is empty in the container
+This means the said directory was not mounted by oswitch, or was incorrectly
+mounted. On Linux host, directories that can conflict with paths within
+container are not mounted. On Mac, `boot2docker` can get in the way.
+
+Please [report](https://github.com/yeban/oswitch/issues/new) this on our [issue
+tracker](https://github.com/yeban/oswitch/issues). To help us debug, please
+include:
+
+1. the directory in question
+2. the operating system you are running
+
+##### Q. oswitch does not work with my docker image
+Please [report](https://github.com/yeban/oswitch/issues/new) this on our [issue
+tracker](https://github.com/yeban/oswitch/issues) with oswitch's output. If the
+image you are using is not available via docker hub or another public
+repository, please include the Dockerfile as well.
+
+## Roadmap
+
+1. ~~make it possible to use docker containers without inheriting our
+current baseimage~~
+2. ~~gem distribution for easier installation~~
+3.  brew recipe for Mac
+4. test on QMUL's compute cluster
+5.  create an SELinux policy to run oswitch on CentOS without having to disable
+SELinux entirely
+6. rpm and deb packages
+7. make available images for common bioinformatics software
+8. deploy at [RAL/JASMIN](http://www.jasmin.ac.uk)
+
+## Contribute
+
+    $ git clone https://github.com/yeban/oswitch
+    $ cd oswitch
+    $ gem install bundler && bundle
+    $ bundle exec bin/oswitch biolinux
+
+## Contributors & Funding
+
+* Anurag Priyam - [a.priyam@qmul.ac.uk](mailto:a.priyam@qmul.ac.uk) | [@yeban](//twitter.com/yeban)
+* [Bruno Vieira](https://github.com/bmpvieira) ([@bmpvieira](//twitter.com/bmpvieira))
+* [Saurabh Kumar](https://github.com/sa1)
+* Richard Nichols - [http://www.sbcs.qmul.ac.uk/staff/richardnichols.html](http://www.sbcs.qmul.ac.uk/staff/richardnichols.html) | [@qmwugbt112](//twitter.com/qmwugbt112)
+* Yannick Wurm  - [http://wurmlab.github.io](http://wurmlab.github.io) |
+  [@yannick__](//twitter.com/yannick__)
+
+---
+
+<p align="center">
+  Development funded as part of NERC EOS Cloud at
+  <a href="http://wurmlab.github.io/">Wurm Lab</a>,
+  <a href="http://sbcs.mul.ac.uk/">Queen Mary University of London</a>.
+</p>

data/bin/oswitch

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+
+require 'oswitch'
+require 'optparse'
+
+options = {}
+collect = lambda do |key|
+  lambda do |value|
+    options[key] = value
+  end
+end
+
+optspec = OptionParser.new do |opts|
+  opts.banner = "Usage: oswitch [options] <package> [cmd]"
+
+  opts.on '-l', '--list',
+          'List packages.',
+          &collect[:list]
+
+  opts.on '-h', '--help',
+          'Print help.',
+          &collect[:help]
+end
+
+begin
+  optspec.order!
+rescue OptionParser::InvalidOption,
+       OptionParser::MissingArgument => e
+  puts e
+  exit
+end
+
+if options[:help] || (options.empty? && ARGV.empty?)
+  puts optspec
+  exit
+end
+
+if options[:list]
+  puts OSwitch.packages
+  exit
+end
+
+package = ARGV[0]
+command = ARGV[1..-1]
+OSwitch.to(package, command)

data/lib/oswitch.rb

@@ -0,0 +1,293 @@
+require 'timeout'
+require 'colorize'
+require 'fileutils'
+require 'shellwords'
+
+# OSwitch leverages docker to provide access to complex Bioinformatics software
+# (even Biolinux!) in just one command.
+#
+# Images are built on the user's system on demand and executed in a container.
+# Containers are removed after execution.
+#
+# Volumes from host OS are mounted in the container just the same, including
+# home directory. USER, HOME, SHELL, and PWD are preserved.
+class OSwitch
+
+  class ENOPKG < StandardError
+
+    def initialize(name)
+      @name = name
+    end
+
+    def to_s
+      "Recipe to run #@name not available."
+    end
+  end
+
+  class ENODKR < StandardError
+
+    def to_s
+      "***** Docker not installed / correctly setup / running.
+      Are you able to run 'docker info'?"
+    end
+  end
+
+  include Timeout
+
+  # Captures a docker image's metadata.
+  Image = Struct.new :repository, :tag, :id, :created, :size
+
+  class Image
+
+    # Model Image's eigenclass as a collection of Image objects.
+    class << self
+
+      include Enumerable
+
+      def all
+        `docker images`.split("\n").drop(1).
+          map{|l| Image.new(*l.split(/\s{2,}/))}
+      end
+
+      def each(&block)
+        all.each(&block)
+      end
+
+      def get(imgname)
+        repository, tag = imgname.split(':')
+        return if not repository or repository.empty?
+        tag = 'latest' if not tag or tag.empty?
+        find {|img| img.repository == repository and img.tag == tag}
+      end
+
+      def exists?(imgname)
+        !!get(imgname)
+      end
+    end
+  end
+
+  # Linux specific code.
+  module Linux
+
+    def uid
+      Process.uid
+    end
+
+    def gid
+      Process.gid
+    end
+
+    # Parse /proc/mounts for mountpoints.
+    def mountpoints
+      mtab = IO.readlines '/proc/mounts'
+      mountpoints = mtab.map{ |line| line.split(/\s+/)[1]}
+      mountpoints.map!{ |mount| unescape(mount) }
+      # Ignore common system mountpoints.
+      mountpoints.reject!{ |mount| mount =~ /^\/$/ }
+      mountpoints.reject!{ |mount| mount =~ /^\/(proc|sys|usr|boot|tmp|dev|var|bin|etc|lib).*/ }
+      # Mount /run/media/* but ignore other /run/ mountpoints.
+      mountpoints.reject!{ |mount| mount =~ /^\/run.*/ unless mount =~ /^\/run\/(media.*)/ }
+
+      # Add home dir.
+      mountpoints << home
+    end
+
+    private
+
+    def unescape(mount)
+      mount.gsub(/\\040/, " ").gsub(/\\012/, "\n").gsub(/\\134/, "\\").gsub(/\\011/, "\t")
+    end
+  end
+
+  # Mac OS X specific code.
+  module Darwin
+
+    BLACKLIST =
+      %r{
+      ^/$|
+      ^/(bin|cores|dev|etc|home|Incompatible\ Software|
+         installer\.failurerequests|lost\+found|net|
+         Network|opt|private|sbin|System|Users|tmp|
+         usr|var|Volumes$)
+      }x
+
+    def uid
+      `boot2docker ssh id -u`.chomp
+    end
+
+    def gid
+      `boot2docker ssh id -g`.chomp
+    end
+
+    def mountpoints
+      volumes = Dir['/Volumes/*'].map {|v| File.symlink?(v) ? File.readlink(v) : v}
+      volumes = volumes | Dir['/*']
+      volumes.reject! { |mount| mount =~ BLACKLIST }
+      volumes << home
+    end
+  end
+
+  # NOTE:
+  #   This won't work on JRuby, as it sets RUBY_PLATFORM to 'java'.
+  case RUBY_PLATFORM
+  when /linux/
+    include Linux
+  when /darwin/
+    include Darwin
+  end
+
+  DOTDIR = File.expand_path('~/.oswitch')
+
+  class << self
+    # Invoke as `OSwitch.to` instead of `OSwitch.new`.
+    alias_method :to, :new
+    private :new
+
+    def packages
+      Dir["#{DOTDIR}/*"].
+        select {|entry| File.directory? entry}.
+        map {|pkg|
+          pkg.gsub("#{DOTDIR}/", '')
+        }
+    end
+  end
+
+  def initialize(package, command = [])
+    @package = package.strip
+    @command = command.join(' ')
+    @imgname = "oswitch_#{@package}"
+    @cntname = "#{@package.gsub(%r{/|:}, '_')}-#{Process.pid}"
+    exec
+  end
+
+  attr_reader :package, :command, :imgname, :cntname
+
+  def exec
+    ping and build and switch
+  rescue ENODKR, ENOPKG => e
+    puts e
+    exit
+  end
+
+  private
+
+  def switch
+    cmdline = "docker run --name #{cntname} --hostname #{cntname} -it --rm" \
+      " -w #{cwd} #{mountargs} #{imgname} "
+    if command.empty?
+      # Display motd and run interactive shell.
+      cmdline << "#{shell} -c \"echo #{motd}; #{shell} -i\""
+    else
+      cmdline << "#{shell} -c \"#{command}\""
+    end
+    Kernel.exec cmdline
+  end
+
+  def build
+    return true if Image.exists? imgname
+    write_context && system("docker build -t #{imgname} #{context_dir}")
+  end
+
+  # Ping docker daemon. Raise error if no response within 10s.
+  def ping
+    pong = timeout 5, ENODKR do
+      system 'docker info > /dev/null 2>&1'
+    end
+    pong or raise ENODKR
+  end
+
+  ## Code to generate context dir that will be built into a docker image. ##
+
+  # Write data to context dir.
+  def write_context
+    create_context_dir
+    write_dockerfile
+  end
+
+  # Create context dir.
+  def create_context_dir
+    FileUtils.mkdir_p context_dir
+    FileUtils.cp_r(template_files, context_dir)
+  end
+
+  # Write Dockerfile.
+  def write_dockerfile
+    dockerfile = File.join(context_dir, 'Dockerfile')
+    File.write(dockerfile, dockerfile_data)
+  end
+
+  # Generate String that get written to Dockerfile.
+  def dockerfile_data
+    data = ["FROM #{package}"]
+    data << 'COPY _switch /'
+    data << 'COPY wheel /etc/sudoers.d/'
+    data << "RUN /_switch #{userargs} 2>&1 | tee /tmp/oswitch.log"
+    data << 'ENV LC_ALL en_US.UTF-8'
+    data << "USER #{username}"
+    data << "ENTRYPOINT [\"#{shell}\", \"-c\"]"
+    data.join("\n")
+  end
+
+  # Location of context dir.
+  def context_dir
+    File.join(DOTDIR, package)
+  end
+
+  # Location of template dir.
+  def template_dir
+    File.expand_path('../context/', File.dirname(__FILE__))
+  end
+
+  # Template files.
+  def template_files
+    Dir[File.join(template_dir, '*')]
+  end
+
+
+  ## Data required to switchify a container. ##
+  def username
+    ENV['USER']
+  end
+
+  def home
+    ENV['HOME']
+  end
+
+  def shell
+    File.basename ENV['SHELL']
+  end
+
+  def cwd
+    Dir.pwd
+  end
+
+  def motd
+    str =<<MOTD
+################################################################################
+You are now running: #{package}, in container: #{cntname}.
+
+Container is distinct from the shell your launched this container from. Changes
+you make here will be lost unless it's made to one of the directories below:
+
+ - #{mountpoints.join("\n - ")}
+
+It's possible you may not be able to write to one or more directories above,
+but it should be possible to read data from all. Home directory is often the
+safest to write to.
+
+Press Ctrl-D or type 'exit' to go back.
+################################################################################
+MOTD
+    str.blue.shellescape
+  end
+
+  def mountargs
+    mountpoints.map do |mountpoint|
+      "-v '#{mountpoint}':'#{mountpoint}'"
+    end.join(' ')
+  end
+
+  def userargs
+    [uid, gid, username, home, shell].join(' ')
+  end
+end

data/oswitch.gemspec

@@ -0,0 +1,27 @@
+Gem::Specification.new do |s|
+  # meta
+  s.name        = 'oswitch'
+  s.version     = '0.1.0'
+  s.authors     = ['Anurag Priyam']
+  s.email       = ['anurag08priyam@gmail.com']
+  s.homepage    = 'https://github.com/yeban/oswitch'
+  s.license     = 'MIT'
+
+  s.summary     = "Use docker image as the host operating system's user and \
+access host operating system's filesystem."
+  s.description = <<DESC
+Use any docker image as the host operating system's user (same user name, same
+uid, same gid, and even the same shell!) and access to host operating system's
+filesystem.
+DESC
+
+  # dependencies
+  s.add_dependency('colorize',    '~> 0.7.5')
+
+  # gem
+  s.files         = Dir['lib/**/*'] + Dir['Dockerfiles/**/*']
+  s.files         = s.files + ['Gemfile', 'oswitch.gemspec']
+  s.files         = s.files + ['README.mkd']
+  s.require_paths = ['lib']
+  s.executables   = ['oswitch']
+end

metadata

@@ -0,0 +1,68 @@
+--- !ruby/object:Gem::Specification
+name: oswitch
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Anurag Priyam
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-02-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: colorize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.5
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.5
+description: |
+  Use any docker image as the host operating system's user (same user name, same
+  uid, same gid, and even the same shell!) and access to host operating system's
+  filesystem.
+email:
+- anurag08priyam@gmail.com
+executables:
+- oswitch
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- README.mkd
+- bin/oswitch
+- lib/oswitch.rb
+- oswitch.gemspec
+homepage: https://github.com/yeban/oswitch
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.2
+signing_key: 
+specification_version: 4
+summary: Use docker image as the host operating system's user and access host operating
+  system's filesystem.
+test_files: []