bio-pipengine 0.9.2 → 0.9.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +26 -55
- data/VERSION +1 -1
- data/lib/bio/pipengine.rb +1 -1
- data/lib/bio/pipengine/job.rb +7 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 82814e5bfa7e4fb2f89ac53262722ac769ce0ad7
|
|
4
|
+
data.tar.gz: e595dc961368cde6009e039bfe40ad9607960450
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 802301520a0405d748429810cfa8e398dfd92b9cec937d9fa37bcb217d7c2c507be61e6f2d25de2fa04dc0f0fe1ce863303748d09194772a62378a3c8ea2d2e4
|
|
7
|
+
data.tar.gz: 8bf3b5a453843891c16b1d501209320868967203d23601dda97b59e22b07f96f1ff8d968318bdb6da339d0632c41260895c1cc69c38d3687a16ffb72a914b7a9
|
data/README.md
CHANGED
|
@@ -3,9 +3,9 @@ PipEngine
|
|
|
3
3
|
|
|
4
4
|
A simple launcher for complex biological pipelines.
|
|
5
5
|
|
|
6
|
-
PipEngine will generate runnable shell scripts, already configured for the PBS/Torque job scheduler, for each sample in the pipeline. It allows to run a complete pipeline or just a single step
|
|
6
|
+
PipEngine will generate runnable shell scripts, already configured for the PBS/Torque job scheduler, for each sample in the pipeline. It allows to run a complete pipeline or just a single step depending on the needs.
|
|
7
7
|
|
|
8
|
-
PipEngine is best suited for NGS pipelines, but it can be used for any kind of pipeline that can be runned on a job scheduling system.
|
|
8
|
+
PipEngine is best suited for NGS pipelines, but it can be used for any kind of pipeline that can be runned on a job scheduling system and which is "sample" centric, i.e. you have from one side a list of samples with their corresponding raw data, and from the other side a pipeline that you would like to apply to them.
|
|
9
9
|
|
|
10
10
|
Installation
|
|
11
11
|
============
|
|
@@ -16,10 +16,12 @@ If you already have Ruby, just install PipEngine using RubyGems:
|
|
|
16
16
|
gem install bio-pipengine
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
If you don't have Ruby installed
|
|
19
|
+
If you don't have Ruby installed we reccomend you use the Anaconda Package Manager.
|
|
20
|
+
|
|
21
|
+
Download the installer from [here](http://conda.pydata.org/miniconda.html) and once installed you can simply type:
|
|
20
22
|
|
|
21
23
|
```shell
|
|
22
|
-
|
|
24
|
+
conda install -c bioconda ruby
|
|
23
25
|
```
|
|
24
26
|
|
|
25
27
|
and then install PipEngine using RubyGems:
|
|
@@ -56,28 +58,8 @@ PipEngine it's divided into two main sections:
|
|
|
56
58
|
> pipengine -h
|
|
57
59
|
List of available commands:
|
|
58
60
|
run Submit pipelines to the job scheduler
|
|
59
|
-
jobs Show statistics and interact with running jobs
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
Since PipEngine uses the [TORQUE-RM](https://github.com/helios/torque_rm) gem to interact with the job scheduler, at the first run PipEngine will ask few questions to prepare the required configuration file (e.g. provide IP address and username to connect via SSH to the PBS Server or Masternode).
|
|
63
|
-
|
|
64
|
-
Command line for JOBS mode
|
|
65
|
-
--------------------------
|
|
66
|
-
With this mode, PipEngine will interact with the job scheduler (Torque/PBS for now) and will allow performing searches on submitted jobs and send delete commands to remove jobs from the scheduler.
|
|
67
|
-
|
|
68
|
-
```shell
|
|
69
|
-
> pipengine jobs [options]
|
|
70
61
|
```
|
|
71
62
|
|
|
72
|
-
|
|
73
|
-
**Parameters**
|
|
74
|
-
```shell
|
|
75
|
-
--job-id, -i <s+>: Search submitted jobs by Job ID
|
|
76
|
-
--job-name, -n <s+>: Search submitted jobs by Job Name
|
|
77
|
-
--delete, -d <s+>: Delete submitted jobs ('all' to erase everything or type one or more job IDs)
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
|
|
81
63
|
Command line for RUN mode
|
|
82
64
|
-------------------------
|
|
83
65
|
|
|
@@ -90,36 +72,25 @@ With this mode, PipEngine will submit pipeline jobs to the scheduler.
|
|
|
90
72
|
|
|
91
73
|
**Parameters**
|
|
92
74
|
```shell
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
directory names)
|
|
113
|
-
--pbs-opts, -b <s+>: PBS options
|
|
114
|
-
--pbs-queue, -q <s>: PBS queue
|
|
115
|
-
--inspect-pipeline, -i <s>: Show steps
|
|
116
|
-
--mail-exit, -a <s>: Send an Email when the job terminates
|
|
117
|
-
--mail-start, -r <s>: Send an Email when the job starts
|
|
118
|
-
--log <s>: Log script activities, by default stdin.
|
|
119
|
-
Options are fluentd (default: stdin)
|
|
120
|
-
--log-adapter, -e <s>: (stdin|syslog|fluentd) In case of fluentd use
|
|
121
|
-
http://destination.hostname:port/yourtag
|
|
122
|
-
--help, -h: Show this message
|
|
75
|
+
-p, --pipeline=<s> YAML file with pipeline and sample details (default: pipeline.yml)
|
|
76
|
+
-f, --samples-file=<s> YAML file with samples name and directory paths (default: samples.yml)
|
|
77
|
+
-l, --samples=<s+> List of sample names to run the pipeline
|
|
78
|
+
-s, --steps=<s+> List of steps to be executed
|
|
79
|
+
-d, --dry Dry run. Just create the job script without submitting it to the batch system
|
|
80
|
+
-t, --tmp=<s> Temporary output folder
|
|
81
|
+
-c, --create-samples=<s+> Create samples.yml file from a Sample directory (only for CASAVA projects)
|
|
82
|
+
-m, --multi=<s+> List of samples to be processed by a given step (the order matters)
|
|
83
|
+
-g, --group=<s> Specify the group of samples to run the pipeline steps on (do not specify --multi)
|
|
84
|
+
-a, --allgroups Apply the step(s) to all the groups defined into the samples file
|
|
85
|
+
-n, --name=<s> Analysis name
|
|
86
|
+
-o, --output-dir=<s> Output directory (override standard output directory names)
|
|
87
|
+
-b, --pbs-opts=<s+> PBS options
|
|
88
|
+
-q, --pbs-queue=<s> PBS queue
|
|
89
|
+
-i, --inspect-pipeline=<s> Show steps
|
|
90
|
+
--log=<s> Log script activities, by default stdin. Options are fluentd (default: stdin)
|
|
91
|
+
-e, --log-adapter=<s> (stdin|syslog|fluentd) In case of fluentd use http://destination.hostname:port/yourtag
|
|
92
|
+
--tag=<s+> Overwrite tags present in samples.yml and pipeline.yml files (e.g. tag1=value1 tag2=value2)
|
|
93
|
+
-h, --help Show this message
|
|
123
94
|
```
|
|
124
95
|
|
|
125
96
|
PipEngine accepts two input files:
|
|
@@ -683,4 +654,4 @@ If a specific queue needs to be selected for sending the jobs to PBS, the ```--p
|
|
|
683
654
|
Copyright
|
|
684
655
|
=========
|
|
685
656
|
|
|
686
|
-
|
|
657
|
+
©2016 Francesco Strozzi, Raoul Jean Pierre Bonnal
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.9.
|
|
1
|
+
0.9.3
|
data/lib/bio/pipengine.rb
CHANGED
|
@@ -49,7 +49,7 @@ module Bio
|
|
|
49
49
|
########### START ###########
|
|
50
50
|
|
|
51
51
|
# create output directory (jobs scripts will be saved there)
|
|
52
|
-
FileUtils.mkdir_p samples_file["resources"]["output"] unless options[:dry] #&& options[:spooler]!="pbs"
|
|
52
|
+
FileUtils.mkdir_p samples_file["resources"]["output"] #unless options[:dry] #&& options[:spooler]!="pbs"
|
|
53
53
|
|
|
54
54
|
# check if the requested steps are multi-samples
|
|
55
55
|
run_multi = check_and_run_multi(samples_file,pipeline,samples_list,options)
|
data/lib/bio/pipengine/job.rb
CHANGED
|
@@ -120,7 +120,7 @@ module Bio
|
|
|
120
120
|
file.puts "#!/usr/bin/env bash"
|
|
121
121
|
file.puts "#PBS -N #{self.name}"
|
|
122
122
|
file.puts "#PBS -d #{self.output}"
|
|
123
|
-
file.puts "#PBS -q #{options[:pbs_queue]}"
|
|
123
|
+
file.puts "#PBS -q #{options[:pbs_queue]}" if options[:pbs_queue]
|
|
124
124
|
if options[:pbs_opts]
|
|
125
125
|
file.puts "#PBS -l #{options[:pbs_opts].join(",")}"
|
|
126
126
|
else
|
|
@@ -153,6 +153,12 @@ module Bio
|
|
|
153
153
|
else
|
|
154
154
|
cmd = sub_placeholders(cmd,sample,step) # normal step, perform usual substitutions
|
|
155
155
|
end
|
|
156
|
+
|
|
157
|
+
# Check that all placeholders have been substituted, if not terminate with an error
|
|
158
|
+
cmd.scan(/<\S+>/).each do |unsubstituted_tag|
|
|
159
|
+
@@logger_error.error("Found an unsubstituted tag #{unsubstituted_tag} . Terminating the execution".red)
|
|
160
|
+
exit
|
|
161
|
+
end
|
|
156
162
|
return cmd
|
|
157
163
|
end
|
|
158
164
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: bio-pipengine
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.9.
|
|
4
|
+
version: 0.9.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Francesco Strozzi
|
|
@@ -9,7 +9,7 @@ authors:
|
|
|
9
9
|
autorequire:
|
|
10
10
|
bindir: bin
|
|
11
11
|
cert_chain: []
|
|
12
|
-
date: 2016-
|
|
12
|
+
date: 2016-05-09 00:00:00.000000000 Z
|
|
13
13
|
dependencies:
|
|
14
14
|
- !ruby/object:Gem::Dependency
|
|
15
15
|
name: trollop
|