rest-ftp-daemon 0.41 → 0.55
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +8 -8
- data/.gitignore +1 -3
- data/Gemfile.lock +8 -2
- data/README.md +98 -33
- data/bin/rest-ftp-daemon +23 -15
- data/config.ru +2 -1
- data/lib/rest-ftp-daemon.rb +1 -4
- data/lib/rest-ftp-daemon/api/defaults.rb +2 -0
- data/lib/rest-ftp-daemon/api/index.haml +105 -0
- data/lib/rest-ftp-daemon/api/jobs.rb +1 -1
- data/lib/rest-ftp-daemon/api/root.rb +50 -3
- data/lib/rest-ftp-daemon/api/workers.rb +49 -0
- data/lib/rest-ftp-daemon/common.rb +1 -1
- data/lib/rest-ftp-daemon/config.rb +21 -12
- data/lib/rest-ftp-daemon/job.rb +43 -16
- data/lib/rest-ftp-daemon/logger.rb +1 -1
- data/lib/rest-ftp-daemon/notification.rb +2 -0
- data/rest-ftp-daemon.gemspec +5 -2
- data/rest-ftp-daemon.yml.sample +15 -0
- metadata +33 -3
- data/lib/rest-ftp-daemon/www.rb +0 -6
checksums.yaml
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
3
|
metadata.gz: !binary |-
|
4
|
-
|
4
|
+
YjFjZjg2NTk1Y2M5NzVmNGM4MWJmYjA1YTEwN2JhZjVjOTlkN2I2OQ==
|
5
5
|
data.tar.gz: !binary |-
|
6
|
-
|
6
|
+
OGViZWY5NzFjNjI2NDRlYzVkOTNhNjliY2UxMDg3MTRmMTFhMTVmMA==
|
7
7
|
SHA512:
|
8
8
|
metadata.gz: !binary |-
|
9
|
-
|
10
|
-
|
11
|
-
|
9
|
+
ODExM2M0MjhhOTE2OGRjNTIyNmFlZjU5MzkxZTY2YzVkMjRmMDljYzhmZDRl
|
10
|
+
M2ZmZWVjZDE5NjYzNTM3NzJiMGVkOGQ3YTJkZjJkYzE3Yzc0ODk5M2M3MmRm
|
11
|
+
MWIzNTk3ZDc5ODc5YWJjZTY3MGRhNzhlMTNiYTA5NTRiYzhhNDQ=
|
12
12
|
data.tar.gz: !binary |-
|
13
|
-
|
14
|
-
|
15
|
-
|
13
|
+
NWIxNTNlNzk5NDY3NGQ1ZjI4NzY1ZDlmMWI0NTRkYzU2NDRiMzg1YmZlODVi
|
14
|
+
NTlhNzVkNDk4Yjc5MzdiMjgwMTExNzA4MGQwNGZhZDdmZTA5ZGY4Njg2MTUz
|
15
|
+
OWMxNDA4NjQwMzUxZTYxNGFhM2FkYjRjY2U4NTFiZmQ0NGEzZGU=
|
data/.gitignore
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,15 +1,18 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
rest-ftp-daemon (0.
|
4
|
+
rest-ftp-daemon (0.55)
|
5
|
+
facter
|
5
6
|
grape
|
6
7
|
json
|
8
|
+
settingslogic
|
7
9
|
thin (~> 1.6)
|
8
10
|
|
9
11
|
GEM
|
10
12
|
remote: http://rubygems.org/
|
11
13
|
specs:
|
12
|
-
|
14
|
+
CFPropertyList (2.2.8)
|
15
|
+
activesupport (4.1.6)
|
13
16
|
i18n (~> 0.6, >= 0.6.9)
|
14
17
|
json (~> 1.7, >= 1.7.7)
|
15
18
|
minitest (~> 5.1)
|
@@ -27,6 +30,8 @@ GEM
|
|
27
30
|
thread_safe (~> 0.3, >= 0.3.1)
|
28
31
|
equalizer (0.0.9)
|
29
32
|
eventmachine (1.0.3)
|
33
|
+
facter (2.2.0)
|
34
|
+
CFPropertyList (~> 2.2.6)
|
30
35
|
grape (0.9.0)
|
31
36
|
activesupport
|
32
37
|
builder
|
@@ -50,6 +55,7 @@ GEM
|
|
50
55
|
rack-mount (0.8.3)
|
51
56
|
rack (>= 1.0.0)
|
52
57
|
rake (10.3.2)
|
58
|
+
settingslogic (2.0.9)
|
53
59
|
thin (1.6.2)
|
54
60
|
daemons (>= 1.0.9)
|
55
61
|
eventmachine (>= 1.0.0)
|
data/README.md
CHANGED
@@ -1,38 +1,36 @@
|
|
1
1
|
rest-ftp-daemon
|
2
2
|
====================================================================================
|
3
3
|
|
4
|
+
|
5
|
+
|
4
6
|
This is a pretty simple FTP client daemon, controlled through a RESTfull API.
|
5
7
|
|
6
8
|
As of today, its main features are :
|
7
9
|
|
10
|
+
* Allow environment-specific configuration in a YAML file
|
8
11
|
* Delegate a transfer job by ``POST```'ing a simple JSON structure
|
9
12
|
* Spawn a dedicated thread to handle this job in its own context
|
10
13
|
* Report transfer status, progress and errors for each job in realtime
|
11
14
|
* Expose JSON status of workers on ```GET /jobs/``` for automated monitoring
|
12
15
|
* Parralelize jobs as soon as they arrive
|
16
|
+
* Handle job queues and priority as an attribute of the job
|
17
|
+
* Allow dynamic evaluation of priorities, and change of any attribute until the job is picked
|
18
|
+
* Provide RESTful notifications to the requesting client
|
19
|
+
* Allow authentication in FTP target in a standard URI-format
|
20
|
+
* Allow configuration-based path templates to abstract local mounts or remote FTPs (endpoint tokens)
|
21
|
+
|
13
22
|
|
14
23
|
Expected features in a short-time range :
|
15
24
|
|
16
|
-
* Handle job queues
|
17
|
-
* Handle job priorities
|
18
25
|
* Allow change of priorities or other attributes after a job has been started
|
19
|
-
* Provide RESTful notifications to the requesting client
|
20
26
|
* Offer a basic dashboard directly within the daemon HTTP interface
|
21
27
|
* Periodically send an update-notification with transfer status and progress
|
22
28
|
* Allow fallback file source when first file path is unavailable (failover)
|
23
|
-
* Some refactoring may be needed after thos steps
|
24
29
|
* Provide swagger-style API documentation
|
25
30
|
* Authenticate API clients
|
26
31
|
|
27
32
|
|
28
33
|
|
29
|
-
Documentation TODO
|
30
|
-
------------------------------------------------------------------------------------
|
31
|
-
overwrite: any non empty value allows overwriting
|
32
|
-
todo: queues
|
33
|
-
|
34
|
-
|
35
|
-
|
36
34
|
Installation
|
37
35
|
------------------------------------------------------------------------------------
|
38
36
|
|
@@ -51,51 +49,117 @@ Start the daemon:
|
|
51
49
|
rest-ftp-daemon start
|
52
50
|
```
|
53
51
|
|
52
|
+
Start the daemon on a specific port :
|
53
|
+
|
54
|
+
```
|
55
|
+
rest-ftp-daemon -p4000 start
|
56
|
+
```
|
57
|
+
|
54
58
|
Check that the daemon is running and giving status info
|
55
59
|
|
56
60
|
```
|
57
61
|
http://localhost:3000/
|
58
62
|
```
|
59
63
|
|
60
|
-
|
64
|
+
Configuration
|
65
|
+
------------------------------------------------------------------------------------
|
66
|
+
Most of the configuration options live in a YAML configuration file, containing two main sections:
|
61
67
|
|
68
|
+
* the ``defaults`` section should be left as-is and will be used is no other environment-specific value is provided.
|
69
|
+
* the ``production`` section can receive personnalized settings according to your environment-specific setup and paths.
|
62
70
|
|
63
|
-
|
71
|
+
Configuration priority is defined as follows (from most important to last resort):
|
72
|
+
|
73
|
+
* command-line parameters
|
74
|
+
* config file defaults section
|
75
|
+
* config file environment section
|
76
|
+
* application internal defaults
|
77
|
+
|
78
|
+
|
79
|
+
As a starting point, ``rest-ftp-daemon.yml.sample`` is an exemple config file that can be copied into the expected location ``/etc/rest-ftp-daemon.yml``.
|
80
|
+
|
81
|
+
|
82
|
+
Logging
|
64
83
|
------------------------------------------------------------------------------------
|
65
84
|
|
66
|
-
|
85
|
+
The application will not log to any file by default, if not specified in its configuration.
|
86
|
+
Otherwise separate logging paths can be provided for the Thin webserver, API related messages, and workers related messages. Providing and empty value will simply activate logging to ``STDOUT``.
|
87
|
+
|
88
|
+
|
89
|
+
Usage examples
|
90
|
+
------------------------------------------------------------------------------------
|
67
91
|
|
68
|
-
Start a job to transfer a file named "file.iso" to a local FTP server
|
92
|
+
* Start a job to transfer a file named "file.iso" to a local FTP server
|
69
93
|
|
70
94
|
```
|
71
95
|
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
|
72
96
|
'{"source":"~/file.iso","target":"ftp://anonymous@localhost/incoming/dest2.iso"}' "http://localhost:3000/jobs"
|
73
97
|
```
|
74
98
|
|
75
|
-
|
99
|
+
Requesting notifications is achieved by passing a "notify" key in the request, with a callback URL.
|
100
|
+
This URL will be called at some points, ``POST```'ing a generic JSON structure with progress information.
|
101
|
+
|
102
|
+
|
103
|
+
* Start a job requesting notifications ``POST```'ed on "http://requestb.in/1321axg1"
|
76
104
|
|
77
105
|
```
|
78
106
|
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
|
79
107
|
'{"source":"~/file.dmg","target":"ftp://anonymous@localhost/incoming/dest4.dmg","notify":"http://requestb.in/1321axg1"}' "http://localhost:3000/jobs"
|
80
108
|
```
|
81
109
|
|
82
|
-
|
110
|
+
* Start a job with all the above plus a priority
|
111
|
+
|
112
|
+
```
|
113
|
+
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
|
114
|
+
'{"source":"~/file.dmg","priority":"3", target":"ftp://anonymous@localhost/incoming/dest4.dmg","notify":"http://requestb.in/1321axg1"}' "http://localhost:3000/jobs"
|
115
|
+
```
|
116
|
+
|
117
|
+
* Start a job using endpoint tokens
|
118
|
+
|
119
|
+
First define ``nas`` ans ``ftp1`` in the configuration file :
|
83
120
|
|
84
121
|
```
|
85
|
-
|
122
|
+
defaults: &defaults
|
123
|
+
|
124
|
+
development:
|
125
|
+
<<: *defaults
|
126
|
+
|
127
|
+
endpoints:
|
128
|
+
nas: "~/"
|
129
|
+
ftp1: "ftp://anonymous@localhost/incoming/"
|
86
130
|
```
|
87
131
|
|
88
|
-
|
132
|
+
Thos tokens will be expanded when the job is ran :
|
89
133
|
|
90
134
|
```
|
91
|
-
curl -H "Content-Type: application/json" -X
|
135
|
+
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
|
136
|
+
'{"source":"~/file.dmg","priority":"3", target":"ftp://anonymous@localhost/incoming/dest4.dmg","notify":"http://requestb.in/1321axg1"}' "http://localhost:3000/jobs"
|
137
|
+
```
|
138
|
+
|
139
|
+
NB: a special token [RANDOM] helps to generate a random filename when needed
|
140
|
+
|
141
|
+
* Get status of a specific job based on its ID
|
142
|
+
|
143
|
+
```
|
144
|
+
curl -H "Content-Type: application/json" -X GET -D /dev/stdout "http://localhost:3000/jobs/3"
|
145
|
+
```
|
146
|
+
|
147
|
+
|
148
|
+
* Delete a specific job based on its ID
|
149
|
+
|
150
|
+
```
|
151
|
+
curl -H "Content-Type: application/json" -X DELETE -D /dev/stdout "http://localhost:3000/jobs/3"
|
92
152
|
```
|
93
153
|
|
94
154
|
|
95
155
|
Getting status
|
96
156
|
------------------------------------------------------------------------------------
|
97
157
|
|
98
|
-
|
158
|
+
* A global JSON status is provided on ``` GET /index.json ```
|
159
|
+
|
160
|
+
* A nice dashboard gives a global view of the daemon, jobs in queue, and system status, exposed on ``` GET /```
|
161
|
+
|
162
|
+
* The server exposes its jobs list on ``` GET /jobs ```
|
99
163
|
|
100
164
|
```
|
101
165
|
http://localhost:3000/jobs
|
@@ -120,18 +184,19 @@ This query will return a job list :
|
|
120
184
|
"transferred": 37100000
|
121
185
|
},
|
122
186
|
{
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
"
|
129
|
-
|
130
|
-
|
131
|
-
"
|
132
|
-
"
|
133
|
-
|
134
|
-
|
187
|
+
source: "[nas]/file.dmg",
|
188
|
+
target: "[ftp2]/dest4.dmg",
|
189
|
+
notify: "http://requestb.in/1321axg1",
|
190
|
+
updated_at: "2014-09-17 22:56:14 +0200",
|
191
|
+
id: 2,
|
192
|
+
started_at: "2014-09-17 22:56:01 +0200",
|
193
|
+
status: "uploading",
|
194
|
+
error: 0,
|
195
|
+
debug_source: "/Users/bruno/file.dmg",
|
196
|
+
debug_target: "#<URI::FTP:0x007ffa9289e650 URL:ftp://uuuuuuuu:yyyyyyyyy@ftp.xxxxxx.fr/subdir/dest4.dmg>",
|
197
|
+
file_size: 32093208,
|
198
|
+
file_progress: 5.6,
|
199
|
+
file_sent: 1800000
|
135
200
|
}
|
136
201
|
]
|
137
202
|
```
|
data/bin/rest-ftp-daemon
CHANGED
@@ -2,29 +2,37 @@
|
|
2
2
|
|
3
3
|
# Libs and init
|
4
4
|
require "thin"
|
5
|
-
|
6
|
-
# Initialize some local constants
|
7
5
|
APP_ROOT = File.dirname(__FILE__) + '/../'
|
8
|
-
|
9
|
-
|
10
|
-
# Include config file
|
11
|
-
require File.expand_path "#{APP_ROOT}/lib/rest-ftp-daemon/config.rb"
|
6
|
+
DEVELOPMENT = ARGV.include?("development")
|
12
7
|
|
13
|
-
#
|
8
|
+
# Include config file and build rack commandline
|
9
|
+
require File.expand_path("#{APP_ROOT}/lib/rest-ftp-daemon/config.rb")
|
14
10
|
argv = ARGV
|
15
|
-
argv << ["-
|
16
|
-
argv << ["-
|
17
|
-
argv << ["
|
18
|
-
argv << ["
|
19
|
-
|
20
|
-
#
|
11
|
+
argv << ["-p", Settings.port.to_s] unless ARGV.include?("-p") || Settings.port.nil?
|
12
|
+
argv << ["-e", Settings.namespace] unless ARGV.include?("-e") || Settings.namespace.nil?
|
13
|
+
argv << ["-l", Settings.logs.thin] unless ARGV.include?("-l") unless Settings.logs.thin.nil? unless Settings.logs.nil?
|
14
|
+
argv << ["--daemonize"] if (Settings.daemonize==1 || Settings.daemonize.nil?) unless ARGV.include?("-d")
|
15
|
+
|
16
|
+
# Rackup file
|
17
|
+
argv << ["-R", File.expand_path("#{APP_ROOT}/config.ru")] unless ARGV.include?("-R")
|
18
|
+
|
19
|
+
# Display compiled configuration
|
20
|
+
puts
|
21
|
+
puts "Daemon name \t #{Settings.name}"
|
22
|
+
puts "Version \t #{Settings.version}"
|
23
|
+
puts "Environment \t #{Settings.namespace}"
|
24
|
+
puts "Config file \t #{Settings.source}"
|
25
|
+
puts "Parameters \t #{argv.flatten}"
|
26
|
+
puts Settings.to_hash.to_yaml( :Indent => 4, :UseHeader => true, :UseVersion => false )
|
21
27
|
|
22
28
|
# Start Thin with this rackup configuration
|
23
|
-
puts
|
29
|
+
puts
|
24
30
|
begin
|
25
31
|
Thin::Runner.new(argv.flatten).run!
|
32
|
+
rescue Thin::PidFileExist
|
33
|
+
puts "EXITING: daemon was already running (Thin::PidFileExist)"
|
26
34
|
rescue Thin::PidFileNotFound
|
27
|
-
puts "
|
35
|
+
puts "EXITING: daemon was not running (Thin::PidFileNotFound)"
|
28
36
|
end
|
29
37
|
|
30
38
|
|
data/config.ru
CHANGED
@@ -7,8 +7,9 @@ APP_STARTED = Time.now
|
|
7
7
|
|
8
8
|
# Create worker pool
|
9
9
|
$queue = RestFtpDaemon::JobQueue.new
|
10
|
-
$pool = RestFtpDaemon::WorkerPool.new(
|
10
|
+
$pool = RestFtpDaemon::WorkerPool.new(Settings.workers.to_i)
|
11
11
|
|
12
12
|
# Start REST FTP Daemon
|
13
13
|
#run Rack::Cascade.new [Rack::File.new("/public"), RestFtpDaemon::API::Root]
|
14
|
+
#run Rack::URLMap.new("/" => Frontend.new, "/api" => Api.new)
|
14
15
|
run Rack::Cascade.new [RestFtpDaemon::API::Root]
|
data/lib/rest-ftp-daemon.rb
CHANGED
@@ -1,8 +1,6 @@
|
|
1
1
|
# Global libs
|
2
|
+
require 'rubygems'
|
2
3
|
require 'json'
|
3
|
-
require 'grape'
|
4
|
-
require 'net/ftp'
|
5
|
-
require 'net/http'
|
6
4
|
require 'securerandom'
|
7
5
|
# require 'celluloid/autostart'
|
8
6
|
|
@@ -18,5 +16,4 @@ require 'rest-ftp-daemon/notification'
|
|
18
16
|
require 'rest-ftp-daemon/api/defaults'
|
19
17
|
require 'rest-ftp-daemon/api/jobs'
|
20
18
|
require 'rest-ftp-daemon/api/root'
|
21
|
-
require 'rest-ftp-daemon/www'
|
22
19
|
|
@@ -0,0 +1,105 @@
|
|
1
|
+
!!! 5
|
2
|
+
%html{:lang => "en"}
|
3
|
+
%head
|
4
|
+
%meta{:charset => "utf-8"}/
|
5
|
+
%link{ href:"//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css" , rel: "stylesheet"}
|
6
|
+
|
7
|
+
%title="#{Settings.name} dashboard"
|
8
|
+
|
9
|
+
:css
|
10
|
+
|
11
|
+
#footer {
|
12
|
+
padding: 15px 0;
|
13
|
+
_border-top: 1px solid silver;
|
14
|
+
}
|
15
|
+
|
16
|
+
.progress {
|
17
|
+
margin-bottom: 0;
|
18
|
+
}
|
19
|
+
|
20
|
+
%body
|
21
|
+
|
22
|
+
.container
|
23
|
+
%h1
|
24
|
+
= Settings.name
|
25
|
+
%small
|
26
|
+
= "v#{Settings.version}"
|
27
|
+
= "[#{Settings.namespace}]"
|
28
|
+
|
29
|
+
|
30
|
+
%h2 System status
|
31
|
+
.btn-group.btn-group-sm
|
32
|
+
.btn.btn-default.btn-warning Processors
|
33
|
+
.btn.btn-default= @info_procs
|
34
|
+
|
35
|
+
.btn-group.btn-group-sm
|
36
|
+
.btn.btn-default.btn-warning Load average
|
37
|
+
.btn.btn-default= @info_load.round(1)
|
38
|
+
|
39
|
+
.btn-group.btn-group-sm
|
40
|
+
.btn.btn-default.btn-warning Normalized load
|
41
|
+
.btn.btn-default= "#{@info_norm} %"
|
42
|
+
|
43
|
+
.btn-group.btn-group-sm
|
44
|
+
.btn.btn-default.btn-success IP Address
|
45
|
+
.btn.btn-default= @info_ipaddr
|
46
|
+
|
47
|
+
.btn-group.btn-group-sm
|
48
|
+
.btn.btn-default.btn-success Memory free
|
49
|
+
.btn.btn-default= @info_memfree
|
50
|
+
|
51
|
+
.btn-group.btn-group-sm
|
52
|
+
.btn.btn-default.btn-info Settings.workers
|
53
|
+
.btn.btn-default= Settings.workers
|
54
|
+
|
55
|
+
|
56
|
+
%h2
|
57
|
+
Jobs on this system (#{@jobs_all})
|
58
|
+
%small
|
59
|
+
queued (#{@jobs_queued}) / popped (#{@jobs_popped})
|
60
|
+
|
61
|
+
%table.table.table-striped.table-hover.table-condensed
|
62
|
+
%tr
|
63
|
+
%th= "ID"
|
64
|
+
%th= "priority"
|
65
|
+
%th= "source"
|
66
|
+
%th= "target"
|
67
|
+
%th= "status"
|
68
|
+
%th= "error"
|
69
|
+
%th= "progress"
|
70
|
+
|
71
|
+
- @jobs.each do |job|
|
72
|
+
- error = job.get :error
|
73
|
+
- status = job.get :status
|
74
|
+
- progress = job.get :progress
|
75
|
+
|
76
|
+
- if error!=0 && !error.nil?
|
77
|
+
- trclass = "danger"
|
78
|
+
- elsif status == :uploading
|
79
|
+
- trclass = "info"
|
80
|
+
- elsif status == :finished
|
81
|
+
- trclass = "success"
|
82
|
+
- else
|
83
|
+
- trclass = "warning"
|
84
|
+
|
85
|
+
%tr{class: trclass}
|
86
|
+
|
87
|
+
%td= job.id
|
88
|
+
%td= job.get :priority
|
89
|
+
%td= job.get :source
|
90
|
+
%td= job.get :target
|
91
|
+
%td= status
|
92
|
+
%td= error
|
93
|
+
%td
|
94
|
+
- unless progress.nil?
|
95
|
+
.progress
|
96
|
+
.progress-bar{style:"width: #{progress}%;"}
|
97
|
+
= "#{progress} %"
|
98
|
+
|
99
|
+
|
100
|
+
%h2 Endpoint tokens
|
101
|
+
- endpoints = Settings.endpoints || {}
|
102
|
+
- endpoints.each do |token, value|
|
103
|
+
.btn-group.btn-group-sm
|
104
|
+
.btn.btn-default.btn-danger= token
|
105
|
+
.btn.btn-default= value
|
@@ -3,7 +3,7 @@ module RestFtpDaemon
|
|
3
3
|
|
4
4
|
class Jobs < Grape::API
|
5
5
|
include RestFtpDaemon::API::Defaults
|
6
|
-
logger ActiveSupport::Logger.new
|
6
|
+
logger ActiveSupport::Logger.new Settings.logs.api, 'daily' unless Settings.logs.api.nil?
|
7
7
|
|
8
8
|
params do
|
9
9
|
optional :overwrite, type: Integer, default: false
|
@@ -1,9 +1,14 @@
|
|
1
|
+
require 'haml'
|
2
|
+
require "facter"
|
3
|
+
require "sys/cpu"
|
4
|
+
|
5
|
+
|
1
6
|
module RestFtpDaemon
|
2
7
|
module API
|
3
8
|
|
4
9
|
class Root < Grape::API
|
5
10
|
include RestFtpDaemon::API::Defaults
|
6
|
-
logger ActiveSupport::Logger.new
|
11
|
+
logger ActiveSupport::Logger.new Settings.logs.api, 'daily' unless Settings.logs.api.nil?
|
7
12
|
#add_swagger_documentation
|
8
13
|
|
9
14
|
mount RestFtpDaemon::API::Jobs => '/jobs'
|
@@ -41,12 +46,54 @@ module RestFtpDaemon
|
|
41
46
|
|
42
47
|
# Server global status
|
43
48
|
get '/' do
|
49
|
+
# Prepare data
|
50
|
+
@jobs = $queue.all
|
51
|
+
@jobs_all = $queue.all_size
|
52
|
+
@jobs_queued = $queue.queued_size
|
53
|
+
@jobs_popped = $queue.popped_size
|
54
|
+
|
55
|
+
# Initialize UsageWatch
|
56
|
+
Facter.loadfacts
|
57
|
+
@info_load = Sys::CPU.load_avg.first.to_f
|
58
|
+
@info_procs = (Facter.value :processorcount).to_i
|
59
|
+
@info_ipaddr = Facter.value :ipaddress
|
60
|
+
@info_memfree = Facter.value :memoryfree
|
61
|
+
|
62
|
+
# Compute normalized load
|
63
|
+
if @info_procs.zero?
|
64
|
+
@info_norm = "N/A"
|
65
|
+
else
|
66
|
+
@info_norm = (100 * @info_load / @info_procs).round(1)
|
67
|
+
end
|
68
|
+
|
69
|
+
# Compile haml template
|
70
|
+
@name = "Test"
|
71
|
+
template = File.read("#{File.dirname(__FILE__)}/index.haml")
|
72
|
+
haml_engine = Haml::Engine.new(template)
|
73
|
+
output = haml_engine.render(binding)
|
74
|
+
#[200, {'Content-Type' => 'text/html'}, [output]]
|
75
|
+
|
76
|
+
# Generate output
|
77
|
+
# env['api.format'] = :binary
|
78
|
+
# env['api.format'] = :txt
|
79
|
+
|
80
|
+
# Send response
|
81
|
+
env['api.format'] = :html
|
82
|
+
format "html"
|
83
|
+
status 200
|
84
|
+
content_type "text/html"
|
85
|
+
body output
|
86
|
+
|
87
|
+
end
|
88
|
+
|
89
|
+
# Server global status
|
90
|
+
get '/index.json' do
|
44
91
|
info "GET /"
|
45
92
|
status 200
|
46
93
|
return {
|
47
|
-
name: RestFtpDaemon::NAME,
|
48
94
|
hostname: `hostname`.chomp,
|
49
|
-
version:
|
95
|
+
version: Settings.version,
|
96
|
+
config: Settings.to_hash,
|
50
97
|
started: APP_STARTED,
|
51
98
|
uptime: (Time.now - APP_STARTED).round(1),
|
52
99
|
status: job_list_by_status,
|
@@ -0,0 +1,49 @@
|
|
1
|
+
# module RestFtpDaemon
|
2
|
+
# module API
|
3
|
+
|
4
|
+
# class Workers < Grape::API
|
5
|
+
# include RestFtpDaemon::API::Defaults
|
6
|
+
# logger ActiveSupport::Logger.new Settings.logs.api, 'daily' unless Settings.logs.api.nil?
|
7
|
+
|
8
|
+
# helpers do
|
9
|
+
# def info message, level = 0
|
10
|
+
# Jobs.logger.add(Logger::INFO, "#{' '*level} #{message}", "API::Workers")
|
11
|
+
# end
|
12
|
+
|
13
|
+
# def worker_list
|
14
|
+
# return {
|
15
|
+
# busy: $pool.busy_size,
|
16
|
+
# idle: $pool.idle_size,
|
17
|
+
# to_s: $pool.to_s,
|
18
|
+
# }
|
19
|
+
# #return $workers.list.size
|
20
|
+
# $workers.list.map do |thread|
|
21
|
+
# #next unless thread[:job].is_a? Worker
|
22
|
+
# "worker"
|
23
|
+
# #thread[:worker].inspect
|
24
|
+
# end
|
25
|
+
# end
|
26
|
+
|
27
|
+
# end
|
28
|
+
|
29
|
+
# # List jobs
|
30
|
+
# desc "Get a list of workers"
|
31
|
+
# get do
|
32
|
+
# info "GET /workers"
|
33
|
+
# begin
|
34
|
+
# response = worker_list
|
35
|
+
# rescue RestFtpDaemonException => exception
|
36
|
+
# status 501
|
37
|
+
# api_error exception
|
38
|
+
# rescue Exception => exception
|
39
|
+
# status 501
|
40
|
+
# api_error exception
|
41
|
+
# else
|
42
|
+
# status 200
|
43
|
+
# response
|
44
|
+
# end
|
45
|
+
# end
|
46
|
+
|
47
|
+
# end
|
48
|
+
# end
|
49
|
+
# end
|
@@ -1,15 +1,24 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
|
4
|
-
VERSION = "0.41"
|
5
|
-
PORT = 3000
|
1
|
+
require 'settingslogic'
|
2
|
+
DEVELOPMENT = false unless defined? DEVELOPMENT
|
3
|
+
CONFIG_FILE = "/etc/rest-ftp-daemon.yml"
|
6
4
|
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
# Logging
|
12
|
-
APP_LOGTO = "/tmp/#{NAME}.log"
|
13
|
-
LOG_TRIM_PROGNAME = 18
|
5
|
+
class Settings < Settingslogic
|
6
|
+
namespace DEVELOPMENT ? "development" : "production"
|
7
|
+
suppress_errors namespace!="development"
|
8
|
+
end
|
14
9
|
|
10
|
+
# Fix application defaults and load config file if found
|
11
|
+
if File.exists? CONFIG_FILE
|
12
|
+
Settings.source CONFIG_FILE
|
13
|
+
else
|
14
|
+
Settings.source Hash.new
|
15
15
|
end
|
16
|
+
|
17
|
+
# Forced shared settings
|
18
|
+
Settings[:name] = "rest-ftp-daemon"
|
19
|
+
Settings[:version] = 0.55
|
20
|
+
|
21
|
+
# Forced fixed settings
|
22
|
+
Settings[:default_trim_progname] = "18"
|
23
|
+
Settings[:default_chunk_size] = "1000000"
|
24
|
+
|
data/lib/rest-ftp-daemon/job.rb
CHANGED
@@ -1,3 +1,6 @@
|
|
1
|
+
require 'net/ftp'
|
2
|
+
|
3
|
+
|
1
4
|
module RestFtpDaemon
|
2
5
|
class Job < RestFtpDaemon::Common
|
3
6
|
|
@@ -10,22 +13,15 @@ module RestFtpDaemon
|
|
10
13
|
@target = nil
|
11
14
|
@source = nil
|
12
15
|
|
13
|
-
# Logger
|
14
|
-
#@logger = ActiveSupport::Logger.new APP_LOGTO, 'daily'
|
15
|
-
|
16
16
|
# Init context
|
17
17
|
set :id, id
|
18
18
|
set :started_at, Time.now
|
19
|
-
set :status, :
|
19
|
+
set :status, :created
|
20
20
|
|
21
21
|
# Send first notification
|
22
22
|
notify "rftpd.queued"
|
23
23
|
end
|
24
24
|
|
25
|
-
# def job_id
|
26
|
-
# get :id
|
27
|
-
# end
|
28
|
-
|
29
25
|
def progname
|
30
26
|
job_id = get(:id)
|
31
27
|
"JOB #{job_id}"
|
@@ -99,6 +95,11 @@ module RestFtpDaemon
|
|
99
95
|
@status = text
|
100
96
|
end
|
101
97
|
|
98
|
+
def get attribute
|
99
|
+
return nil unless @params.is_a? Enumerable
|
100
|
+
@params[attribute.to_s]
|
101
|
+
end
|
102
|
+
|
102
103
|
protected
|
103
104
|
|
104
105
|
def up_time
|
@@ -130,9 +131,28 @@ module RestFtpDaemon
|
|
130
131
|
@params[attribute.to_s] = value
|
131
132
|
end
|
132
133
|
|
133
|
-
def
|
134
|
-
|
135
|
-
|
134
|
+
def expand_path_from path
|
135
|
+
File.expand_path replace_token_in_path(path)
|
136
|
+
end
|
137
|
+
|
138
|
+
def expand_url_from path
|
139
|
+
URI replace_token_in_path(path) rescue nil
|
140
|
+
end
|
141
|
+
|
142
|
+
def replace_token_in_path path
|
143
|
+
# Ensure endpoints are not a nil value
|
144
|
+
return path unless Settings.endpoints.is_a? Enumerable
|
145
|
+
newpath = path.clone
|
146
|
+
|
147
|
+
# Replace endpoints defined in config
|
148
|
+
Settings.endpoints.each do |from, to|
|
149
|
+
newpath.gsub! "[#{from}]", to
|
150
|
+
end
|
151
|
+
|
152
|
+
# Replace with the special RAND token
|
153
|
+
newpath.gsub! "[RANDOM]", SecureRandom.hex(8)
|
154
|
+
|
155
|
+
return newpath
|
136
156
|
end
|
137
157
|
|
138
158
|
def prepare
|
@@ -141,15 +161,18 @@ module RestFtpDaemon
|
|
141
161
|
|
142
162
|
# Check source
|
143
163
|
raise JobSourceMissing unless @params["source"]
|
144
|
-
@source =
|
164
|
+
@source = expand_path_from @params["source"]
|
145
165
|
set :debug_source, @source
|
146
|
-
raise JobSourceNotFound unless File.exists? @source
|
147
166
|
|
148
167
|
# Check target
|
149
168
|
raise JobTargetMissing unless @params["target"]
|
150
|
-
@target =
|
169
|
+
@target = expand_url_from @params["target"]
|
151
170
|
set :debug_target, @target.inspect
|
171
|
+
|
172
|
+
# Check compliance
|
152
173
|
raise JobTargetUnparseable if @target.nil?
|
174
|
+
raise JobSourceNotFound unless File.exists? @source
|
175
|
+
|
153
176
|
end
|
154
177
|
|
155
178
|
def transfer_fake
|
@@ -200,20 +223,24 @@ module RestFtpDaemon
|
|
200
223
|
|
201
224
|
# Do transfer
|
202
225
|
set :status, :uploading
|
203
|
-
|
226
|
+
chunk_size = Settings.transfer.chunk_size || Settings[:default_chunk_size]
|
227
|
+
ftp.putbinaryfile(@source, target_name, chunk_size) do |block|
|
204
228
|
# Update counters
|
205
229
|
transferred += block.bytesize
|
206
230
|
|
207
231
|
# Update job info
|
208
232
|
percent = (100.0 * transferred / source_size).round(1)
|
209
|
-
set :
|
233
|
+
set :progress, percent
|
210
234
|
set :file_sent, transferred
|
211
235
|
end
|
212
236
|
|
213
237
|
# Close FTP connexion
|
214
238
|
notify "rftpd.ended"
|
239
|
+
set :progress, nil
|
215
240
|
ftp.close
|
216
241
|
end
|
217
242
|
|
243
|
+
private
|
244
|
+
|
218
245
|
end
|
219
246
|
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
class Logger
|
2
2
|
def format_message(severity, timestamp, progname, msg)
|
3
3
|
stamp = Time.now.strftime("%Y-%m-%d %H:%M:%S")
|
4
|
-
progname = "%-#{
|
4
|
+
progname = "%-#{Settings[:default_trim_progname]}s" % progname
|
5
5
|
"#{stamp} #{severity} #{progname} #{msg}\n"
|
6
6
|
end
|
7
7
|
end
|
data/rest-ftp-daemon.gemspec
CHANGED
@@ -4,7 +4,8 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
require 'rest-ftp-daemon/config'
|
5
5
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name = RestFtpDaemon::NAME
|
7
|
+
#spec.name = RestFtpDaemon::NAME
|
8
|
+
spec.name = Settings[:name]
|
8
9
|
spec.date = Time.now.strftime("%Y-%m-%d")
|
9
10
|
spec.authors = ["Bruno MEDICI"]
|
10
11
|
spec.email = "rest-ftp-daemon@bmconseil.com"
|
@@ -16,7 +17,7 @@ Gem::Specification.new do |spec|
|
|
16
17
|
spec.files = `git ls-files -z`.split("\x0")
|
17
18
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
18
19
|
spec.require_paths = ["lib"]
|
19
|
-
spec.version =
|
20
|
+
spec.version = Settings[:version]
|
20
21
|
|
21
22
|
spec.required_ruby_version = '>= 1.9.3'
|
22
23
|
|
@@ -25,6 +26,8 @@ Gem::Specification.new do |spec|
|
|
25
26
|
|
26
27
|
spec.add_runtime_dependency "thin", "~> 1.6"
|
27
28
|
spec.add_runtime_dependency "grape"
|
29
|
+
spec.add_runtime_dependency "facter"
|
30
|
+
spec.add_runtime_dependency "settingslogic"
|
28
31
|
spec.add_runtime_dependency "json"
|
29
32
|
|
30
33
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rest-ftp-daemon
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: '0.
|
4
|
+
version: '0.55'
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Bruno MEDICI
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-09-
|
11
|
+
date: 2014-09-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -66,6 +66,34 @@ dependencies:
|
|
66
66
|
- - ! '>='
|
67
67
|
- !ruby/object:Gem::Version
|
68
68
|
version: '0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: facter
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ! '>='
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ! '>='
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: settingslogic
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ! '>='
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :runtime
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ! '>='
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
69
97
|
- !ruby/object:Gem::Dependency
|
70
98
|
name: json
|
71
99
|
requirement: !ruby/object:Gem::Requirement
|
@@ -98,8 +126,10 @@ files:
|
|
98
126
|
- config.ru
|
99
127
|
- lib/rest-ftp-daemon.rb
|
100
128
|
- lib/rest-ftp-daemon/api/defaults.rb
|
129
|
+
- lib/rest-ftp-daemon/api/index.haml
|
101
130
|
- lib/rest-ftp-daemon/api/jobs.rb
|
102
131
|
- lib/rest-ftp-daemon/api/root.rb
|
132
|
+
- lib/rest-ftp-daemon/api/workers.rb
|
103
133
|
- lib/rest-ftp-daemon/common.rb
|
104
134
|
- lib/rest-ftp-daemon/config.rb
|
105
135
|
- lib/rest-ftp-daemon/exceptions.rb
|
@@ -108,8 +138,8 @@ files:
|
|
108
138
|
- lib/rest-ftp-daemon/logger.rb
|
109
139
|
- lib/rest-ftp-daemon/notification.rb
|
110
140
|
- lib/rest-ftp-daemon/worker_pool.rb
|
111
|
-
- lib/rest-ftp-daemon/www.rb
|
112
141
|
- rest-ftp-daemon.gemspec
|
142
|
+
- rest-ftp-daemon.yml.sample
|
113
143
|
- test/helper.rb
|
114
144
|
- test/test_rest-ftp-daemon.rb
|
115
145
|
homepage: http://github.com/bmedici/rest-ftp-daemon
|