rda-python-dscheck 2.0.3__tar.gz → 2.0.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/PKG-INFO +6 -2
- rda_python_dscheck-2.0.5/README.md +6 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/pyproject.toml +1 -1
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/dscheck.usg +92 -88
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/pg_check.py +1 -1
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/PKG-INFO +6 -2
- rda_python_dscheck-2.0.3/README.md +0 -2
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/LICENSE +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/MANIFEST.in +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/setup.cfg +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/PgCheck.py +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/__init__.py +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/ds_check.py +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck/dscheck.py +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/SOURCES.txt +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/dependency_links.txt +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/entry_points.txt +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/requires.txt +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/top_level.txt +0 -0
- {rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/tests/test_dscheck.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: rda_python_dscheck
|
|
3
|
-
Version: 2.0.
|
|
3
|
+
Version: 2.0.5
|
|
4
4
|
Summary: RDA python package to add and process batch jobs
|
|
5
5
|
Author-email: Zaihua Ji <zji@ucar.edu>
|
|
6
6
|
Project-URL: Homepage, https://github.com/NCAR/rda-python-dscheck
|
|
@@ -14,5 +14,9 @@ License-File: LICENSE
|
|
|
14
14
|
Requires-Dist: rda_python_common
|
|
15
15
|
Dynamic: license-file
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
# dscheck
|
|
18
|
+
|
|
19
|
+
Python project to add and process batch jobs for the [NSF NCAR Geoscience Data Exchange (GDEX)](https://gdex.ucar.edu).
|
|
20
|
+
|
|
21
|
+
The user guide for this utility tool can be viewed at: [User guide](https://gdex-docs-dscheck.readthedocs.io).
|
|
18
22
|
|
|
@@ -2,28 +2,28 @@
|
|
|
2
2
|
1 INTRODUCTION
|
|
3
3
|
|
|
4
4
|
Program 'dscheck' is a utility to add, remove, view and process recorded commands
|
|
5
|
-
of other utility programs in Research Data Archive Management System (
|
|
5
|
+
of other utility programs in Research Data Archive Management System (GDEXMS). For
|
|
6
6
|
delayed execution, or called batch process, of utility programs, 'dsarch', 'dsupdt'
|
|
7
7
|
and 'dsrqst', the commands information and the directories where the commands are
|
|
8
|
-
initiated are saved into
|
|
8
|
+
initiated are saved into GDEXDB as check records. For any other specialist-defined
|
|
9
9
|
commands, they can be also put in delayed mode if the commands are added to
|
|
10
10
|
'dscheck' control via Action -AC (-AddCheck).
|
|
11
11
|
|
|
12
12
|
The check records are processed automatically via a centralized 'dscheck' daemon,
|
|
13
13
|
although they can be processed manually on command line. During the execution of a
|
|
14
|
-
recorded command, the check record is locked in
|
|
14
|
+
recorded command, the check record is locked in GDEXDB to prevent multiple executions
|
|
15
15
|
of the same command. The check records that retain command information are
|
|
16
16
|
automatically purged into check history when the commands are finished.
|
|
17
17
|
|
|
18
18
|
When a recorded command fails due to failures of computer system, storage disk/tape,
|
|
19
19
|
or the Database Management System, the check record is normally purged with status 'E'
|
|
20
20
|
for error, unless check-reprocessing ability is built into the utility program, such
|
|
21
|
-
as 'dsarch'. For a check-reprocessing command, the check record is retained in
|
|
22
|
-
|
|
21
|
+
as 'dsarch'. For a check-reprocessing command, the check record is retained in GDEXDB
|
|
22
|
+
until the command is processed successfully or the number of executions reaches the
|
|
23
23
|
try limits allowed. Utility programs 'dsrqst' and 'dsupdt' carry their own failure-
|
|
24
24
|
recovering ability and they do not need check-reprocessing.
|
|
25
25
|
|
|
26
|
-
The purged check records are retained in
|
|
26
|
+
The purged check records are retained in GDEXDB. The check history can be viewed via
|
|
27
27
|
utility program 'viewcheckusage'.
|
|
28
28
|
|
|
29
29
|
Program 'dscheck' supports the following major functions:
|
|
@@ -33,21 +33,21 @@ Program 'dscheck' supports the following major functions:
|
|
|
33
33
|
a check. Without the daemon control information, a recorded command will not be
|
|
34
34
|
started automatically.
|
|
35
35
|
- ADD a check record for delayed command execution of any specialist-defined commands
|
|
36
|
-
- View utility command information currently saved in
|
|
36
|
+
- View utility command information currently saved in GDEXDB
|
|
37
37
|
- Email the current status of a specified utility command or a list of commands,
|
|
38
38
|
and include error messages if any, to a specialist
|
|
39
39
|
- Delete recorded command information, because the command is not needed anymore
|
|
40
40
|
- Unlock a given recorded command in case lock information was not cleaned
|
|
41
|
-
properly when the command
|
|
41
|
+
properly when the command has failed
|
|
42
42
|
- Interrupt a utility command that is under execution and kill recursively all
|
|
43
43
|
the associated child processes
|
|
44
44
|
- Add the due 'dsrqst' and 'dsupdt' actions into dscheck records
|
|
45
|
-
- Process commands that have been recorded into
|
|
46
|
-
previously.
|
|
45
|
+
- Process commands that have been recorded into GDEXDB or the ones that have
|
|
46
|
+
failed previously.
|
|
47
47
|
|
|
48
48
|
The specialist who executes a utility command under 'dscheck' control remains
|
|
49
|
-
the exclusive owner of the check record in
|
|
50
|
-
processed or deleted accidentally by other specialists.
|
|
49
|
+
the exclusive owner of the check record in GDEXDB. This prevents the command from being
|
|
50
|
+
processed or deleted accidentally by other specialists.
|
|
51
51
|
|
|
52
52
|
In the following sections, general usages of 'dscheck' are described first; and
|
|
53
53
|
detail descriptions of Action options are given; and finally Mode and Info
|
|
@@ -62,7 +62,7 @@ options are explained.
|
|
|
62
62
|
Quotes [] indicate optional. A pipeline '|' in parentheses as in format (A|B)
|
|
63
63
|
means either A or B can be used. The options applied to 'dscheck' are divided
|
|
64
64
|
into three categories, Action, Mode, and Information (Info for short) options.
|
|
65
|
-
Action options are used to specify what
|
|
65
|
+
Action options are used to specify what task this utility program is to execute,
|
|
66
66
|
Mode options are used to modify behaviors of the actions, and the Info options
|
|
67
67
|
are used to pass information, one or multiple values, to run 'dscheck'. An option
|
|
68
68
|
can be given in form of either short name or long name, -DS or -Dataset for
|
|
@@ -77,18 +77,18 @@ and certain Mode options can be applied to alter the behaviors of the Action.
|
|
|
77
77
|
|
|
78
78
|
All options, except Info option -IF (-InputFile), can be given either on command
|
|
79
79
|
line or in input files. Input file names are presented per Info option -IF and
|
|
80
|
-
can only be provided on command line.
|
|
81
|
-
(-InputFile) for
|
|
80
|
+
can only be provided on command line. Refer to the description of Info option -IF
|
|
81
|
+
(-InputFile) for details on how to present options in input files. One or multiple
|
|
82
82
|
input files, combined with options on command line, are allowed to run 'dscheck'.
|
|
83
83
|
The option name, -IF (-InputFile), itself can be omitted if a single input file is
|
|
84
84
|
given on command line and all other option information are provided inside the
|
|
85
85
|
input file.
|
|
86
86
|
|
|
87
87
|
When information of daemons and checks are retrieved, Info options are used to
|
|
88
|
-
specify conditions for querying information from
|
|
88
|
+
specify conditions for querying information from GDEXDB. Some special signs can be
|
|
89
89
|
used to further confine the information with special and complicated conditions;
|
|
90
90
|
they are '!', '<', '>' and '<>'. These special signs, if provided on command
|
|
91
|
-
line, must be quoted or escaped to avoid
|
|
91
|
+
line, must be quoted or escaped to avoid being interpreted by the Unix shell.
|
|
92
92
|
The '!', or \!, means exclusion to the following value(s) and it must be the
|
|
93
93
|
first item following an Info option name, while '<' or '>' mean greater or less
|
|
94
94
|
than the following value and '<>' means between the following two values.
|
|
@@ -102,11 +102,11 @@ command line as
|
|
|
102
102
|
|
|
103
103
|
A description is displayed for an option given either before or after -(h|help).
|
|
104
104
|
If no option is specified or 'dscheck' is issued by itself, this whole document
|
|
105
|
-
is displayed
|
|
106
|
-
printed from the saved file
|
|
105
|
+
is displayed via UNIX utility 'more'. A hard copy of this help document can be
|
|
106
|
+
printed from the saved file, dscheck.usg, under python package rda_python_dscheck.
|
|
107
107
|
|
|
108
108
|
#The online HTML version of this document is available at
|
|
109
|
-
#
|
|
109
|
+
#https://gdex-docs-dscheck.readthedocs.io
|
|
110
110
|
|
|
111
111
|
|
|
112
112
|
3 ACTION OPTIONS
|
|
@@ -117,27 +117,27 @@ simultaneously are blocked.
|
|
|
117
117
|
|
|
118
118
|
Based on the information being manipulated, the actions are divided into three
|
|
119
119
|
categories:
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
120
|
+
Daemon Control Actions - create, delete, modify and view daemon control information
|
|
121
|
+
in GDEXDB, of specified specialist, command and hostname
|
|
122
|
+
Check Actions - add, delete, unlock and view check information of the
|
|
123
|
+
active individual checks
|
|
124
|
+
Check Process Actions - process checks by starting commands on remote hosts as
|
|
125
|
+
configured in daemon controls and purge checks by recording
|
|
126
|
+
the commands and their execution information into check
|
|
127
|
+
history; interrupt the current executing commands by killing
|
|
128
|
+
the current process and its all children; and email status
|
|
129
|
+
of current checks
|
|
130
130
|
Daemon Host Connectivity - check connectivity of daemon hosts for specialists
|
|
131
131
|
|
|
132
132
|
3.1 Daemon Control Actions
|
|
133
133
|
|
|
134
134
|
A daemon control record for a command, a specialist and a hostname is used to
|
|
135
135
|
configure how many concurrent processes the specified command can be executed
|
|
136
|
-
for the specialist on specified hostname, and the priority the
|
|
136
|
+
for the specialist on specified hostname, and the priority the hostname is
|
|
137
137
|
picked to start the command. A running centralized daemon reads this record
|
|
138
138
|
periodically in case the configuration is changed while the daemon is still
|
|
139
139
|
running, so that specialists can reset the values in daemon control records to
|
|
140
|
-
change the
|
|
140
|
+
change the behavior of dscheck daemon dynamically without shutting the daemon down.
|
|
141
141
|
|
|
142
142
|
Daemon control information can be created, modified and viewed via Actions
|
|
143
143
|
included in this section:
|
|
@@ -148,7 +148,7 @@ categories:
|
|
|
148
148
|
|
|
149
149
|
3.1.1 Set Daemon Control
|
|
150
150
|
-SD or -SetDaemon, creates and modifies daemon control information into
|
|
151
|
-
|
|
151
|
+
GDEXDB for given specialist login names, commands, and hostnames of computers on
|
|
152
152
|
which the check commands are processed. One or multiple records can be set each
|
|
153
153
|
time.
|
|
154
154
|
|
|
@@ -162,13 +162,13 @@ categories:
|
|
|
162
162
|
[-(PO|Priority) HostListOrder]
|
|
163
163
|
|
|
164
164
|
Mode option that can be specified for this action include:
|
|
165
|
-
-(ND|NewDaemon) - sets a new daemon control record into
|
|
165
|
+
-(ND|NewDaemon) - sets a new daemon control record into GDEXDB
|
|
166
166
|
|
|
167
|
-
If information of a daemon control exists already in
|
|
167
|
+
If information of a daemon control exists already in GDEXDB for a given specialist,
|
|
168
168
|
a command and a hostname, the daemon control record is modified; otherwise, a new
|
|
169
169
|
daemon control record is added if daemon index is 0 and Mode option -ND (-NewDaemon)
|
|
170
170
|
is present. Combination of specialist login name, command name and hostname of
|
|
171
|
-
computer must be unique for
|
|
171
|
+
computer must be unique for each daemon control record.
|
|
172
172
|
|
|
173
173
|
Specify host name 'PBS' for putting the command in the PBS batch control system. If
|
|
174
174
|
a specified command name is not found in the daemon control, the general 'dscheck'
|
|
@@ -176,12 +176,12 @@ categories:
|
|
|
176
176
|
|
|
177
177
|
For example, set daemon control information for schuster, all commands on PBS hosts,
|
|
178
178
|
for maximum 4 checks can be processed at the same time with priority 1, the smaller
|
|
179
|
-
the number the higher the priority is, via input file daemon.ctl
|
|
179
|
+
the number the higher the priority is, via input file daemon.ctl:
|
|
180
180
|
|
|
181
|
-
|
|
181
|
+
dscheck -SD -ND -IF daemon.ctl
|
|
182
182
|
|
|
183
183
|
<<Content of input file daemon.ctl>>
|
|
184
|
-
DaemonIndex<:>Command<:>
|
|
184
|
+
DaemonIndex<:>Specialist<:>Command<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
185
185
|
0<:>schuster<:>ALL<:>PBS<:>4<:>1<:>
|
|
186
186
|
|
|
187
187
|
|
|
@@ -217,11 +217,11 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
217
217
|
I -(DI|DaemonIndex) daemon control index
|
|
218
218
|
C -(CM|Command) command names of utility programs
|
|
219
219
|
H -(HN|Hostname) computer hostname
|
|
220
|
-
M -(MH|MatchHost)
|
|
220
|
+
M -(MH|MatchHost) Flags to control hostname match
|
|
221
221
|
S -(SN|Specialist) DECS specialist the daemon set for
|
|
222
222
|
P -(PL|ProcessLimit) Max check count to be processed at the same times
|
|
223
223
|
O -(PO|Priority) host priority a specified command to start on
|
|
224
|
-
|
|
224
|
+
|
|
225
225
|
Daemon control information can be retrieved for specified specialist per option
|
|
226
226
|
-SN (-Specialist), and/or other conditions. Info option -SN, -CM and -HN accept
|
|
227
227
|
wildcard input of '%' for matching any number of characters.
|
|
@@ -232,14 +232,14 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
232
232
|
option -SN (-Specialist). To view all control records, you simply provide option
|
|
233
233
|
-SN with value of 'ALL'.
|
|
234
234
|
|
|
235
|
-
For example, to get all daemon control information currently set for you
|
|
235
|
+
For example, to get all daemon control information currently set for you:
|
|
236
236
|
|
|
237
237
|
dscheck GD
|
|
238
238
|
|
|
239
239
|
|
|
240
240
|
3.1.3 Delete Daemon Control
|
|
241
241
|
-DL or -Delete (Alias: -RM, -Remove), deletes one or multiple daemon control records
|
|
242
|
-
from
|
|
242
|
+
from GDEXDB for given daemon control indices.
|
|
243
243
|
|
|
244
244
|
dscheck -(DL|Delete)
|
|
245
245
|
-(DI|DaemonIndex) DaemonControlIndices
|
|
@@ -289,7 +289,7 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
289
289
|
as the owner of the added check record, and the current path is defaulted as the
|
|
290
290
|
working directory when the command is executed later.
|
|
291
291
|
|
|
292
|
-
|
|
292
|
+
Specify additional PBS batch options via Info option -QS (-QsubOptions) to add a check;
|
|
293
293
|
and specify a parent check index to put the current command on hold until the parent
|
|
294
294
|
check is finished.
|
|
295
295
|
|
|
@@ -297,29 +297,33 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
297
297
|
A simple shell script can be used to wrap a complicated command.
|
|
298
298
|
|
|
299
299
|
For example, to list file names in the current directory with name containing 'test'
|
|
300
|
-
by catching the standard output into a log and display it on screen
|
|
300
|
+
by catching the standard output into a log and display it on screen:
|
|
301
|
+
|
|
302
|
+
dscheck -AC -CM test1.sh -HN PBS
|
|
301
303
|
|
|
302
304
|
<<Content of shell script test1.sh>>
|
|
303
305
|
#!/bin/sh
|
|
304
306
|
(ls -l | grep test) | tee test1.out
|
|
305
307
|
|
|
306
308
|
For example, to list file names in the current directory with name containing 'test'
|
|
307
|
-
by catching the standard output and error into separate log files
|
|
308
|
-
|
|
309
|
+
by catching the standard output and error into separate log files:
|
|
310
|
+
|
|
311
|
+
dscheck -AC -CM test2.sh -HN PBS
|
|
312
|
+
|
|
309
313
|
<<Content of shell script test2.sh>>
|
|
310
314
|
#!/bin/sh
|
|
311
315
|
(ls -l | grep test) 1> test2.out 2>test2.err
|
|
312
316
|
|
|
313
317
|
For example, to add testing command 'test2' into 'dscheck' for delayed mode execution on
|
|
314
|
-
PBS
|
|
318
|
+
PBS:
|
|
315
319
|
|
|
316
|
-
|
|
320
|
+
dscheck -AC -CM test3 -HN PBS
|
|
317
321
|
|
|
318
|
-
The command '
|
|
322
|
+
The command 'test3' must be executable at the current working directory on PBS machines.
|
|
319
323
|
|
|
320
324
|
|
|
321
325
|
3.2.2 Get Check
|
|
322
|
-
-GC or -GetCheck, gets check information recorded in
|
|
326
|
+
-GC or -GetCheck, gets check information recorded in GDEXDB.
|
|
323
327
|
|
|
324
328
|
dscheck -(GC|GetCheck) [Mode Options]
|
|
325
329
|
[-(FN|FieldNames) FieldNameString]
|
|
@@ -343,7 +347,7 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
343
347
|
of a given field
|
|
344
348
|
|
|
345
349
|
Use Info option -FN (-FieldNames) to specify what check fields to retrieve.
|
|
346
|
-
It defaults to "
|
|
350
|
+
It defaults to "COVTUPFJDNW" if -FN is not given.
|
|
347
351
|
|
|
348
352
|
Valid check field names and their corresponding Info options:
|
|
349
353
|
|
|
@@ -354,21 +358,21 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
354
358
|
T -(DS|Dataset) dataset ID, the original command run against
|
|
355
359
|
A -(AN|ActionName) action name for a given command
|
|
356
360
|
U -(ST|Status) check status for a recorded command
|
|
357
|
-
B -(DF|DownFlags) Storage system down flags:
|
|
358
|
-
P -(PQ|PBSQueue) PBS batch queue name:
|
|
361
|
+
B -(DF|DownFlags) Storage system down flags: D-DRDATA,G-GLADE,O-ObjectStore
|
|
362
|
+
P -(PQ|PBSQueue) PBS batch queue name: gdex or htc
|
|
359
363
|
R -(PI|ParentIndex) parent check index the current one to wait on
|
|
360
364
|
F -(FC|FileCount) number of files need to be processed
|
|
361
365
|
J -(DC|DoneCount) number of files are processed already
|
|
362
366
|
K -(TC|TryCount) number of tries the command is executed
|
|
363
367
|
L -(MC|MaxCount) upper limit for number of command tries
|
|
364
|
-
Z -(SZ|DataSize) total
|
|
368
|
+
Z -(SZ|DataSize) total bytes of data processed for the command
|
|
365
369
|
D -(CD|CheckDate) date the command is initially recorded
|
|
366
370
|
Y -(CT|CheckTime) time the command is initially recorded
|
|
367
371
|
H -(HN|HostName) host names the command can or cannot run on
|
|
368
372
|
N -(SN|Specialist) specialist login name who starts the command
|
|
369
373
|
W -(WD|WorkDir) working directory where the command is started
|
|
370
374
|
M -(MO|Modules) include modules to load to batch job script
|
|
371
|
-
I -(EV|Environments) include environment
|
|
375
|
+
I -(EV|Environments) include environment variables to load to batch job script
|
|
372
376
|
Q -(QS|QsubOptions) additional PBS batch options for qsub
|
|
373
377
|
X -(AX|ArgumentExtra) additional argument line beyond 100 characters
|
|
374
378
|
E -(ER|ErrorMessage) error message from failed command
|
|
@@ -379,7 +383,7 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
379
383
|
|
|
380
384
|
|
|
381
385
|
3.2.3 Delete Check
|
|
382
|
-
-DL or -Delete, deletes one or multiple 'dscheck' records from
|
|
386
|
+
-DL or -Delete, deletes one or multiple 'dscheck' records from GDEXDB
|
|
383
387
|
for given check indices.
|
|
384
388
|
|
|
385
389
|
dscheck -(DL|Delete)
|
|
@@ -391,23 +395,23 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
391
395
|
|
|
392
396
|
|
|
393
397
|
3.2.4 Unlock Check
|
|
394
|
-
-UL or -UnLockCheck, (Alias: -UnLock), unlocks check records
|
|
395
|
-
commands aborted abnormally during
|
|
398
|
+
-UL or -UnLockCheck, (Alias: -UnLock), unlocks check records whose
|
|
399
|
+
commands aborted abnormally during processing.
|
|
396
400
|
|
|
397
401
|
Process ID and computer hostname are saved in a check record when the recorded
|
|
398
402
|
command is running. If the process aborts abnormally, the PID and hostname may
|
|
399
403
|
sometimes be not cleaned properly. Use this action to clean up the locking
|
|
400
404
|
information to allow the command to be reprocessed or purged.
|
|
401
405
|
|
|
402
|
-
dscheck -(
|
|
403
|
-
|
|
404
|
-
|
|
406
|
+
dscheck -(UL|UnLockCheck)
|
|
407
|
+
-(CI|CheckIndex) CheckIndices
|
|
408
|
+
[-(DB|Debug) DebugModeInfo]
|
|
405
409
|
|
|
406
410
|
It is mandatory to provide a check index to remove lock on a recorded command.
|
|
407
411
|
|
|
408
412
|
|
|
409
413
|
3.3 Check Process Actions
|
|
410
|
-
Delayed mode command executions recorded in
|
|
414
|
+
Delayed mode command executions recorded in GDEXDB are automatically started,
|
|
411
415
|
or restarted, by the common 'dscheck' daemon. Running commands can be interrupted
|
|
412
416
|
at any time and the child processed of the interrupted commands are also cleaned up.
|
|
413
417
|
The status of the current check records can be gathered and emailed to a specialist.
|
|
@@ -454,13 +458,13 @@ DaemonIndex<:>Command<:>Specialist<:>Hostname<:>ProcessLimit<:>Priority<:>
|
|
|
454
458
|
according to the configured information in daemon controls.
|
|
455
459
|
|
|
456
460
|
3.3.2 Interrupt Check
|
|
457
|
-
-IC or -InterruptCheck, interrupts checks
|
|
458
|
-
and also kills recursively all the child processes
|
|
461
|
+
-IC or -InterruptCheck, interrupts checks whose commands are currently running
|
|
462
|
+
and also kills recursively all the child processes running under those
|
|
459
463
|
commands. Locks are cleaned too when the checks are interrupted.
|
|
460
464
|
|
|
461
|
-
dscheck -(
|
|
462
|
-
|
|
463
|
-
|
|
465
|
+
dscheck -(IC|InterruptCheck) [Mode Option]
|
|
466
|
+
-(CI|CheckIndex) CheckIndices
|
|
467
|
+
[-(DB|Debug) DebugModeInfo]
|
|
464
468
|
|
|
465
469
|
Mode option that can be specified for this action:
|
|
466
470
|
-(FI|ForceInterrupt) - it must be present for this action to interrupt a check
|
|
@@ -518,13 +522,13 @@ Use proper Mode options to modify behaviors of Action options. Mode options
|
|
|
518
522
|
are all optional. No value is allowed to be passed in following any Mode option.
|
|
519
523
|
|
|
520
524
|
-AW or -AnyWhere, for Action -AC (-AddCheck), sets Working directory empty in
|
|
521
|
-
|
|
525
|
+
check record to start processing the check anywhere.
|
|
522
526
|
|
|
523
|
-
-BG or -BackGround (Alias: -b), background process. When it
|
|
524
|
-
screen display is turned off for both standard
|
|
525
|
-
|
|
526
|
-
-CP or -CheckPending, if present for Action -PC (-ProcessCheck),
|
|
527
|
-
long pending checks
|
|
527
|
+
-BG or -BackGround (Alias: -b), background process. When it is present,
|
|
528
|
+
screen display is turned off for both standard output and errors.
|
|
529
|
+
|
|
530
|
+
-CP or -CheckPending, if present for Action -PC (-ProcessCheck), checks and kills
|
|
531
|
+
long pending checks.
|
|
528
532
|
|
|
529
533
|
-CS or -CheckStatus, if present, displays detailed status information of the
|
|
530
534
|
recorded commands, including progress percentage for a running command and
|
|
@@ -545,7 +549,7 @@ are all optional. No value is allowed to be passed in following any Mode option.
|
|
|
545
549
|
a given dataset listed for another specialist.
|
|
546
550
|
|
|
547
551
|
-NC or -NoCommand (Alias: -NoRemoteCommand), does not issue remote commands if
|
|
548
|
-
this Mode option is present for action -PC (-
|
|
552
|
+
this Mode option is present for action -PC (-ProcessCheck).
|
|
549
553
|
|
|
550
554
|
-ND or -NewDaemon, a new daemon control can only be added when this Mode option is
|
|
551
555
|
present and daemon control index is given as 0 when Action -SD (-SetDaemon) of
|
|
@@ -558,7 +562,7 @@ are all optional. No value is allowed to be passed in following any Mode option.
|
|
|
558
562
|
-WR or -WithdsRqst, adds requests due to be built or purged of command 'dsrqst'
|
|
559
563
|
for 'dscheck' Action -PC (-ProcessCheck) in non-daemon mode.
|
|
560
564
|
|
|
561
|
-
-WU or -WithdsUpdt, adds due update controls of command '
|
|
565
|
+
-WU or -WithdsUpdt, adds due update controls of command 'dsupdt' for 'dscheck'
|
|
562
566
|
Action -PC (-ProcessCheck) in non-daemon mode.
|
|
563
567
|
|
|
564
568
|
5 INFORMATION OPTIONS
|
|
@@ -593,7 +597,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
593
597
|
action -PC(-ProcessCheck). It defaults to '' to use the local host name. Specify
|
|
594
598
|
PBS to process batch jobs.
|
|
595
599
|
|
|
596
|
-
-MT or -MaxrunTime, specify the
|
|
600
|
+
-MT or -MaxrunTime, specify the maximum run time for daemon mode. It defaults to 0
|
|
597
601
|
for unlimit time. For examples, 5000 means seconds, and 1D means 1 day for 86400
|
|
598
602
|
seconds.
|
|
599
603
|
|
|
@@ -633,7 +637,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
633
637
|
|
|
634
638
|
-CD or -CheckDate, the check date of the recorded command is first processed.
|
|
635
639
|
|
|
636
|
-
-CI or -
|
|
640
|
+
-CI or -CheckIndex, check record indices for commands recorded in GDEXDB. A check
|
|
637
641
|
record is automatically purged if the command is finished.
|
|
638
642
|
|
|
639
643
|
-CM or -Command (Alias: -CommandName), the command name recorded in check record.
|
|
@@ -646,14 +650,14 @@ Information options are used to pass information, one or multiple values, into
|
|
|
646
650
|
single integer value, for example, 1000 means to log debug messages for debug
|
|
647
651
|
levels 1 to 1000; or a range of values, for example, 200-1000 means to log
|
|
648
652
|
debug messages from debug levels 200 to 1000. The default debug file path is
|
|
649
|
-
'$DSSHOME}/dssdb/log' and the default debug file name is 'mydss.dbg'.
|
|
650
|
-
the second and third values for this option to override the
|
|
653
|
+
'${DSSHOME}/dssdb/log' and the default debug file name is 'mydss.dbg'. Provide
|
|
654
|
+
the second and third values for this option to override the defaults
|
|
651
655
|
respectively.
|
|
652
656
|
|
|
653
657
|
-DC or -DoneCount, the number of files that are processed successfully already.
|
|
654
658
|
|
|
655
659
|
-DF or -DownFlags, storage system down flags. The current supported flags are:
|
|
656
|
-
|
|
660
|
+
D-DRDATA, G-GLADE, O-ObjectStore. It can hold multiple flags, up to 5,
|
|
657
661
|
for all down storage systems.
|
|
658
662
|
|
|
659
663
|
-DI or -DaemonIndex, daemon control indices. A daemon control record retains
|
|
@@ -666,7 +670,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
666
670
|
|
|
667
671
|
-EV or -Environments, (Alias: -Envs), specifies environment variables, in form of
|
|
668
672
|
VarName=VarValue and separated by ',', needed to be set to execute a command as
|
|
669
|
-
a batch job. The environment
|
|
673
|
+
a batch job. The environment variables will be set in the batch starting script.
|
|
670
674
|
|
|
671
675
|
-FC or -FileCount, the number of files to be processed by a check command.
|
|
672
676
|
|
|
@@ -685,7 +689,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
685
689
|
option -AO (-ActOption, default to <!>); and different equal sign of single
|
|
686
690
|
value assignment can be provided by Info option -ES, (-EqualSign, default to
|
|
687
691
|
'<=>'). Multi-value assignments can be given in columns delimited with
|
|
688
|
-
separator specified per option -
|
|
692
|
+
separator specified per option -DV (-Divider, default to '<:>'). It starts
|
|
689
693
|
with a column title line for multi-value option names and the rest holds
|
|
690
694
|
values corresponding to each column titles. The value information stops at
|
|
691
695
|
the end of the file or when a new column name line or another single value
|
|
@@ -693,11 +697,11 @@ Information options are used to pass information, one or multiple values, into
|
|
|
693
697
|
additional separator must be appended for each line, including the column
|
|
694
698
|
title line to end lines properly.
|
|
695
699
|
|
|
696
|
-
-MC or -MaxCount, the maximum number of tries
|
|
697
|
-
if the command
|
|
700
|
+
-MC or -MaxCount, the maximum number of tries a check command can be processed
|
|
701
|
+
if the command has failed.
|
|
698
702
|
|
|
699
703
|
-MH or -MatchHost (Alias: -MatchHostname), flag to control hostname match.
|
|
700
|
-
'G' - general match, including
|
|
704
|
+
'G' - general match, including empty hostname specified, or exclusive hostnames
|
|
701
705
|
given but the hostname is not in the list, and 'M' - match only the hostname
|
|
702
706
|
specified is identical to the current host.
|
|
703
707
|
|
|
@@ -718,7 +722,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
718
722
|
-PQ or -PBSQueue, specifies PBS batch queue index the current check to submit to.
|
|
719
723
|
|
|
720
724
|
-QS or -QsubOptions, (Alias: -PBSOptions), specifies options to execute a command
|
|
721
|
-
as a batch job via qsub on PBS nodes. The qsub options must be quoted when
|
|
725
|
+
as a batch job via qsub on PBS nodes. The qsub options must be quoted when presented
|
|
722
726
|
on command line, such as, -QS '-l walltime=12:00:00'.
|
|
723
727
|
|
|
724
728
|
-SN or -Specialist, the specialist who runs the original command. This Info
|
|
@@ -728,7 +732,7 @@ Information options are used to pass information, one or multiple values, into
|
|
|
728
732
|
-ST or -Status, check command status for a check record. Include command
|
|
729
733
|
progress percentage if under process and error message for a failed command.
|
|
730
734
|
|
|
731
|
-
-SZ or -DataSize, the total bytes of data
|
|
735
|
+
-SZ or -DataSize, the total bytes of data processed for the check record.
|
|
732
736
|
|
|
733
737
|
-TC or -TryCount, the number of tries for a check command being processed. It
|
|
734
738
|
is up to the number of tries specified via info option -MC.
|
|
@@ -48,7 +48,7 @@ class PgCheck(PgOPT, PgCMD):
|
|
|
48
48
|
'FI': [0, 'ForceInterrrupt', 0],
|
|
49
49
|
'FO': [0, 'FormatOutput', 0],
|
|
50
50
|
'LO': [0, 'LogOn', 0],
|
|
51
|
-
'MD': [0, '
|
|
51
|
+
'MD': [0, 'MyDataset', 3],
|
|
52
52
|
'NC': [0, 'NoCommand', 0],
|
|
53
53
|
'ND': [0, 'NewDaemon', 0],
|
|
54
54
|
'NT': [0, 'NoTrim', 0],
|
{rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: rda_python_dscheck
|
|
3
|
-
Version: 2.0.
|
|
3
|
+
Version: 2.0.5
|
|
4
4
|
Summary: RDA python package to add and process batch jobs
|
|
5
5
|
Author-email: Zaihua Ji <zji@ucar.edu>
|
|
6
6
|
Project-URL: Homepage, https://github.com/NCAR/rda-python-dscheck
|
|
@@ -14,5 +14,9 @@ License-File: LICENSE
|
|
|
14
14
|
Requires-Dist: rda_python_common
|
|
15
15
|
Dynamic: license-file
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
# dscheck
|
|
18
|
+
|
|
19
|
+
Python project to add and process batch jobs for the [NSF NCAR Geoscience Data Exchange (GDEX)](https://gdex.ucar.edu).
|
|
20
|
+
|
|
21
|
+
The user guide for this utility tool can be viewed at: [User guide](https://gdex-docs-dscheck.readthedocs.io).
|
|
18
22
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/SOURCES.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
{rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/requires.txt
RENAMED
|
File without changes
|
{rda_python_dscheck-2.0.3 → rda_python_dscheck-2.0.5}/src/rda_python_dscheck.egg-info/top_level.txt
RENAMED
|
File without changes
|
|
File without changes
|