rda-python-dsupdt 1.0.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- rda_python_dsupdt/PgUpdt.py +1921 -0
- rda_python_dsupdt/__init__.py +1 -0
- rda_python_dsupdt/dsupdt.py +2454 -0
- rda_python_dsupdt/dsupdt.usg +1784 -0
- rda_python_dsupdt-1.0.1.dist-info/LICENSE +21 -0
- rda_python_dsupdt-1.0.1.dist-info/METADATA +15 -0
- rda_python_dsupdt-1.0.1.dist-info/RECORD +10 -0
- rda_python_dsupdt-1.0.1.dist-info/WHEEL +5 -0
- rda_python_dsupdt-1.0.1.dist-info/entry_points.txt +2 -0
- rda_python_dsupdt-1.0.1.dist-info/top_level.txt +1 -0
|
@@ -0,0 +1,1784 @@
|
|
|
1
|
+
|
|
2
|
+
1 INTRODUCTION
|
|
3
|
+
|
|
4
|
+
Program 'dsupdt' is a utility for periodic dataset updates by calling 'dsarch'
|
|
5
|
+
to archive data onto CISL Research Data Archive (RDA) Servers.
|
|
6
|
+
|
|
7
|
+
This application is implemented to:
|
|
8
|
+
- Configure update information of controls/local files/remote files for a dataset
|
|
9
|
+
- Download/retrieve data files on remote servers or other local machines
|
|
10
|
+
- Call specialist-defined routines to validate/convert the downloaded remote
|
|
11
|
+
data files
|
|
12
|
+
- Build local files based on the remote files, tar and/or compress actions
|
|
13
|
+
are automatically applied if necessary
|
|
14
|
+
- Archive local files onto RDA Servers by calling 'dsarch'
|
|
15
|
+
- Clean the local files and/or other temporary files after the data files
|
|
16
|
+
are archived successfully
|
|
17
|
+
- Check availability of files on remote servers
|
|
18
|
+
- Start update actions via a centralized daemon 'dscheck', via cron job,
|
|
19
|
+
or manually
|
|
20
|
+
|
|
21
|
+
A dataset must be set to work with 'dsarch' before it can be configured for
|
|
22
|
+
periodic update by 'dsupdt'. Check help document of 'dsarch' for information
|
|
23
|
+
of setting dataset under control of 'dsarch'.
|
|
24
|
+
|
|
25
|
+
With update information being configured into RDADB for data files, 'dsupdt'
|
|
26
|
+
downloads and archives data as in the following steps:
|
|
27
|
+
Server Files on remote server or at other local area =========DOWNLOAD/COPY=>
|
|
28
|
+
Remote Files at local working area ========================VALIDATE/CONVERT=>
|
|
29
|
+
Remote Files ready for building Local Files =============TAR/COMPRESS/BUILD=>
|
|
30
|
+
Local Files ready for archive =======================================DSARCH=>
|
|
31
|
+
Files on RDA Servers
|
|
32
|
+
|
|
33
|
+
If not given explicitly, a server file name is the same as remote file name.
|
|
34
|
+
For cases of one-to-one matching between remote files and local files, a
|
|
35
|
+
remote file name defaults to given local file name if it is not specified.
|
|
36
|
+
|
|
37
|
+
A local file update record is the minimal configuration to update data files
|
|
38
|
+
for a given dataset. Add remote file configuration records for situations that
|
|
39
|
+
the remote file name is different from the local file name or multiple remote
|
|
40
|
+
files are involved for building a single local file.
|
|
41
|
+
|
|
42
|
+
Set update control records to start update actions on, or not on, specified
|
|
43
|
+
data process computers, or on all available ones. Without update control
|
|
44
|
+
record, an update action can be started routinely via cron jabs manually on
|
|
45
|
+
a specified computer.
|
|
46
|
+
|
|
47
|
+
To prevent a DSS specialist from executing 'dsupdt' accidentally to update
|
|
48
|
+
data files via update configuration records owned by other specialists, only
|
|
49
|
+
the specialist who owns a update record can execute 'dsupdt' against it to
|
|
50
|
+
update data. Validation procedures are also added to prevent DSS specialists
|
|
51
|
+
from setting up 'dsupdt' configuration on a wrong dataset. If an input file is
|
|
52
|
+
used to hold option information for setting up update configuration of a given
|
|
53
|
+
dataset, the input file name must start with the dataset number, as in format of
|
|
54
|
+
'dsnnn.n.*'; the wildcard '*' matches one or multiple characters valid for file
|
|
55
|
+
names. Another validation is to check if a specialist is allowed to execute
|
|
56
|
+
'dsupdt' on a given dataset. Action stops if a specialist is not listed as
|
|
57
|
+
a owner, unless a Mode option -MD (-MyDataset) is present. The option -MD tells
|
|
58
|
+
'dsupdt' to go ahead finish an Action forcefully.
|
|
59
|
+
|
|
60
|
+
Update procedure of an individual data file is skipped if error occurs during
|
|
61
|
+
updating; but execution of 'dsupdt' continues to finish all the other available
|
|
62
|
+
updates. Restarting of 'dsupdt' picks up where it was left for the update files
|
|
63
|
+
unfinished previously. An email notice is sent to the specialist who starts
|
|
64
|
+
update actions of 'dsupdt'. As default, the email includes detail information
|
|
65
|
+
of successful updates, and error messages if any.
|
|
66
|
+
|
|
67
|
+
In the following sections, general usage of 'dsupdt' is described first; and
|
|
68
|
+
detail descriptions of different options are given next; and examples are
|
|
69
|
+
interspersed through out the document.
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
2 GENERAL DSUPDT USAGE
|
|
73
|
+
|
|
74
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] [Action Option] [Mode Options] [Info Options]
|
|
75
|
+
or
|
|
76
|
+
dsupdt [-(IF|InputFile)] InputFileNames
|
|
77
|
+
|
|
78
|
+
Quotes [] indicate optional. A pipeline '|' in parentheses as in format (A|B)
|
|
79
|
+
means either A or B can be used. The options applied to 'dsupdt' are divided
|
|
80
|
+
into three categories, Action, Mode, and Information (Info for short) options.
|
|
81
|
+
Action options are used to specify what tasks this utility to execute; Mode
|
|
82
|
+
options are used to modify behaviors of the actions; and the Info options
|
|
83
|
+
are used to pass information, one or multiple values, to run 'dsupdt'. An option
|
|
84
|
+
can be given in form of either short name or long name, -DS or -Dataset for
|
|
85
|
+
example. Some options have alias names for convenience. Option names can be
|
|
86
|
+
given in either upper or lower cases, while the values following Info options
|
|
87
|
+
are case sensitive.
|
|
88
|
+
|
|
89
|
+
Option -DS is an Info option which is used to specify a dataset number. If a
|
|
90
|
+
dataset number is given as the first option following 'dsupdt' the option name
|
|
91
|
+
-DS (-Dataset) can be omitted. Some actions are allowed to be executed without
|
|
92
|
+
dataset number, which means that the actions are applied on all available update
|
|
93
|
+
control or file records of different datasets.
|
|
94
|
+
|
|
95
|
+
Specify one of the Action options each time to execute 'dsupdt'. Based on what
|
|
96
|
+
Action is chosen, some of the Info options are mandatory and others are optional
|
|
97
|
+
and certain Mode options can be applied to alter the behaviors of the Actions.
|
|
98
|
+
|
|
99
|
+
All options, except Info option -IF (-InputFile), can be given either on command
|
|
100
|
+
line or in input files. Input file names are presented per Info option -IF and
|
|
101
|
+
can only be provided on command line. Check description of Info option -IF for
|
|
102
|
+
detail on how to present options in input files. One or multiple input files,
|
|
103
|
+
combined with options on command line, are allowed to run 'dsupdt'. The option
|
|
104
|
+
name, -IF (-InputFile), itself can be omitted if a single input file is given on
|
|
105
|
+
command line, and action and additional option information are all provided
|
|
106
|
+
inside the input file.
|
|
107
|
+
|
|
108
|
+
When information of update controls/local files/remote files are retrieved,
|
|
109
|
+
Info options are used to specify conditions for querying information from
|
|
110
|
+
RDADB. Some special signs can be used to further confine the information with
|
|
111
|
+
special and complicated conditions; they are '!', '<', '>' and '<>'. These
|
|
112
|
+
special signs, if provided on command line, must be quoted or escaped to avoid
|
|
113
|
+
of being interpreted by computer OS. The '!', or \!, means exclusion to the
|
|
114
|
+
following value(s) and it must be the first item following an Info option name,
|
|
115
|
+
while '<' or '>' mean greater or less than the following value, and '<>' means
|
|
116
|
+
condition between the following two values. Combine '!' and '<', as syntax "'!'
|
|
117
|
+
'<' OptionValue", to make a condition of 'large than or equal to OptionValue'.
|
|
118
|
+
|
|
119
|
+
Description of an individual option is displayed if 'dsupdt' is issued on
|
|
120
|
+
command line as
|
|
121
|
+
|
|
122
|
+
dsupdt [Option] -(h|help) [Option]
|
|
123
|
+
|
|
124
|
+
A description is displayed for an option given either before or after -(h|help).
|
|
125
|
+
If no option is specified or 'dsupdt' is issued by itself, this whole document
|
|
126
|
+
is displayed per UNIX utility 'more'. A hard copy of this help document can be
|
|
127
|
+
printed from the saved file: $DSSHOME/dssdb/prog_usage/dsupdt.usg; $DSSHOME is
|
|
128
|
+
an environment variable for DSS home directory setup on individual computers.
|
|
129
|
+
|
|
130
|
+
#The online HTML version of this document is available at
|
|
131
|
+
#http://dss.ucar.edu/internal/docs/dsupdt/
|
|
132
|
+
|
|
133
|
+
|
|
134
|
+
3 ACTION OPTIONS
|
|
135
|
+
|
|
136
|
+
Action options are used to specify what tasks 'dsupdt' executes. No values
|
|
137
|
+
are allowed to follow Action options. Multiple tasks may be processed with a
|
|
138
|
+
single execution of 'dsupdt' depending on what Action option is chosen. Some of
|
|
139
|
+
the comprehensive actions include automatically other simpler actions as
|
|
140
|
+
default; and others include additional actions when certain Mode options are
|
|
141
|
+
present. Execute 'dsupdt' for one Action as a time, and multiple Action options
|
|
142
|
+
provided simultaneously are blocked.
|
|
143
|
+
|
|
144
|
+
Some actions are setting information into and some getting information from
|
|
145
|
+
RDADB for a given dataset, and others are allowed to work on multiple datasets
|
|
146
|
+
if dataset number is omitted.
|
|
147
|
+
|
|
148
|
+
Based on the information being manipulated, the actions are divided into three
|
|
149
|
+
categories:
|
|
150
|
+
Configuration Actions - create, delete, modify and retrieve information of
|
|
151
|
+
update controls, local files and remote files in RDADB,
|
|
152
|
+
of a specified dataset
|
|
153
|
+
All Information Actions - modify and retrieve information of update controls,
|
|
154
|
+
local files and remote files of a specified dataset
|
|
155
|
+
all together
|
|
156
|
+
Update Actions - download remote files, build local files, archive local
|
|
157
|
+
files onto RDA Servers, clean the temporary data files,
|
|
158
|
+
check update status of files on remote servers, and
|
|
159
|
+
unlock update controls or local files that are locked
|
|
160
|
+
by aborted update processes.
|
|
161
|
+
|
|
162
|
+
3.1 Configuration Actions
|
|
163
|
+
Update control, local file and remote file information can be created,
|
|
164
|
+
modified, viewed or deleted via Actions in this section:
|
|
165
|
+
Set Update Controls - create or modify information of update control
|
|
166
|
+
records for a given dataset
|
|
167
|
+
Get Update Controls - retrieve information of update control records
|
|
168
|
+
for one or all datasets
|
|
169
|
+
Set Local Files - create or modify information of local file records
|
|
170
|
+
for a given dataset
|
|
171
|
+
Get Local Files - retrieve information of local file records
|
|
172
|
+
for one or all datasets
|
|
173
|
+
Set Remote Files - create or modify information of remote file records
|
|
174
|
+
for a given dataset
|
|
175
|
+
Get Remote Files - retrieve information of remote file records
|
|
176
|
+
for one or all datasets
|
|
177
|
+
Delete Update Information - delete update control and local/remote file records
|
|
178
|
+
|
|
179
|
+
3.1.1 Set Update Controls
|
|
180
|
+
-SC or -SetControl (Alias: -SetUpdateControl), creates or modifies update
|
|
181
|
+
control information into RDADB for a given dataset. These configuration records
|
|
182
|
+
are used to schedule data update actions and the actions are automatically
|
|
183
|
+
started via a centralized daemon 'dscheck'. One or multiple control records can
|
|
184
|
+
be set each time for a given dataset.
|
|
185
|
+
|
|
186
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(SC|SetControl) [Mode Options]
|
|
187
|
+
-(CI|ControlIndex) UpdateControlIndices
|
|
188
|
+
[-(ID|ControlID) ControlIdString]
|
|
189
|
+
[-(SN|Specialist) DSSSpecialist]
|
|
190
|
+
[-(PI|ParentIndex) ParentControlIndedices]
|
|
191
|
+
[-(AN|ActionName) DSUPDTActionNames] # UF, DR, BL, PB, CL, CU
|
|
192
|
+
[-(FQ|Frequency) UpdateControlFrequencies] # i.e., 1W, 1M
|
|
193
|
+
[-(CO|ControlOffset) UpdateControlOffsets]
|
|
194
|
+
[-(CT|ControlTime) UpdateControlTimes]
|
|
195
|
+
[-(RI|RetryInterval) ControlRetryInterval] # i.e., 1D, 1D3H, 1M
|
|
196
|
+
[-(VI|ValidInterval) UpdateValidInterval] # i.e., 1D, 2D, 1M
|
|
197
|
+
[-(UC|UpdateControl) UpdateControlOptions]
|
|
198
|
+
[-(MC|EMailControl) EmailControlOptions]
|
|
199
|
+
[-(EC|ErrorControl) ErrorControlOptions]
|
|
200
|
+
[-(KF|KeepFile) KeepFileOptions]
|
|
201
|
+
[-(HO|HourOffset) TimeZoneOffsets]
|
|
202
|
+
[-(DT|DataTime) DataUpdatedTimes]
|
|
203
|
+
[-(HN|HostName) HostNames]
|
|
204
|
+
[-(XC|ExecuteCoomand) CoomandAfterUpdate]
|
|
205
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
206
|
+
[-(DB|Debug) DebugModeInfo]
|
|
207
|
+
|
|
208
|
+
Mode option that can be specified for setting local file action:
|
|
209
|
+
-(MD|MyDataset) - sets information into RDADB no matter the specialist who
|
|
210
|
+
runs 'dsupdt' owns the dataset or not
|
|
211
|
+
-(NC|NewControl) - sets a new update control record into RDADB
|
|
212
|
+
|
|
213
|
+
If information of an update control exists already in RDADB, the control record
|
|
214
|
+
is modified. A new update control record is added if control index is 0 and Mode
|
|
215
|
+
option -NC (-NewControl) is present. A dataset name via Info option -DS is
|
|
216
|
+
only mandatory when a new control record is added.
|
|
217
|
+
|
|
218
|
+
For example, set three new update control local files of d277000 via
|
|
219
|
+
input file 'd277000.cntl'
|
|
220
|
+
|
|
221
|
+
dsupdt SC -NC -IF d277000.cntl
|
|
222
|
+
|
|
223
|
+
<<Content of input file d277000.cntl>>
|
|
224
|
+
Dataset<=>d277000
|
|
225
|
+
ControlIndex<:>Specialist<:>ParentIndex<:>Action<:>Frequency<:>ControlOffset<:>ControlTime<:>RetryInterval<:>UpdateControl<:>EMailControl<:>ErrorControl<:>KeepFile<:>HourOffset<:>
|
|
226
|
+
0<:>zji<:>0<:>UF<:>1W<:>1D9H<:>2011-10-24 09:00:00<:>12H<:><:>A<:>N<:>N<:>0<:>
|
|
227
|
+
0<:>zji<:>0<:>UF<:>1M<:>1D15H<:>2011-11-02 15:00:00<:>1D<:><:>A<:>N<:>N<:>0<:>
|
|
228
|
+
0<:>zji<:>0<:>UF<:>1M<:>2D15H<:>2011-11-03 15:00:00<:>1D<:><:>A<:>N<:>N<:>0<:>
|
|
229
|
+
|
|
230
|
+
3.1.2 Get Update Controls
|
|
231
|
+
-GC or -GetControl (Alias: -GetUpdateControl), retrieves update control
|
|
232
|
+
information for a given dataset.
|
|
233
|
+
|
|
234
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(GC|GetControl) [Mode Option]
|
|
235
|
+
[-(FN|FieldNames) FieldNameString]
|
|
236
|
+
[-(CI|ControlIndex) UpdateControlIndices]
|
|
237
|
+
[-(ID|ControlID) ControlIdString]
|
|
238
|
+
[-(SN|Specialist) DSSSpecialist]
|
|
239
|
+
[-(PI|ParentIndex) ParentControlIndedices]
|
|
240
|
+
[-(AN|ActionName) DSUPDTActionNames] # UF, DR, BL, PB, CL, CU
|
|
241
|
+
[-(FQ|Frequency) UpdateControlFrequencies] # i.e., 1W, 1M
|
|
242
|
+
[-(CO|ControlOffset) UpdateControlOffsets]
|
|
243
|
+
[-(CT|ControlTime) UpdateControlTimes]
|
|
244
|
+
[-(OF|OutputFile) OutputFileName]
|
|
245
|
+
[-(DB|Debug) DebugModeInfo]
|
|
246
|
+
|
|
247
|
+
Mode option that can be specified for getting update control Action:
|
|
248
|
+
-(FO|FormatOutput) - format the column output with a fix width for all values
|
|
249
|
+
of a given field
|
|
250
|
+
|
|
251
|
+
Use Info option -FN (-FieldNames) to specify what control fields to be
|
|
252
|
+
retrieved. It defaults to 'CLNPAFOTRVUJEKZ' if option -FN is not provided.
|
|
253
|
+
Update information of all available fields is retrieved if -FN ALL is given.
|
|
254
|
+
|
|
255
|
+
Valid field names of update controls and their corresponding Info option names:
|
|
256
|
+
|
|
257
|
+
Names Info Options Descriptions
|
|
258
|
+
C -(CI|ControlIndex) index of update control record
|
|
259
|
+
L -(ID|ControlID) descriptive id for a control record
|
|
260
|
+
N -(SN|Specialist) DSS specialist who owns the control record
|
|
261
|
+
P -(PI|ParentIndex) parent index, refers to a dcupdt.cindex if not 0
|
|
262
|
+
A -(AN|ActionName) Update action name for 'dsupdt'
|
|
263
|
+
F -(FQ|Frequency) frequency of update control, i.e., 1W, 1M
|
|
264
|
+
O -(CO|ControlOffset) time offset for next process, i.e. 2D10H, 3H15N
|
|
265
|
+
T -(CT|ControlTime) time to run dsupdt, format: YYY-MM-DD HH:NN:SS
|
|
266
|
+
R -(RI|RetryInterval) retry interval for failed dsupdt, i.e. 1M2D, 2H
|
|
267
|
+
V -(VI|ValidInterval) update valid interval, i.e., 2D, 1M
|
|
268
|
+
U -(UC|UpdateControl) any of B-Begin,C-Current,E-Reset End Date/Time,
|
|
269
|
+
F-Future,G-GMT,M-Multi,N-New,O-Miss,
|
|
270
|
+
Y-Skip Feb. 29,Z-Allow Zero Filesize
|
|
271
|
+
J -(MC|EMailControl) one of A-All, E-Error, N-No, S-Summary, B-Summary when error
|
|
272
|
+
E -(EC|ErrorControl) one of I-Ignore Error, Q-Quit on Error, N-Neither
|
|
273
|
+
K -(KF|KeepFile) one of S-Server File, R-Remote file, B-Keep Both, N-Neither
|
|
274
|
+
Z -(HO|HourOffsset) time zone hour offset, hours ahead of local time
|
|
275
|
+
D -(DT|DataTime) time data updated to, format: YYY-MM-DD HH:NN:SS
|
|
276
|
+
H -(HN|HostName) hostnames this cindex can/cannot be processed on
|
|
277
|
+
S -(SB|SBatchOptions) additional PBS batch options
|
|
278
|
+
Q -(QS|QSubOptions) additional SLURM batch options
|
|
279
|
+
Y -(CC|CarbonCopy) carbon copies for additional email addresses
|
|
280
|
+
X -(XC|ExecuteCommand) Command to be executed after successful update
|
|
281
|
+
|
|
282
|
+
If dataset number is missed, update control information is displayed for all
|
|
283
|
+
available control records, of different datasets, owned by the specialist who
|
|
284
|
+
runs 'dsupdt'.
|
|
285
|
+
|
|
286
|
+
For example, to get the update control information of d277000 for default
|
|
287
|
+
fields
|
|
288
|
+
|
|
289
|
+
dsupdt d277000 GC
|
|
290
|
+
|
|
291
|
+
<<Content of the output>>
|
|
292
|
+
Dataset<=>d277000
|
|
293
|
+
ControlIndex<:>Specialist<:>ParentIndex<:>Action<:>Frequency<:>ControlOffset<:>ControlTime<:>RetryInterval<:>UpdateControl<:>EMailControl<:>ErrorControl<:>KeepFile<:>HourOffset<:>
|
|
294
|
+
1<:>zji<:>0<:>UF<:>1W<:>1D9H<:>2011-10-24 09:00:00<:>12H<:><:>A<:>N<:>N<:>0<:>
|
|
295
|
+
2<:>zji<:>0<:>UF<:>1M<:>1D15H<:>2011-11-02 15:00:00<:>1D<:><:>A<:>N<:>N<:>0<:>
|
|
296
|
+
3<:>zji<:>0<:>UF<:>1M<:>2D15H<:>2011-11-03 15:00:00<:>1D<:><:>A<:>N<:>N<:>0<:>
|
|
297
|
+
|
|
298
|
+
3.1.3 Set Local Files
|
|
299
|
+
-SL or -SetLocalFile (Alias: -SetLocal), creates or modifies update information
|
|
300
|
+
of local files into RDADB for a given dataset. One or multiple records can be
|
|
301
|
+
processed each time.
|
|
302
|
+
|
|
303
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(SL|SetLocalFile) [Mode Options]
|
|
304
|
+
-(LI|LocalIndex) LocalFileIndices
|
|
305
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
306
|
+
[-(CI|ControlIndex) UpdateControlIndices]
|
|
307
|
+
[-(PI|ParentIndex) ParentLocalIndedices]
|
|
308
|
+
[-(FA|FileArchived) ArchivedFileNames] # auto generated if missed
|
|
309
|
+
[-(DC|DownloadCommand) DownloadCommand]
|
|
310
|
+
[-(AN|ActionName) DSARCHActionNames] # AS, AW or AQ
|
|
311
|
+
[-(OP|Options) DSARCHOptions]
|
|
312
|
+
[-(XO|ExecOrder) ExecuteOrderIndices]
|
|
313
|
+
[-(FQ|Frequency) DataFrequencies] # i.e., 1W, 1M
|
|
314
|
+
[-(EP|EndPeriod) PerirodEndAt] # 0=Sunday/EndMonth if FQ=1W/1M
|
|
315
|
+
[-(DI|DueInternval) DataDueInterval] # i.e., 1M2D
|
|
316
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
317
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
318
|
+
[-(VI|ValidInterval) UpdateValidInterval] # i.e., 1D, 2D, 1M
|
|
319
|
+
[-(AT|AgeTime) RemoteFileAgeTime] # i.e., 1D, 2D, 1M
|
|
320
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
321
|
+
[-(CL|CleanCommand) FileCleaningCommand]
|
|
322
|
+
[-(MR|MissRemote) AllowMissRemote] # 'Y' or 'N'
|
|
323
|
+
[-(PR|ProcessRemote) AdditionalRemoteFilePrecess] # user defined
|
|
324
|
+
[-(BC|BuildCommand) CoomandBuildLocalFile] # user defined
|
|
325
|
+
[-(SN|Specialist) DSSSpecialist]
|
|
326
|
+
[-(DE|Description) FileDescription] # note, include temporal patterns
|
|
327
|
+
[-(DB|Debug) DebugModeInfo]
|
|
328
|
+
|
|
329
|
+
Mode option that can be specified for setting local file action:
|
|
330
|
+
-(MD|MyDataset) - sets information into RDADB no matter the specialist who
|
|
331
|
+
runs 'dsupdt' owns the dataset or not
|
|
332
|
+
-(NL|NewLocfile) - sets a new update record into RDADB
|
|
333
|
+
-(RO|ResetOrder) - resets the executing order indices of the local file
|
|
334
|
+
records according to the order as they are given.
|
|
335
|
+
|
|
336
|
+
If information of a local file exists already in RDADB for a given local file
|
|
337
|
+
index, the update record is modified. A new local file record is added if
|
|
338
|
+
local file index is 0 and Mode option -NL (-NewLocfile) is present. A dataset
|
|
339
|
+
name via Info option -DS is only mandatory when a new control record is added.
|
|
340
|
+
|
|
341
|
+
Patterns quoted by delimiters can be set in local file names per -LF
|
|
342
|
+
(-LocalFile), remote file names per -RF (-RemoteFile), download command per
|
|
343
|
+
-DC (-DownloadCommand) and archive file note per -DE (-Description). The
|
|
344
|
+
default delimiters are '<' and '>', which can be set differently per Info
|
|
345
|
+
option -PD (-PatternDelimiter). A pattern is use for place holder of
|
|
346
|
+
temporal or generic matching information of data files and later can be
|
|
347
|
+
dynamically replaced with given values during update actions. The pattern
|
|
348
|
+
<YYYYMM> is a place holder for a 4-digital year and a 2-digital month. For
|
|
349
|
+
example, a local file name is given as 'filename.<YYYYMM>.ext', and the year
|
|
350
|
+
and month are replaced by year value 2007 and month value 10 if the data file
|
|
351
|
+
is updated to end date of 2007-10-31. Replacing <YYYYMM> with 200710 to get
|
|
352
|
+
the local file name as 'filename.200710.ext'.
|
|
353
|
+
|
|
354
|
+
An local file configuration for a dataset is linked to an existing update
|
|
355
|
+
control record for the same dataset, if the Control Index is set into
|
|
356
|
+
the local file record via Info option -CI (-ControlIndex). With this setup,
|
|
357
|
+
the update action set in the update control record is scheduled to process
|
|
358
|
+
automatically against this local file configuration.
|
|
359
|
+
|
|
360
|
+
For example, set update information of two new local files of d744004 via
|
|
361
|
+
input file 'd744004.loc'
|
|
362
|
+
|
|
363
|
+
dsupdt SL -NL -IF d744004.loc
|
|
364
|
+
|
|
365
|
+
<<Content of input file d744004.loc>>
|
|
366
|
+
Dataset<=>d744004
|
|
367
|
+
LocalIndex<:>LocalFile<:>Action<:>ExecOrder<:>ControlIndex<:>FileArchived<:>DownloadCommand<:>Options<:>Frequency<:>DueInterval<:>EndDate<:>ValidInterval<:>WorkDir<:>CleanCommand<:>
|
|
368
|
+
0<:>uv.<YYYYMM>.bln<:>AW<:>1<:>0<:><:>cp -p /huron/ftp/rossby/upload/morzel/<:>-GI 11 -DF BINARY<:>1M<:><:>2009-08-31<:><:>$UPDTDATA/zji/BLNWIND<:>rm -f __FN__<:>
|
|
369
|
+
0<:>curl.<YYYYMM>.bln<:>AB<:>2<:>0<:><:>cp -p /huron/ftp/rossby/upload/morzel/<:>-GI 12 -DF BINARY<:>1M<:><:>2009-08-31<:><:>$UPDTDATA/zji/BLNWIND<:>rm -f __FN__<:>
|
|
370
|
+
|
|
371
|
+
3.1.4 Get Local Files
|
|
372
|
+
-GL or -GetLocalFile (Alias: -GetLocal), retrieves local file update
|
|
373
|
+
information for a given dataset. Update information of specified local
|
|
374
|
+
files are retrieved if the local file indices or names are provided.
|
|
375
|
+
|
|
376
|
+
dsupdt [[-(DS|Dataset] dsnnn.n] -(GL|GetLocalFile) [Mode Option]
|
|
377
|
+
[-(FN|FieldNames) FieldNameString]
|
|
378
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
379
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
380
|
+
[-(AN|ActionName) DSARCHActionName]
|
|
381
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
382
|
+
[-(FQ|Frequency) DataFrequencies] # i.e., 1W, 1M
|
|
383
|
+
[-(OF|OutputFile) OutputFileName]
|
|
384
|
+
[-(DB|Debug) DebugModeInfo]
|
|
385
|
+
|
|
386
|
+
Mode option that can be specified for getting local file Action:
|
|
387
|
+
-(FO|FormatOutput) - format the column output with a fix width for all values
|
|
388
|
+
of a given field
|
|
389
|
+
|
|
390
|
+
Use Info option -FN (-FieldNames) to specify what local file fields to be
|
|
391
|
+
retrieved. It defaults to 'LFAIUCOQJNVWZ' if option -FN is not provided.
|
|
392
|
+
Update information of all available fields is retrieved if -FN ALL is given.
|
|
393
|
+
|
|
394
|
+
Valid field names of local files and their corresponding Info option names:
|
|
395
|
+
|
|
396
|
+
Names Info Options Descriptions
|
|
397
|
+
L -(LI|LocalIndex) index of update record
|
|
398
|
+
F -(LF|LocalFile) local, original or source file names
|
|
399
|
+
A -(AN|ActionName) Action name for 'dsarch'
|
|
400
|
+
I -(CI|ControlIndex) update control indices
|
|
401
|
+
P -(PI|ParentIndex) parent index, refers to a dlupdt.lindex if not 0
|
|
402
|
+
U -(FA|FileArchived) archived file names on RDA Servers
|
|
403
|
+
C -(DC|DownloadCommand) command for download Remote file
|
|
404
|
+
O -(OP|Options) options following the Action
|
|
405
|
+
X -(XO|ExecOrder) order indices for update execution
|
|
406
|
+
Q -(FQ|Frequency) frequency of data files, i.e., 1W, 1M
|
|
407
|
+
E -(EP|EndPeriod) 0=Sunday/EndMonth if FQ=1W/1M
|
|
408
|
+
J -(ED|EndDate) data end date of next update
|
|
409
|
+
K -(EH|EndHour) data end hour of next update
|
|
410
|
+
N -(DI|DueInterval) data next due period, i.e., 1M2D
|
|
411
|
+
V -(VI|ValidInterval) update valid interval, i.e., 2D, 1M
|
|
412
|
+
T -(AT|AgeTime) remote file age interval, i.e., 2D, 1M
|
|
413
|
+
W -(WD|WorkDir) working directory for processing update
|
|
414
|
+
Z -(CL|CleanCommand) command for removing data files after update
|
|
415
|
+
M -(MR|MissRemote) flag, Y/N, if allow missing one of remote files
|
|
416
|
+
R -(PR|ProcessRemote) external utility to process remote files
|
|
417
|
+
B -(BC|BuildCommand) external utility to build local files
|
|
418
|
+
S -(SN|Specialist) DSS specialist who owns the local file record
|
|
419
|
+
D -(DE|Description) File description, include temporal patterns
|
|
420
|
+
|
|
421
|
+
Local file update information can be retrieved for specified local file
|
|
422
|
+
names per option -LF (-LocalFile). Info option -LF accepts wildcard input
|
|
423
|
+
of '%' for matching any number of characters.
|
|
424
|
+
|
|
425
|
+
If dataset number is missed, update information is displayed for all available
|
|
426
|
+
local files, of different datasets, owned by the specialist who runs 'dsupdt'.
|
|
427
|
+
|
|
428
|
+
For example, to get the local file information of d744004 for default fields for
|
|
429
|
+
local file indices 33 and 34
|
|
430
|
+
|
|
431
|
+
dsupdt d744004 GL -LI 33 34
|
|
432
|
+
|
|
433
|
+
<<Content of the output>>
|
|
434
|
+
Dataset<=>d744004
|
|
435
|
+
LocalIndex<:>LocalFile<:>Action<:>ExecOrder<:>ControlIndex<:>FileArchived<:>DownloadCommand<:>Options<:>Frequency<:>DueInterval<:>EndDate<:>ValidInterval<:>WorkDir<:>CleanCommand<:>
|
|
436
|
+
33<:>uv.<YYYYMM>.bln<:>AW<:>1<:>0<:><:>cp -p /huron/ftp/rossby/upload/morzel/<:>-GI 11 -DF BINARY<:>1M<:><:>2009-08-31<:><:>$UPDTDATA/zji/BLNWIND<:>rm -f __FN__<:>
|
|
437
|
+
34<:>curl.<YYYYMM>.bln<:>Aw<:>2<:>0<:><:>cp -p /huron/ftp/rossby/upload/morzel/<:>-GI 12 -DF BINARY<:>1M<:><:>2009-08-31<:><:>$UPDTDATA/zji/BLNWIND<:>rm -f __FN__<:>
|
|
438
|
+
|
|
439
|
+
3.1.5 Set Remote Files
|
|
440
|
+
-SR or -SetRemoteFile (Alias: -SetRemote), creates and modifies update
|
|
441
|
+
information of remote files into RDADB for given dataset and local
|
|
442
|
+
file(s). One or multiple remote files can be processed each time.
|
|
443
|
+
|
|
444
|
+
dsupdt [-(DS|Dataset)] dsnnn.n -(SR|SetRemoteFile) [Mode Option]
|
|
445
|
+
-(LI|LocalIndex) LocalFileIndices
|
|
446
|
+
-(RF|RemoteFile) RemoteFileNames
|
|
447
|
+
[-(DO|DownloadOrder) DownloadOrderIndecies]
|
|
448
|
+
[-(DC|DownloadCommand) DownloadCommand] # override the one for
|
|
449
|
+
[-(SF|ServerFile) FileNamesOnRemoteServer]
|
|
450
|
+
[-(BT|BeginTime) BeginningDay/Hour] # ':' delimited for multiple ones
|
|
451
|
+
[-(ET|EndTime) EndingDay/Hour] # ':' delimited for multiple ones
|
|
452
|
+
[-(TI|TimeInterval) TimeInterval] # i.e., 2H, 5D
|
|
453
|
+
[-(DB|Debug) DebugModeInfo]
|
|
454
|
+
|
|
455
|
+
Mode option that can be specified for this action include:
|
|
456
|
+
-(MD|MyDataset) - sets information into RDADB no matter the specialist who
|
|
457
|
+
runs 'dsupdt' owns the dataset or not
|
|
458
|
+
|
|
459
|
+
Set update information of remote files only when they are needed. For
|
|
460
|
+
simple situation that remote and local file names are identical, information
|
|
461
|
+
of the local file record is enough to process data update. A remote file
|
|
462
|
+
update record is only required if the remote file is different from local
|
|
463
|
+
file, including situations that multiple remote files are downloaded and
|
|
464
|
+
tarred for a single local file, or a single remote file may be available from
|
|
465
|
+
multiple servers.
|
|
466
|
+
|
|
467
|
+
Local file indices and remote file names must be provided per options -LI
|
|
468
|
+
(-LocalIndex) and -RF (-RemoteFile), respectively, for this action to work.
|
|
469
|
+
|
|
470
|
+
If a download command is specified per Info option -DC (-DownloadCommand) in
|
|
471
|
+
the remote file record, it is used for downloading remote file rather than
|
|
472
|
+
the one specified in the local file record.
|
|
473
|
+
|
|
474
|
+
Set file name on the remote server per Info option -SF (-ServerFile) if it
|
|
475
|
+
is different from the remote file name. If a server file name is not
|
|
476
|
+
set for a remote file record, it is assumed to be the same as the remote
|
|
477
|
+
file name.
|
|
478
|
+
|
|
479
|
+
Data files downloaded remotely or generated via local processes are temporarily
|
|
480
|
+
staged on a local disk. They are called remote files until they are further
|
|
481
|
+
validated and converted into local files.
|
|
482
|
+
|
|
483
|
+
Set download order indices per Info option -DO (-DownloadOrder) if a
|
|
484
|
+
remote file is accessible from multiple remote servers. Remote file download
|
|
485
|
+
action tries the web server with the lowest download index first; and, if
|
|
486
|
+
fails, it tries the other servers with higher indices until successful
|
|
487
|
+
download.
|
|
488
|
+
|
|
489
|
+
3.1.6 Get Remote Files
|
|
490
|
+
-GR or -GetRemoteFile (Alias: -GetRemote), gets remote file update
|
|
491
|
+
information for given condition.
|
|
492
|
+
|
|
493
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(GR|GetRemotefile) [Mode Option]
|
|
494
|
+
[-(FN|FieldNames) FieldNameString]
|
|
495
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
496
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
497
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
498
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
499
|
+
[-(SF|ServerFile) ServerFileNames]
|
|
500
|
+
[-(OF|OutputFile) OutputFileName]
|
|
501
|
+
[-(DB|Debug) DebugModeInfo]
|
|
502
|
+
|
|
503
|
+
Mode option that can be specified for getting remote file Action:
|
|
504
|
+
-(FO|FormatOutput) - format the column output with a fix width for all values
|
|
505
|
+
of a given field
|
|
506
|
+
|
|
507
|
+
Use Info option -FN (-FieldNames) to specify what remote file fields to
|
|
508
|
+
retrieve. It defaults to 'LFDSCBET' if option -FN is not provided.
|
|
509
|
+
|
|
510
|
+
Valid remote file field names and their corresponding Info options:
|
|
511
|
+
|
|
512
|
+
Names Info Options Descriptions
|
|
513
|
+
L -(LI|LocalIndex) indices of records
|
|
514
|
+
F -(RF|RemoteFile) remote file names
|
|
515
|
+
D -(DO|DownloadOrder) download order indices
|
|
516
|
+
S -(SF|ServerFile) file names on remote server
|
|
517
|
+
C -(DC|DownloadCommand) command for download Remote file
|
|
518
|
+
B -(BT|BeginTime) for remote file names, 0 - the first day/hour
|
|
519
|
+
E -(ET|EndTime) for remote file names, 0 - the last day/hour
|
|
520
|
+
T -(TI|TimeInterval) for remote file names, i.e., 2H, 5D
|
|
521
|
+
|
|
522
|
+
Update information of all remote files from different datasets, owned by the
|
|
523
|
+
specialist who runs 'dsupdt', may be viewed at the same time if dataset number
|
|
524
|
+
is omitted.
|
|
525
|
+
|
|
526
|
+
3.1.7 Delete Update Information
|
|
527
|
+
-DL or -Delete, (Alias: -RM|-Remove), deletes one or multiple update
|
|
528
|
+
control, local file or remote file records from RDADB.
|
|
529
|
+
|
|
530
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(DL|Delete) [Mode Option]
|
|
531
|
+
-(CI|ControlIndex) UpdateControlIndices
|
|
532
|
+
[-(DB|Debug) DebugModeInfo]
|
|
533
|
+
or
|
|
534
|
+
|
|
535
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(DL|Delete) [Mode Option]
|
|
536
|
+
-(LI|LocalIndex) LocalFileIndices
|
|
537
|
+
[-(DB|Debug) DebugModeInfo]
|
|
538
|
+
or
|
|
539
|
+
|
|
540
|
+
dsupdt [[-(DS|Dataset)] dsnnn.n] -(DL|Delete) [Mode Option]
|
|
541
|
+
-(LI|LocalIndex) LocalFileIndices
|
|
542
|
+
-(RF|RemoteFile) RemoteFileNames
|
|
543
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
544
|
+
[-(DB|Debug) DebugModeInfo]
|
|
545
|
+
|
|
546
|
+
Mode option that can be specified for deleting update information Action:
|
|
547
|
+
-(MD|MyDataset) - delete records no matter the specialist who runs 'dsupdt'
|
|
548
|
+
owns the dataset or not
|
|
549
|
+
|
|
550
|
+
Use this action to delete update information. Control index must be given
|
|
551
|
+
to delete an update control record. Local file index must be given to delete
|
|
552
|
+
both local file and remote file records, while remote file name must be also
|
|
553
|
+
provided to delete a remote file record. Remote file records associated with a
|
|
554
|
+
local file record are all removed if the local file record is deleted.
|
|
555
|
+
|
|
556
|
+
|
|
557
|
+
3.2 All Information Actions
|
|
558
|
+
|
|
559
|
+
For convenient, all update information configured for a dataset can be created,
|
|
560
|
+
modified or viewed together via comprehensive actions of:
|
|
561
|
+
Get All Information - retrieve configuration information of update controls,
|
|
562
|
+
local files and remote files for a dataset
|
|
563
|
+
Set All Information - modify configuration information of update controls,
|
|
564
|
+
local files and remote files for a dataset
|
|
565
|
+
|
|
566
|
+
3.2.1 Get All Information
|
|
567
|
+
-GA or -GetAll, retrieves update control, local file and remote file information
|
|
568
|
+
for a given dataset. This is a comprehensive action, combining three Get actions,
|
|
569
|
+
-GC (-GetControl), -GL (-GetLocalFile) and -GR (-GetRemoteFile).
|
|
570
|
+
|
|
571
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(GA|GetAll) [Mode Option]
|
|
572
|
+
[-(CI|ControlIndex) UpdateControlIndices]
|
|
573
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
574
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
575
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
576
|
+
[-(OF|OutputFile) OutputFileName]
|
|
577
|
+
[-(DB|Debug) DebugModeInfo]
|
|
578
|
+
|
|
579
|
+
Mode option that can be specified for getting all Action:
|
|
580
|
+
-(FO|FormatOutput) - format the column output with a fix width for all values
|
|
581
|
+
of a given field
|
|
582
|
+
|
|
583
|
+
Section headers are generated as [DCUPDT] for update controls, [DLUPDT] for
|
|
584
|
+
local files and [DRUPDT] for remote files.
|
|
585
|
+
|
|
586
|
+
Info option -OF (-OutputFile) is normally used to specify a file name to
|
|
587
|
+
save the retrieved information for later usage. Result of this action is
|
|
588
|
+
displayed on screen if no output file is provided. The output file can be
|
|
589
|
+
edited and treated as a input file to save the changes back to RDADB per
|
|
590
|
+
action -SA (-SetAll).
|
|
591
|
+
|
|
592
|
+
For example, to get all update control, local file and remote file information
|
|
593
|
+
of d277000 for control index 1 with all fields
|
|
594
|
+
|
|
595
|
+
dsupdt d277000 GA -CI 2 -OF d277000.c1
|
|
596
|
+
|
|
597
|
+
<<Content of output file d277000.c1>>
|
|
598
|
+
Dataset<=>d277000
|
|
599
|
+
[DCUPDT]
|
|
600
|
+
ControlIndex<:>Specialist<:>ParentIndex<:>Action<:>Frequency<:>ControlOffset<:>ControlTime<:>RetryInterval<:>UpdateControl<:>EMailControl<:>ErrorControl<:>KeepFile<:>HourOffset<:>DataTime<:>HostName<:>
|
|
601
|
+
1<:>zji<:>0<:>UF<:>1W<:>1D9H<:>2011-10-24 09:00:00<:>12H<:><:>A<:>N<:>N<:>0<:><:><:>
|
|
602
|
+
[DLUPDT]
|
|
603
|
+
LocalIndex<:>LocalFile<:>Action<:>ExecOrder<:>ControlIndex<:>FileArchived<:>DownloadCommand<:>Options<:>Frequency<:>EndPeriod<:>DueInterval<:>EndDate<:>EndHour<:>RetryInterval<:>ValidInterval<:>AgeTime<:>WorkDir<:>MissRemote<:>ProcessRemote<:>BuildCommand<:>CleanCommand<:>Specialist<:>Description<:>
|
|
604
|
+
51<:>oisst.<YYYY>.asc.gz<:>AB<:>1<:>1<:><:>ncftpget ftp://ftp.emc.ncep.noaa.gov/cmb/sst/oisst_v2/ASCII_UPDATE/<:>-GI 1 -AF GZ -DF ASCII<:>1W<:>6<:><:>2011-10-22<:><:><:><:><:>$UPDTDATA/zji/NCEPSST<:>N<:><:><:>rm -f -LF<:>zji<:><YYYY/MM/DD>-<YYYY/MM/DD><:>
|
|
605
|
+
[DRUPDT]
|
|
606
|
+
LocalIndex<:>LocalFile<:>RemoteFile<:>ExecOrder<:>DownloadOrder<:>ServerFile<:>DownloadCommand<:>BeginTime<:>EndTime<:>TimeInterval<:>
|
|
607
|
+
51<:>oisst.<YYYY>.asc.gz<:>oisst.<YYYY>.asc.Z<:>1<:>0<:><:><:><:><:><:>
|
|
608
|
+
|
|
609
|
+
This example shows a weekly update of a local file with remote file name
|
|
610
|
+
different from the local file name.
|
|
611
|
+
|
|
612
|
+
3.2.2 Set All Information
|
|
613
|
+
-SA or -SetAll, and creates and modifies update control, local file and remote
|
|
614
|
+
file information for a given dataset number. This is a comprehensive action,
|
|
615
|
+
combining three Set actions, -SC (-SetControl), -SL (-SetLocalFile) and
|
|
616
|
+
-SR (-SetRemoteFile).
|
|
617
|
+
|
|
618
|
+
dsupdt [-(DS|dataset)] dsnnn.n -(SA|SetAll) [Mode Options]
|
|
619
|
+
-(IF|InputFile) Input Files
|
|
620
|
+
[-(DB|Debug) DebugModeInfo]
|
|
621
|
+
|
|
622
|
+
Mode option that can be specified for this action include:
|
|
623
|
+
-(MD|MyDataset) - sets information into RDADB no matter the specialist who
|
|
624
|
+
runs 'dsupdt' owns the dataset or not
|
|
625
|
+
-(NC|NewControl) - sets a new update control record into RDADB
|
|
626
|
+
-(NL|NewLocfile) - sets a new update record into RDADB
|
|
627
|
+
-(RO|ResetOrder) - resets the executing order indices of the local file
|
|
628
|
+
records according to the order as they are given per
|
|
629
|
+
option -LF (-LocalFile)
|
|
630
|
+
|
|
631
|
+
At least one input file is need to execute this action since section headers
|
|
632
|
+
can be specified in an input file only. One way of creating an input file is
|
|
633
|
+
to get one per action -GA (-GetAll) for a dataset. The output file then
|
|
634
|
+
can be edited and used as an input file to set the update information back
|
|
635
|
+
to RDADB.
|
|
636
|
+
|
|
637
|
+
|
|
638
|
+
3.3 Update Actions
|
|
639
|
+
|
|
640
|
+
Data update activities can be accomplished through either separate Actions
|
|
641
|
+
or one comprehensive Action -UF (-UpdateFile). Actions included in this
|
|
642
|
+
section are:
|
|
643
|
+
Update Data Files - a comprehensive action to download/convert remote
|
|
644
|
+
files, build up local files, archive data files onto
|
|
645
|
+
RDA servers, and clean temporary data files. It
|
|
646
|
+
combines the following four actions
|
|
647
|
+
Download Remote Files - download data files from remote server, process data
|
|
648
|
+
files locally to generate new files or do both
|
|
649
|
+
Build Local Files - build local files from given remote files
|
|
650
|
+
Process Both Files - a comprehensive action to download remote files and
|
|
651
|
+
validate/convert them and build up local files
|
|
652
|
+
Archive Local Files - archive local files onto RDA Servers via utility
|
|
653
|
+
program 'dsarch'
|
|
654
|
+
Clean Temporary Files - remove the temporary data files involved in the update
|
|
655
|
+
procedure
|
|
656
|
+
Unlock Local Files - unlock one or multiple records for them
|
|
657
|
+
to be re-updated on a different host
|
|
658
|
+
Check Update Status - checks update status of local files via looking at if
|
|
659
|
+
remote files are available for retrieving on remote
|
|
660
|
+
servers.
|
|
661
|
+
|
|
662
|
+
3.3.1 Update Data Files
|
|
663
|
+
-UF or -UpdateFile, comprehensive action to download/copy server files,
|
|
664
|
+
validate/convert the server files into remote files, build up local files from
|
|
665
|
+
available remote files, archive local files onto RDA server, and clean
|
|
666
|
+
the temporary data files generated during the update procedure.
|
|
667
|
+
|
|
668
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(UF|UpdateFile) [Mode Options]
|
|
669
|
+
[-(CI|ControlIndex) UpdateControlIndex]
|
|
670
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
671
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
672
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
673
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
674
|
+
[-(SF|ServerFile) ServerFileNames]
|
|
675
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
676
|
+
[-(DC|DownloadCommand) DownloadCommand]
|
|
677
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
678
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
679
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
680
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
681
|
+
[-(MR|MissRemote) AllowMissRemoteFile]
|
|
682
|
+
[-(FQ|Frequency) DataFrequency]
|
|
683
|
+
[-(PR|ProcessRemote) AdditionalRemoteFilePrecess] # user defined
|
|
684
|
+
[-(BC|BuildCommand) CommandBuildLocalFile] # user defined
|
|
685
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
686
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
687
|
+
[-(VS|ValidSize) MinSizeForValidFile]
|
|
688
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
689
|
+
[-(HO|HourOffset) TimeZoneHourOffset]
|
|
690
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
691
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
692
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
693
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
694
|
+
[-(DB|Debug) DebugModeInfo]
|
|
695
|
+
|
|
696
|
+
Mode options that can be specified for updating file Action:
|
|
697
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
698
|
+
the recorded "dsupdt' command be started anywhere.
|
|
699
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
700
|
+
standard outputs and errors
|
|
701
|
+
-(CN|CheckNewer) - for server file is locally, checks if the file is changed
|
|
702
|
+
on Server; downloads it again if changed.
|
|
703
|
+
-(CP|CurrentPeriod) - allows update of the end date/hour beyond the current
|
|
704
|
+
date/hour if the current date/hour lands in the current
|
|
705
|
+
update period
|
|
706
|
+
a single update if this Mode option is not present;
|
|
707
|
+
-(EE-ErrorEmail) - sends email only when error happens during file updates
|
|
708
|
+
-(FU|ForceUpdate) - if present, force update at lease for one end data/time,
|
|
709
|
+
although the update is not due yet;
|
|
710
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful file
|
|
711
|
+
update action
|
|
712
|
+
-(IE|IgnoreError) - works with Mode option -MU (-MultipleUpdate) to skip
|
|
713
|
+
the error updates and continue finish the later
|
|
714
|
+
updates
|
|
715
|
+
-(KR|KeepRemote) - keep remote file on local disk by copying it to local
|
|
716
|
+
file instead of moving
|
|
717
|
+
-(KS|KeepServer) - keep server file on local disk by copying it to remove
|
|
718
|
+
file instead of moving
|
|
719
|
+
-(LO|LogOn) - turn detail logging on if option -PL is present
|
|
720
|
+
-(MO|MissedOnly) - update data file only if it is not archived yet
|
|
721
|
+
-(MU|MultipleUpdate) - allows multiple updates if multiple data update
|
|
722
|
+
periods are available; it defaults to process
|
|
723
|
+
a single update if this Mode option is not present;
|
|
724
|
+
-(NE|NoEmail) - does not send email to the specialist after update
|
|
725
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
726
|
+
-(QE|QuitError) - quits processing update of a dataset when an error
|
|
727
|
+
happens, instead of continuing on other update local files
|
|
728
|
+
-(RA|RetryArchive) - force archive of local files by passing option -RA
|
|
729
|
+
to 'dsarch'
|
|
730
|
+
-(RD|RetryDownload) - retry download the remote file if it is local already
|
|
731
|
+
-(RE|ResetEndTime) - if present, reset end datae/hour according to the local file
|
|
732
|
+
timestamp;
|
|
733
|
+
-(SE|SummaryEmail) - send a summary email to the specialist after update
|
|
734
|
+
without detail logging information
|
|
735
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
736
|
+
for replace temporal pattern of end time in file names
|
|
737
|
+
and download commands
|
|
738
|
+
-UT|UpdateTime) - forces to update the data end time and next due or
|
|
739
|
+
update times
|
|
740
|
+
|
|
741
|
+
Dataset number is provided per Info option -DS (-Dataset) to process data
|
|
742
|
+
update of a single dataset. Some Info options as listed can be provided at
|
|
743
|
+
the update running time to identify specified update records and others to
|
|
744
|
+
override the update information saved in RDADB. Two special Info options -CD
|
|
745
|
+
(-CurrentDate) and -CH (-CurrentHour) can only be proved at run time to
|
|
746
|
+
specify current date/hour rather than the default current time.
|
|
747
|
+
|
|
748
|
+
Multiple update periods may be processed if more than one update frequency
|
|
749
|
+
periods exist between the data end date/hour and the current date/hour and
|
|
750
|
+
the Mode option -MU (-MultipleUpdate) is present at the run time of update
|
|
751
|
+
actions.
|
|
752
|
+
|
|
753
|
+
As mentioned in the introduction section, an individual data file update
|
|
754
|
+
is interrupted if any error encountered during the update process. Error of
|
|
755
|
+
download a remote file is, however, allowed if value 'Y' is set for the update
|
|
756
|
+
record in RDADB via option -MR (-MissRemote) for situation that a local file
|
|
757
|
+
is built from multiple remote files. That means a partial update is allowed.
|
|
758
|
+
The partial update behaviors differently if a valid period is set for a update
|
|
759
|
+
record in RDADB via option -VI (-ValidInterval). The partial update will be put
|
|
760
|
+
on hold until the the remote data file are still not available beyond the valid
|
|
761
|
+
period.
|
|
762
|
+
|
|
763
|
+
If Info Option -PL is present with a value larger than 1, the update actions
|
|
764
|
+
of 'dsupdt' forks multiple child processes to process one update record in
|
|
765
|
+
each process. This helps dataset update with many time consuming independent
|
|
766
|
+
update records.
|
|
767
|
+
|
|
768
|
+
Additional one or multiple email addresses for notification of update result
|
|
769
|
+
can be specified by the Info option -CC (-CarbonCopy). For DSS specialist,
|
|
770
|
+
login user names themselves are acceptable, '-CC zji schuster' for example.
|
|
771
|
+
|
|
772
|
+
This action, -UF (-UpdateFile) can be executed on command line or per cron
|
|
773
|
+
jobs to update files configured in the local file records on a specified
|
|
774
|
+
computer. The local file records can also be linked to update control record
|
|
775
|
+
for this action to be executed via a centralized daemon 'dscheck'. Instead of
|
|
776
|
+
adding Mode options at run time for complicated update actions against local
|
|
777
|
+
file records, most of the Mode options can be pre-configured into update
|
|
778
|
+
control records.
|
|
779
|
+
|
|
780
|
+
Other than this comprehensive update action, simple actions, -DR
|
|
781
|
+
(-DownloadRemote), -BL (-BuildLocal), -AF(-ArchiveFile) or -CF (-CleanFile)
|
|
782
|
+
can be used to do step by step for a download/archive procedure. Their
|
|
783
|
+
command line syntaxes are similar to Action -UF(-UpdateFile) and are described
|
|
784
|
+
following.
|
|
785
|
+
|
|
786
|
+
3.3.2 Download Remote Files
|
|
787
|
+
-DR or -DownloadRemote (Alias: -DownloadRemoteFile), an action to download or
|
|
788
|
+
copy server files onto local working disk, or process data locally to
|
|
789
|
+
generate new data files. These data files that are downloaded/copied/generated
|
|
790
|
+
are called remote files and they are further validated/converted to local files
|
|
791
|
+
via Action -BL (-BuildLocal).
|
|
792
|
+
|
|
793
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(DR|DownloadRemote) [Mode Options]
|
|
794
|
+
[-(CI|ControlIndex) UpdateControlIndex]
|
|
795
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
796
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
797
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
798
|
+
[-(SF|ServerFile) ServerFileNames]
|
|
799
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
800
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
801
|
+
[-(DC|DownloadCommand) DownloadCommand]
|
|
802
|
+
[-(PR|ProcessRemote) AdditionalRemoteFilePrecess] # user defined
|
|
803
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
804
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
805
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
806
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
807
|
+
[-(MR|MissRemote) AllowMissRemoteFile]
|
|
808
|
+
[-(FQ|Frequency) UpdateFrequency]
|
|
809
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
810
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
811
|
+
[-(VS|ValidSize) MinSizeForValidFile]
|
|
812
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
813
|
+
[-(HO|HourOffset) TimeZoneHourOffset]
|
|
814
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
815
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
816
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
817
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
818
|
+
[-(DB|Debug) DebugModeInfo]
|
|
819
|
+
|
|
820
|
+
Mode option that can be specified for downloading file Action:
|
|
821
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
822
|
+
the recorded "dsupdt' command be started anywhere.
|
|
823
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
824
|
+
standard outputs and errors
|
|
825
|
+
-(CN|CheckNewer) - for server file is locally, checks if the file is changed
|
|
826
|
+
on Server; downloads it again if changed.
|
|
827
|
+
-(CP|CurrentPeriod) - allows action of the end date/hour beyond the current
|
|
828
|
+
date/hour if the current date/hour lands in the current
|
|
829
|
+
update period
|
|
830
|
+
-(EE-ErrorEmail) - sends email only whne error happens during retrieving
|
|
831
|
+
remote files
|
|
832
|
+
-(FU|ForceUpdate) - if present, force download at lease for one end data/time,
|
|
833
|
+
although the update is not due yet;
|
|
834
|
+
-(GZ|GMTZone) - uses GMT dates/times as controlling times
|
|
835
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful remote
|
|
836
|
+
file download
|
|
837
|
+
-(IE|IgnoreError) - works with Mode option -MU (-MultipleUpdate) to skip
|
|
838
|
+
the errors for download remote files and continue
|
|
839
|
+
download the later remote files
|
|
840
|
+
-(KS|KeepServer) - keep server file on local disk by copying it to remove
|
|
841
|
+
file instead of moving
|
|
842
|
+
-(LO|LogOn) - turn detail logging on if option -PL is present
|
|
843
|
+
-(MO|MissedOnly) - download remote file only if it is not archived yet
|
|
844
|
+
-(MU|MultipleUpdate) - allows multiple downloads/copies if multiple data update
|
|
845
|
+
periods are available; it defaults to download/copy
|
|
846
|
+
a single remote file if this Mode option is not present
|
|
847
|
+
-(NE|NoEmail) - does not send email to the specialist after download
|
|
848
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
849
|
+
-(QE|QuitError) - quits downloading remote files of a dataset when an
|
|
850
|
+
error happens, instead of continuing for other
|
|
851
|
+
remote files
|
|
852
|
+
-(RD|RetryDownload) - retry download the remote file if it is local already
|
|
853
|
+
-(SE|SummaryEmail) - send a summary email to the specialist after update
|
|
854
|
+
without detail logging information
|
|
855
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
856
|
+
for replace temporal pattern of end time in file names
|
|
857
|
+
and download commands
|
|
858
|
+
|
|
859
|
+
If a remote file is on the local disk already, the download action for this
|
|
860
|
+
remote file is skipped; the existing remote file must removed manually or Mode
|
|
861
|
+
option -RD (-RetryDownload) is present if download/copy is needed again.
|
|
862
|
+
|
|
863
|
+
A server file is the original data file on remote server. It is specified only
|
|
864
|
+
when it is named different from the remote file name.
|
|
865
|
+
|
|
866
|
+
3.3.3 Build Local Files
|
|
867
|
+
-BL or -BuildLocal (Alias: -BuildLocalFile), an action to validate/convert
|
|
868
|
+
remote files and build up local files.
|
|
869
|
+
|
|
870
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(BL|BuildLocal) [Mode Options]
|
|
871
|
+
[-(CI|ControlIndex) UpdateControlIndex]
|
|
872
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
873
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
874
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
875
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
876
|
+
[-(BC|BuildCommand) CommandBuildLocalFile] # user defined
|
|
877
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
878
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
879
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
880
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
881
|
+
[-(FQ|Frequency) UpdateFrequency]
|
|
882
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
883
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
884
|
+
[-(VS|ValidSize) MinSizeForValidFile]
|
|
885
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
886
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
887
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
888
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
889
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
890
|
+
[-(DB|Debug) DebugModeInfo]
|
|
891
|
+
|
|
892
|
+
Mode option that can be specified for building local file Action:
|
|
893
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
894
|
+
the recorded "dsupdt' command be started anywhere.
|
|
895
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
896
|
+
standard outputs and errors
|
|
897
|
+
-(CP|CurrentPeriod) - allows action of the end date/hour beyond the current
|
|
898
|
+
date/hour if the current date/hour lands in the current
|
|
899
|
+
update period
|
|
900
|
+
-(EE-ErrorEmail) - sends email only when error happens during building
|
|
901
|
+
local files
|
|
902
|
+
-(FU|ForceUpdate) - if present, force build at lease for one end data/time,
|
|
903
|
+
although the update is not due yet;
|
|
904
|
+
-(GZ|GMTZone) - uses GMT dates/times as controlling times
|
|
905
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful action
|
|
906
|
+
of building local file
|
|
907
|
+
-(IE|IgnoreError) - works with Mode option -MU (-MultipleUpdate) to skip
|
|
908
|
+
the errors for the action and continue build later
|
|
909
|
+
local files
|
|
910
|
+
-(KR|KeepRemote) - keep remote file on local disk by copying it to local
|
|
911
|
+
file instead of moving
|
|
912
|
+
-(LO|LogOn) - turn detail logging on if option -PL is present
|
|
913
|
+
-(MO|MissedOnly) - build local file only if it is not archived yet
|
|
914
|
+
-(MU|MultipleUpdate) - allows building multiple local files if multiple data
|
|
915
|
+
update periods are available; it defaults to build
|
|
916
|
+
a single local file if this Mode option is not present
|
|
917
|
+
-(NE|NoEmail) - does not send email to the specialist after build
|
|
918
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
919
|
+
-(QE|QuitError) - quits building local files of a dataset when an
|
|
920
|
+
error happens, instead of continuing for other
|
|
921
|
+
local files
|
|
922
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
923
|
+
for replace temporal pattern of end time in file names
|
|
924
|
+
and download commands
|
|
925
|
+
|
|
926
|
+
The additional routines of data validation and conversion for the remote files
|
|
927
|
+
are called by this action if the routines are provided. Nothing is done by
|
|
928
|
+
this action if the local files are simply the same as the remote files.
|
|
929
|
+
Utilities of compress/uncompress and tar/untar are used to convert remote
|
|
930
|
+
files to local files as one to one, multiple to one, or one to multiple.
|
|
931
|
+
Local files are the files in their final states ready to be archived.
|
|
932
|
+
|
|
933
|
+
If a local file is on the local disk already, the build local file action for
|
|
934
|
+
this local file is skipped; the existing local file must removed manually if
|
|
935
|
+
rebuild is needed.
|
|
936
|
+
|
|
937
|
+
3.3.4 Process Both Files
|
|
938
|
+
-PB or -ProcessBoth, a comprehensive action to download remote files and
|
|
939
|
+
validate/convert them and build up local files.
|
|
940
|
+
|
|
941
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(PB|ProcessBoth) [Mode Options]
|
|
942
|
+
[-(CI|ControlIndex) UpdateControlIndex]
|
|
943
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
944
|
+
[-(LF|LocalFile) LocalFileName]
|
|
945
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
946
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
947
|
+
[-(SF|ServerFile) ServerFileNames]
|
|
948
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
949
|
+
[-(DC|DownloadCommand) DownloadCommand]
|
|
950
|
+
[-(PR|ProcessRemote) AdditionalRemoteFilePrecess] # user defined
|
|
951
|
+
[-(BC|BuildCommand) CommandBuildLocalFile] # user defined
|
|
952
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
953
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
954
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
955
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
956
|
+
[-(FQ|Frequency) UpdateFrequency]
|
|
957
|
+
[-(MR|MissRemote) AllowMissRemoteFile]
|
|
958
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
959
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
960
|
+
[-(VS|ValidSize) MinSizeForValidFile]
|
|
961
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
962
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
963
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
964
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
965
|
+
[-(HO|HourOffset) TimeZoneHourOffset]
|
|
966
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
967
|
+
[-(DB|Debug) DebugModeInfo]
|
|
968
|
+
|
|
969
|
+
Mode option that can be specified for downloading file Action:
|
|
970
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
971
|
+
the recorded "dsupdt' command be started anywhere.
|
|
972
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
973
|
+
standard outputs and errors
|
|
974
|
+
-(CN|CheckNewer) - for server file is locally, checks if the file is changed
|
|
975
|
+
on Server; downloads it again if changed.
|
|
976
|
+
-(CP|CurrentPeriod) - allows action of the end date/hour beyond the current
|
|
977
|
+
date/hour if the current date/hour lands in the current
|
|
978
|
+
update period
|
|
979
|
+
-(EE-ErrorEmail) - sends email only when error happens during retrieving
|
|
980
|
+
remote files and building local files
|
|
981
|
+
-(FU|ForceUpdate) - if present, force process at lease for one end data/time,
|
|
982
|
+
although the update is not due yet;
|
|
983
|
+
-(GZ|GMTZone) - uses GMT dates/times as controlling times
|
|
984
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful action
|
|
985
|
+
-(IE|IgnoreError) - works with Mode option -MU (-MultipleUpdate) to skip
|
|
986
|
+
the errors for the action and continue download/build
|
|
987
|
+
the later remote/local files
|
|
988
|
+
-(KR|KeepRemote) - keep remote file on local disk by copying it to local
|
|
989
|
+
file instead of moving
|
|
990
|
+
-(KS|KeepServer) - keep server file on local disk by copying it to remove
|
|
991
|
+
file instead of moving
|
|
992
|
+
-(MO|MissedOnly) - process data file only if it is not archived yet
|
|
993
|
+
-(LO|LogOn) - turn detail logging on if option -PL is present
|
|
994
|
+
-(MU|MultipleUpdate) - allows multiple download/build if multiple data update
|
|
995
|
+
periods are available
|
|
996
|
+
-(NE|NoEmail) - does not send email to the specialist after action
|
|
997
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
998
|
+
-(QE|QuitError) - quits processing updates of a dataset when an
|
|
999
|
+
error happens, instead of continuing for other
|
|
1000
|
+
remote/local files
|
|
1001
|
+
-(RD|RetryDownload) - retry download the remote file if it is local already
|
|
1002
|
+
-(SE|SummaryEmail) - send a summary email to the specialist after action
|
|
1003
|
+
without detail logging information
|
|
1004
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
1005
|
+
for replace temporal pattern of end time in file names
|
|
1006
|
+
and download commands
|
|
1007
|
+
|
|
1008
|
+
3.3.5 Archive Local Files
|
|
1009
|
+
-AF or -ArchiveFile, an action to archive local files onto RDA
|
|
1010
|
+
Server according the DSARCH action previously saved in RDADB or given on
|
|
1011
|
+
command line per info Option -AN (-ActionName).
|
|
1012
|
+
|
|
1013
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(AF|ArchiveFile) [Mode Options]
|
|
1014
|
+
[-(CI|ControlIndex) UpdateControlIndex]
|
|
1015
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
1016
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
1017
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
1018
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
1019
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
1020
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
1021
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
1022
|
+
[-(FQ|Frequency) UpdateFrequency]
|
|
1023
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
1024
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
1025
|
+
[-(VS|ValidSize) MinSizeForValidFile]
|
|
1026
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
1027
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
1028
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
1029
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
1030
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
1031
|
+
[-(DB|Debug) DebugModeInfo]
|
|
1032
|
+
[-(GZ|GMTZone) GreenwichMeanTimeZone]
|
|
1033
|
+
|
|
1034
|
+
Mode option that can be specified for building local file Action:
|
|
1035
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
1036
|
+
the recorded "dsupdt' command be started anywhere.
|
|
1037
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
1038
|
+
standard outputs and errors
|
|
1039
|
+
-(CP|CurrentPeriod) - allows action of the end date/hour beyond the current
|
|
1040
|
+
date/hour if the current date/hour lands in the current
|
|
1041
|
+
update period
|
|
1042
|
+
-(EE-ErrorEmail) - sends email only when error happens during archiving files
|
|
1043
|
+
-(FU|ForceUpdate) - if present, force archive at lease for one end data/time,
|
|
1044
|
+
although the update is not due yet;
|
|
1045
|
+
-(GZ|GMTZone) - uses GMT dates/times as controlling times
|
|
1046
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful action
|
|
1047
|
+
of building local file
|
|
1048
|
+
-(IE|IgnoreError) - works with Mode option -MU (-MultipleUpdate) to skip
|
|
1049
|
+
the errors for archiving local files and continue
|
|
1050
|
+
archive the later local files
|
|
1051
|
+
-(MO|MissedOnly) - archive local file only if it is not archived yet
|
|
1052
|
+
-(MU|MultipleUpdate) - allows archive multiple local files if multiple
|
|
1053
|
+
local files are available for different update periods;
|
|
1054
|
+
it defaults to archive a single local file if this Mode
|
|
1055
|
+
option is not present
|
|
1056
|
+
-(NE|NoEmail) - does not send email to the specialist after archive
|
|
1057
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
1058
|
+
-(RA|RetryArchive) - force archive of local files by passing option -RA
|
|
1059
|
+
to 'dsarch'
|
|
1060
|
+
-(RE|ResetEndTime) - if present, reset end datae/hour according to the local file
|
|
1061
|
+
timestamp;
|
|
1062
|
+
-(SE|SummaryEmail) - send a summary email to the specialist after update
|
|
1063
|
+
without detail logging information
|
|
1064
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
1065
|
+
for replace temporal pattern of end time in file names
|
|
1066
|
+
and download commands
|
|
1067
|
+
-(UT|UpdateTime) - forces to update the data end time and next due or
|
|
1068
|
+
update times
|
|
1069
|
+
|
|
1070
|
+
If a local file is not ready on the local disk, the archive action for this
|
|
1071
|
+
local file is skipped.
|
|
1072
|
+
|
|
1073
|
+
3.3.6 Clean Temporary Files
|
|
1074
|
+
-CF or -CleanFile, an action to clean the temporary files at the local
|
|
1075
|
+
working area. Those files can be either already archived successfully or
|
|
1076
|
+
out of date files.
|
|
1077
|
+
|
|
1078
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(CF|CleanFile) [Mode Options]
|
|
1079
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
1080
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
1081
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
1082
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
1083
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
1084
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
1085
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
1086
|
+
[-(FQ|Frequency) UpdateFrequency]
|
|
1087
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
1088
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
1089
|
+
[-(PL|ProcessLimit) MaxNumberOfChildProcesses]
|
|
1090
|
+
[-(SB|SbatchOptions) SlurmBatchOptions]
|
|
1091
|
+
[-(QS|QsubOptions) PBSBatchOptions]
|
|
1092
|
+
[-(BP|BatchProcess) [BatchProcessHosts]]
|
|
1093
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
1094
|
+
[-(DB|Debug) DebugModeInfo]
|
|
1095
|
+
|
|
1096
|
+
Mode option that can be specified for building local file Action:
|
|
1097
|
+
-(AW|AnyWhere) - works with Info option -BP (-BatchProcess) to allow
|
|
1098
|
+
the recorded "dsupdt' command be started anywhere.
|
|
1099
|
+
-(BG|BackGround) - background process to turn off screen display for both
|
|
1100
|
+
standard outputs and errors
|
|
1101
|
+
-(CP|CurrentPeriod) - allows action of the end date/hour beyond the current
|
|
1102
|
+
date/hour if the current date/hour lands in the current
|
|
1103
|
+
update period
|
|
1104
|
+
-(EE-ErrorEmail) - sends email only when error happens during cleaning files
|
|
1105
|
+
locally
|
|
1106
|
+
-(FU|ForceUpdate) - if present, force clean at lease for one end data/time,
|
|
1107
|
+
although the update is not due yet;
|
|
1108
|
+
-(GZ|GMTZone) - uses GMT dates/times as controlling times
|
|
1109
|
+
-(HU|HourlyUpdate) - forces update of time to hours after successful action
|
|
1110
|
+
of cleaning the temporary files
|
|
1111
|
+
-(MU|MultipleUpdate) - allows cleaning multiple temporary files if multiple
|
|
1112
|
+
update periods are available
|
|
1113
|
+
-(NE|NoEmail) - does not send email to the specialist after clean
|
|
1114
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
1115
|
+
-(SE|SummaryEmail) - sends a summary email to the specialist after update
|
|
1116
|
+
without detail logging information
|
|
1117
|
+
-(UB|UseBeginTime) - when present, uses the beginning time of update period
|
|
1118
|
+
for replace temporal pattern of end time in file names
|
|
1119
|
+
and clear commands
|
|
1120
|
+
|
|
1121
|
+
If a temporary file is not on the local working area, the cleaning action is
|
|
1122
|
+
is ignored.
|
|
1123
|
+
|
|
1124
|
+
3.3.7 Unlock Local Files
|
|
1125
|
+
-UL or -UnLockUpdate (Alias: -UnLock), unlocks update records that their
|
|
1126
|
+
processes abort abnormally. Process IDs and hostnames are saved in an update
|
|
1127
|
+
control and local file records temporarily when the records are under process.
|
|
1128
|
+
If the update processes abort abnormally, the PIDs and hostnames may not be
|
|
1129
|
+
cleaned properly. If the same update records are reprocessed on a different
|
|
1130
|
+
computer, the lock information blocks the update actions. Use this action to
|
|
1131
|
+
clean up the PID information manually to allow the update control and local
|
|
1132
|
+
file records to be re-processed.
|
|
1133
|
+
|
|
1134
|
+
dsupdt -(-UL|UnLockUpdate)
|
|
1135
|
+
-(CI|ControlIndex) UpdateControlIndices
|
|
1136
|
+
[-(DB|Debug) DebugModeInfo]
|
|
1137
|
+
or
|
|
1138
|
+
|
|
1139
|
+
dsupdt -(-UL|UnLockUpdate)
|
|
1140
|
+
-(LI|LocalIndex) LocfileIndices
|
|
1141
|
+
[-(DB|Debug) DebugModeInfo]
|
|
1142
|
+
|
|
1143
|
+
It is mandatory to provide an update control index and local file index to
|
|
1144
|
+
remove locks on the control and local file record, respectively.
|
|
1145
|
+
|
|
1146
|
+
3.3.8 Check Update Status
|
|
1147
|
+
-CU or -CheckUpdate, checks update status via looking at if remote files are available
|
|
1148
|
+
for retrieving. This action is useful to check if remote files are ready to be retrieved
|
|
1149
|
+
for updates for remote data files are ready irregularly while the data themselves
|
|
1150
|
+
are organized into files with regular temporal periods.
|
|
1151
|
+
|
|
1152
|
+
dsupdt [[-(DS|dataset)] dsnnn.n] -(UF|UpdateFile) [Mode Options]
|
|
1153
|
+
[-(LI|LocalIndex) LocalFileIndices]
|
|
1154
|
+
[-(LF|LocalFile) LocalFileNames]
|
|
1155
|
+
[-(CI|ControlIndex) UpdateControlIndices]
|
|
1156
|
+
[-(XO|ExecOrder) ExecOrderIndex]
|
|
1157
|
+
[-(RF|RemoteFile) RemoteFileNames]
|
|
1158
|
+
[-(SF|ServerFile) ServerFileNames]
|
|
1159
|
+
[-(DO|DownloadOrder) DownloadOrderIndices]
|
|
1160
|
+
[-(DC|DownloadCommand) DownloadCommand]
|
|
1161
|
+
[-(ED|EndDate) NextDataEndDate]
|
|
1162
|
+
[-(EH|EndHour) NextDataEndHour]
|
|
1163
|
+
[-(GP|GenericPattern) GenericPatterns]
|
|
1164
|
+
[-(WD|WorkDir) WorkingDirectory]
|
|
1165
|
+
[-(MR|MissRemote) AllowMissRemoteFile]
|
|
1166
|
+
[-(FQ|Frequency) UpdateFrequencies]
|
|
1167
|
+
[-(CD|CurrentDate) CurrentDate] # provide if different than current
|
|
1168
|
+
[-(CH|CurrentHour) CurrentHour] # provide if different than current
|
|
1169
|
+
[-(HO|HourOffset) TimeZoneHourOffset]
|
|
1170
|
+
[-(CC|CarbonCopy) Cc'dEmailAddresses]
|
|
1171
|
+
[-(DB|Debug) DebugModeInfo]
|
|
1172
|
+
|
|
1173
|
+
|
|
1174
|
+
Mode options that can be specified for checking update Action:
|
|
1175
|
+
-(CA|CheckAll) - forces to all remote files instead of quitting checks
|
|
1176
|
+
after first unavailable remote file
|
|
1177
|
+
-(EE|-ErrorEmail) - sends email only when error happens during checking
|
|
1178
|
+
update status
|
|
1179
|
+
-(MU|MultipleUpdate) - checks remote files for multiple update periods
|
|
1180
|
+
-(NE|NoEmail) - does not send email to the specialist after update checking
|
|
1181
|
+
-(NY|NoLeapYear) - skips February 29 for leap years
|
|
1182
|
+
-(SE|SummaryEmail) - send a summary email for update status to the specialist
|
|
1183
|
+
without detail logging information
|
|
1184
|
+
-(UB|UseBeginTime) - when present, use the beginning time of update period
|
|
1185
|
+
for replace temporal pattern of end time in file names
|
|
1186
|
+
and download commands
|
|
1187
|
+
|
|
1188
|
+
Execute this action without other conditions, all datasets that are due for updates
|
|
1189
|
+
are checked for update status.
|
|
1190
|
+
|
|
1191
|
+
4 MODE OPTIONS
|
|
1192
|
+
|
|
1193
|
+
Use proper Mode options to modify behaviors of Action options. Mode options
|
|
1194
|
+
are all optional. No value is allowed to be passed in following any Mode option.
|
|
1195
|
+
|
|
1196
|
+
-AW or -AnyWhere, works with Info option -BP (-BatchProcess) to use empty
|
|
1197
|
+
work directory so that the recorded "dsupdt' command can be started processing
|
|
1198
|
+
any where other than the directory where the command was recorded initially.
|
|
1199
|
+
|
|
1200
|
+
-BG or -BackGround (Alias: -b), background process. When it presents screen
|
|
1201
|
+
display is turned off for both standard outputs and errors.
|
|
1202
|
+
|
|
1203
|
+
-CA or -CheckAll, if presents, allows check update actions -CU (-CheckUpdate)
|
|
1204
|
+
to check availability of all remote files. Without it, the checking procedure
|
|
1205
|
+
quits after the first unavailable remote.
|
|
1206
|
+
|
|
1207
|
+
-CN or -CheckNew (Alias: -CheckNewFile), for server file is locally ready,
|
|
1208
|
+
checks if the file on server is changed; downloads it again if it is changed.
|
|
1209
|
+
This option is ignored if -RD is present. There must be a local copy of the data
|
|
1210
|
+
file for checking newer file on server. The local data file copy is normally
|
|
1211
|
+
the local copy of the server file, but it can be an existing remote file or
|
|
1212
|
+
local file if it is the same file content as the server file, although it
|
|
1213
|
+
may have different file name.
|
|
1214
|
+
|
|
1215
|
+
For this Mode option to work properly, it is specialists' responsibility to
|
|
1216
|
+
maintain a proper local data file and a valid download command that can check
|
|
1217
|
+
data file info on server, such commands as 'cp', 'mv' and 'tar' for a file on
|
|
1218
|
+
local host, 'scp', 'wget' and 'ncftpget' for a file on remote host. For a
|
|
1219
|
+
customized download command, it must have build-in function of checking newer
|
|
1220
|
+
server file.
|
|
1221
|
+
|
|
1222
|
+
If the server time zone is different from the local time, Info Option -HO
|
|
1223
|
+
(-HourOffset) for an offset value in hours must be provided to be able to
|
|
1224
|
+
check newer file on remote server by comparing the file local and remote
|
|
1225
|
+
timestamps.
|
|
1226
|
+
|
|
1227
|
+
-CP or -CurrentPeriod, if presents, allows update actions for the current
|
|
1228
|
+
date/hour lands in the current update period, although the data end date/hour
|
|
1229
|
+
are not due yet.
|
|
1230
|
+
|
|
1231
|
+
-EE or -ErrorEmail, sends email to the specialist after update only when error
|
|
1232
|
+
happens during the update action.
|
|
1233
|
+
|
|
1234
|
+
-FO or -FormatOutput, if present, formats column output results for get
|
|
1235
|
+
actions. A same width, evaluated dynamically, is applied for all values of a
|
|
1236
|
+
given field.
|
|
1237
|
+
|
|
1238
|
+
-FU or -ForceUpdate, if present, force update at lease for one end data/time,
|
|
1239
|
+
although the update may not be due yet.
|
|
1240
|
+
|
|
1241
|
+
-GZ or -GMTZone (Alias: -GMT|GreenwichZone|UTC), uses GMT dates/times as
|
|
1242
|
+
controlling times for update actions, instead of local time.
|
|
1243
|
+
|
|
1244
|
+
-HU or -HourlyUpdate, forces update of time to hours after successful archive
|
|
1245
|
+
action for data file update.
|
|
1246
|
+
|
|
1247
|
+
-IE or -IgnoreError, works with Mode option -MU (-MultipleUpdate) to skip
|
|
1248
|
+
the update errors and continue update for the later archive periods. For archive
|
|
1249
|
+
record with defined valid interval, this Mode option will also make 'dsupdt'
|
|
1250
|
+
to check back to the valid period and re-archive if found any missing archive.
|
|
1251
|
+
|
|
1252
|
+
-KR or -KeepRemote (Alias: KeepRemoteFile), keeps remote file on local disk by
|
|
1253
|
+
copying it to local file instead of moving.
|
|
1254
|
+
|
|
1255
|
+
-KS or -KeepServer (Alias: -KeepServerFile), keeps server file on local disk
|
|
1256
|
+
by copying it to remote file instead of moving.
|
|
1257
|
+
|
|
1258
|
+
-LO or -LO (Alias: -LoggingOn), turns detail logging on when Info option
|
|
1259
|
+
-PL (-ProcessLimit) is present for update actions.
|
|
1260
|
+
|
|
1261
|
+
-MD or -MyDataset, allows a specialist to add or modify update information of
|
|
1262
|
+
a given dataset listed for another specialist.
|
|
1263
|
+
|
|
1264
|
+
-MO or -MissedOnly, skips update action if the data file is archived already.
|
|
1265
|
+
This is often used with Mode option -MU (-MultipleUpdate). This option is
|
|
1266
|
+
ignored if one of the Mode options -RA, -RD or -CN is present.
|
|
1267
|
+
|
|
1268
|
+
-MU or -MultipleUpdate, allows multiple updates if multiple data update
|
|
1269
|
+
periods are available; 'dsupdt' defaults to process a single update if this
|
|
1270
|
+
Mode option is not present.
|
|
1271
|
+
|
|
1272
|
+
-NC or -NewControl, a new update control record can only be added when this Mode
|
|
1273
|
+
option is present and update control index is given as 0 when Action -SC
|
|
1274
|
+
(-SetControl) of 'dsupdt' is executed. This Mode option prevents adding update
|
|
1275
|
+
control records unintentionally.
|
|
1276
|
+
|
|
1277
|
+
-NE or -NoEmail, does not send email notice to the specialist after update.
|
|
1278
|
+
|
|
1279
|
+
-NL or -NewLocfile, a new local file record can only be added when this Mode option
|
|
1280
|
+
is present and local file index is given as 0 when Action -SL (-SetLocalFile) of
|
|
1281
|
+
'dsupdt' is executed. This Mode option prevents adding local file update records
|
|
1282
|
+
unintentionally.
|
|
1283
|
+
|
|
1284
|
+
-NY or -NoLeapYear, skips February 29 for leap years.
|
|
1285
|
+
|
|
1286
|
+
-QE or -QuitError (Alias: QuitOnError), quits processing updates of a dataset
|
|
1287
|
+
when an error happens, instead of continuing on finishing other local file
|
|
1288
|
+
records.
|
|
1289
|
+
|
|
1290
|
+
-RA or -RetryArchive, forces archive of local files by passing option -RA
|
|
1291
|
+
to 'dsarch'.
|
|
1292
|
+
|
|
1293
|
+
-RD or -RetryDownload (Alias: -Redownload), retries download the remote file
|
|
1294
|
+
if it is local already for Action -DR (-DownloadRemote).
|
|
1295
|
+
|
|
1296
|
+
-RE or -ResetEndTime - if present, reset end datae/hour if file timestamp is newer
|
|
1297
|
+
than end date/hoour plus one update frequency.
|
|
1298
|
+
|
|
1299
|
+
-RO or -ResetOrder (Alias: -Reorder), when present, resets the executing
|
|
1300
|
+
orders of the given local file lists as the orders the data files are given
|
|
1301
|
+
for action -SL (-SetLocalFile). Another way of reordering the data files is
|
|
1302
|
+
to provide explicitly the order index values per Info option
|
|
1303
|
+
-XO (-ExecOrder). Valid values are 1, 2, 3, ...
|
|
1304
|
+
|
|
1305
|
+
-SE or -SummaryEmail, sends a summary email to the specialist after data update
|
|
1306
|
+
action without detail logging information.
|
|
1307
|
+
|
|
1308
|
+
-UB or -UseBeginTime when present, uses the beginning time of update period
|
|
1309
|
+
for replacing temporal patterns configured for file names, download
|
|
1310
|
+
commands, and other fields.
|
|
1311
|
+
|
|
1312
|
+
-UT or -UpdateTime, when present, forces to update the data end time and
|
|
1313
|
+
next due update times in local file records no matter the update actions are
|
|
1314
|
+
successful or not.
|
|
1315
|
+
|
|
1316
|
+
|
|
1317
|
+
5 INFORMATION OPTIONS
|
|
1318
|
+
|
|
1319
|
+
Information options are used to pass information, one or multiple values, into
|
|
1320
|
+
'dsupdt'. Two types of Info options are used:
|
|
1321
|
+
Single-Value Info Options - pass a single value to 'dsupdt'
|
|
1322
|
+
Multi-Value Info Options - pass one or multiple values to 'dsupdt'
|
|
1323
|
+
|
|
1324
|
+
|
|
1325
|
+
5.1 Single-Value Info Options
|
|
1326
|
+
|
|
1327
|
+
A single-value Info option is used to pass one value into this application.
|
|
1328
|
+
One value, and one only, must follow a single-value option; otherwise an
|
|
1329
|
+
error message is displayed if no value or more than one value passed in.
|
|
1330
|
+
|
|
1331
|
+
-BP or -BatchProcess (Alias: -d, -DelayedMode), delayed mode execution. When
|
|
1332
|
+
it presents, the 'dsupdt' command is not executed right way, but the command
|
|
1333
|
+
information is recorded into RDADB instead and the command is executed later
|
|
1334
|
+
by the centralized daemon 'dscheck'. One or multiple host names can be specified
|
|
1335
|
+
after option -BP to force the 'dsupdt' command be executed on or not on the hosts.
|
|
1336
|
+
For examples, '-d rda-work' to run on host 'rda-work'; and '-d !rda-work' to
|
|
1337
|
+
run on all configured hosts other than 'rda-work'. A upper limit for number of
|
|
1338
|
+
tries can also be passed in with this options, 1 to 99, as '-d 2' for example.
|
|
1339
|
+
It default to 1 for cammand of 'dsupdt' if not specified.
|
|
1340
|
+
|
|
1341
|
+
-CD or -CurrentDate, provides a current date for update actions when a date
|
|
1342
|
+
is used other than today. This option provides opportunity for specialists to
|
|
1343
|
+
rerun a update for re-archiving or a deleted update action.
|
|
1344
|
+
|
|
1345
|
+
-CH or -CurrentHour, provides a current hour for update actions when an hour
|
|
1346
|
+
is used other than the executing hour. This option provides opportunity for
|
|
1347
|
+
specialists to rerun a update for re-archiving or a deleted update action.
|
|
1348
|
+
|
|
1349
|
+
-DS or -Dataset, for a dataset ID, in format of [a-z]NNNNNN.
|
|
1350
|
+
|
|
1351
|
+
-DV or -Divider (Alias: -Delimiter, -Separator), delimiter for separating
|
|
1352
|
+
columns of multi-value Info options in input files. It is default to '<:>'.
|
|
1353
|
+
|
|
1354
|
+
-ES or -EqualSign, for an equal sign of assigning one value to either a
|
|
1355
|
+
single-value option or multi-value option in input files. It is defaulted
|
|
1356
|
+
to '<=>'.
|
|
1357
|
+
|
|
1358
|
+
-FN or -FieldNames, for a string of single letter field names. Values of
|
|
1359
|
+
the selected fields are retrieved per actions -GL (-GetLocalFile) and
|
|
1360
|
+
-GR (-GetRemoteFile). Values of default fields are retrieved if this option
|
|
1361
|
+
is not specified. Valid field names are listed in corresponding action
|
|
1362
|
+
sections.
|
|
1363
|
+
|
|
1364
|
+
-LN or -LoginName, login name of the current user; it defaults to the current
|
|
1365
|
+
login DSS specialist who starts execution of 'dsupdt'. Set this option if you
|
|
1366
|
+
try to run this utility for a specialist other than yourself. Mode option -MD
|
|
1367
|
+
is added to execute 'dsarch' if a login name provided by option -LN is
|
|
1368
|
+
different than the one who starts 'dsupdt'.
|
|
1369
|
+
|
|
1370
|
+
-OF or -OutputFile, leading an output file name into which the output result
|
|
1371
|
+
of this application is dumped. Output file format is similar to the format of
|
|
1372
|
+
the input files. If this option is not given, the result is displayed on
|
|
1373
|
+
screen.
|
|
1374
|
+
|
|
1375
|
+
-ON or -OrderNames, for a string of single letter field names use to order
|
|
1376
|
+
the results of GET actions, -GC (-GETControl), -GL (-GetLocalFile) or -GR
|
|
1377
|
+
(-GetRemoteFile). Uppercase is for Ascending order while lowercase is for
|
|
1378
|
+
Descending order.
|
|
1379
|
+
|
|
1380
|
+
-PL or -ProcessLimit, defaults to 1. If present with a value larger than 1,
|
|
1381
|
+
the update actions of 'dsupdt' forks multiple child processes to process one
|
|
1382
|
+
update record in each child process. The master program quits when all
|
|
1383
|
+
available update records are handled in child processes. This helps data
|
|
1384
|
+
update for datasets with many time consuming independent update records.
|
|
1385
|
+
|
|
1386
|
+
-AO or -ActOption, for setting Action and Mode options in input files. It is
|
|
1387
|
+
default to '<!>'.
|
|
1388
|
+
|
|
1389
|
+
-VS or -ValidSize, a minimal size for a file to be valid for archiving process.
|
|
1390
|
+
It defaults to 100 bytes and can be reset on command line with the update actions.
|
|
1391
|
+
|
|
1392
|
+
5.2 Multi-Value Info Options
|
|
1393
|
+
|
|
1394
|
+
A multi-value Info option is used to pass multiple values for one Info option
|
|
1395
|
+
into 'dsupdt'. At lease one value must follow each multi-value option.
|
|
1396
|
+
|
|
1397
|
+
-AN or -ActionName, (Alias: -Action), Action options set in update controls
|
|
1398
|
+
and local file records to execute 'dsupdt' for processing controls and 'dsarch'
|
|
1399
|
+
for archiving data, respectively.
|
|
1400
|
+
|
|
1401
|
+
-AT or -AgeTime (Alias: -fileAge, -FileAgeTime), sets the number of days or
|
|
1402
|
+
hours the remote files have to be aged on remote servers to be valid for
|
|
1403
|
+
download. If this value is set, 'dsupdt' tries to check timestamp of a data
|
|
1404
|
+
file on remote server. If the time difference between the current date/time
|
|
1405
|
+
and the data file date/time is shorter than the age time set in RDADB by this
|
|
1406
|
+
option, 'dsupdt' will not download this file.
|
|
1407
|
+
|
|
1408
|
+
-BC or -BuildCommand (Alias: -BuildCmd), external commands for additional
|
|
1409
|
+
processes to build local files instead of the standard way of taring/compressing.
|
|
1410
|
+
|
|
1411
|
+
Options allowed passing information into the command include -LF, -RF, -SF,
|
|
1412
|
+
-DS, -ED, -EH, -SN/LN, and -LI. Each individual option is replaced by a real
|
|
1413
|
+
value at the time 'dsupdt' calls the command.
|
|
1414
|
+
|
|
1415
|
+
If you send email inside your build command and want the email being held until
|
|
1416
|
+
the update action of 'dsupdt' is finished successfully, you can use option -LI
|
|
1417
|
+
to pass the local file index to you command, and save a customized email into
|
|
1418
|
+
field of dlupdt.emnote in the local file record. If you want the email being
|
|
1419
|
+
held until multiple local file records being updated successfully, you can pass
|
|
1420
|
+
to your command the index value of the last local file record. The saved
|
|
1421
|
+
customized email will be sent out only when the update action is completed
|
|
1422
|
+
successfully against all the included local file indices.
|
|
1423
|
+
|
|
1424
|
+
-BT or -BeginTime, used for remote file records to specify beginning times.
|
|
1425
|
+
Combined with the time interval given per Info option -TI, it is used to set
|
|
1426
|
+
the beginning time of creating multiple remote file names based on the
|
|
1427
|
+
temporal pattern of the file name in each update period. It is defaulted to 0,
|
|
1428
|
+
which means the first available time during a update period, i.e., 00 hour for
|
|
1429
|
+
hourly controlled updates and the 1st of a month for monthly updates.
|
|
1430
|
+
|
|
1431
|
+
Beginning time can be set as D:D:D:D:D:D:D:D:D:D:D:D for 12 offset values for
|
|
1432
|
+
setting beginning days of each month in a year, if they are not the first days
|
|
1433
|
+
of each month of a monthly controlled update.
|
|
1434
|
+
|
|
1435
|
+
-CC or -CarbonCopy, sets additional one or multiple email addresses into update
|
|
1436
|
+
control records via Action -SC (-SetControl) to send Cc'd email notification of
|
|
1437
|
+
the update results. This option can also provided on command line to do the same
|
|
1438
|
+
thing. For DSS specialists, login user names themselves are acceptable; otherwise
|
|
1439
|
+
full email addresses are required for email domains other than 'ucar.edu'.
|
|
1440
|
+
|
|
1441
|
+
For example, to make a carbon copy email to 'schuster@ucar.edu' for update
|
|
1442
|
+
results of 'd337000', provide Info option as '-CC schuster'
|
|
1443
|
+
|
|
1444
|
+
dsupdt d337000 UF -MU -IE -CC schuster
|
|
1445
|
+
|
|
1446
|
+
-CI or -ControlIndex (Alias: -UpdateControlIndex), update control indices for
|
|
1447
|
+
set update control records. A single Control index can be provided for updates of
|
|
1448
|
+
multiple local file records. Provide one Update Control index at a time for
|
|
1449
|
+
any update actions.
|
|
1450
|
+
|
|
1451
|
+
-CL or -CleanCommand, sets in RDADB for a local file record, and used later to
|
|
1452
|
+
clean the temporary data files during data update.
|
|
1453
|
+
|
|
1454
|
+
-CO or -ControlOffset, time offsets, i.e., 2D10H, 3H15N, work with control
|
|
1455
|
+
frequency to set next update control time. For a monthly control frequency 1M
|
|
1456
|
+
and the data update is finished for the current month 2011-09, without the
|
|
1457
|
+
control time offset, the update control time for next month is set to
|
|
1458
|
+
2011-10-01 00:00.00. If an offset is 2D10H30N, update control time for next
|
|
1459
|
+
month is set to 2011-10-03 10:30.00 instead.
|
|
1460
|
+
|
|
1461
|
+
-CT or -ControlTime, update control times, in form of YYY-MM-DD HH:NN:SS,
|
|
1462
|
+
for update control records to be due for processing. The control times are
|
|
1463
|
+
reset to next update control periods according to both the control frequencies
|
|
1464
|
+
and control offsets after successful update processes in the current periods.
|
|
1465
|
+
|
|
1466
|
+
-DC or -DownloadCommand (Alias: -Download|-Command), command to download
|
|
1467
|
+
a file on a remote server, copy a file locally somewhere else, or process data
|
|
1468
|
+
and generate a remote file. It can be set in both local and remote file records.
|
|
1469
|
+
This command is used for Action of download remote file, -DR (-DownloadRemote).
|
|
1470
|
+
A command provided on command line at execution is considered first; otherwise,
|
|
1471
|
+
the command specified in the remote file record, if exists, is used. If the
|
|
1472
|
+
command is not set in the remote file record either, the one set for the local
|
|
1473
|
+
file record is used.
|
|
1474
|
+
|
|
1475
|
+
-DB or -Debug, turns on debug mode with specified information. This option
|
|
1476
|
+
provides upto 3 values, they are Debug Level, debug log file path and debug
|
|
1477
|
+
log file name. The debug level is mandatory for this option. It can be a
|
|
1478
|
+
single integer value, for example, 1000 means to log debug messages for debug
|
|
1479
|
+
levels 1 to 1000; or a range of values, for example, 200-1000 means to log
|
|
1480
|
+
debug messages from debug levels 200 to 1000. The default debug file path is
|
|
1481
|
+
'$DSSHOME}/dssdb/log' and the default debug file name is 'mydss.dbg'. Provides
|
|
1482
|
+
the second and third values for this option to override the default ones
|
|
1483
|
+
respectively.
|
|
1484
|
+
|
|
1485
|
+
-DE or -Description (Alias: -Desc|-Note|-FileDesc|-FileDescription), for file
|
|
1486
|
+
descriptions of data files. Multiple lines are allowed for a
|
|
1487
|
+
description if it is passed in from an input file specified by Info option
|
|
1488
|
+
-IF (-InputFile).
|
|
1489
|
+
|
|
1490
|
+
-DO or -DownloadOrder, the download order indices for remote file records
|
|
1491
|
+
for situation that a single remote file is available on multiple remote
|
|
1492
|
+
servers. The location with download index of 0 is tried first. If missed, the
|
|
1493
|
+
second location, index 1, is tried and so on.
|
|
1494
|
+
|
|
1495
|
+
-DT or -DataTime, times, in form of YYY-MM-DD HH:NN:SS, for data being
|
|
1496
|
+
updated upto. They are used for dependencies of the control records.
|
|
1497
|
+
The dependencies of update controls are set via Info option -PI (-ParentIndex).
|
|
1498
|
+
If parent index of an update control record is not empty, the control record
|
|
1499
|
+
for an update period will be blocked until the data time of the parent
|
|
1500
|
+
control record shows it is done for that period.
|
|
1501
|
+
|
|
1502
|
+
-EC or -ErrorControl, a single letter option for error controls to set in
|
|
1503
|
+
update control records:
|
|
1504
|
+
- Set it as I to Ignore errors and continue finish multiple period updates,
|
|
1505
|
+
it equivalents to Mode option -IE;
|
|
1506
|
+
- Set it as Q to force quit on any errors, it equivalents to Mode option -QE;
|
|
1507
|
+
- Set it to N for treating errors normally, that is to stop a multiple period
|
|
1508
|
+
updates when an error is encountered, but allow process continue for next
|
|
1509
|
+
local file update if multiple local files are under a single update control
|
|
1510
|
+
process.
|
|
1511
|
+
|
|
1512
|
+
-ED or -EndDate, for data end date of next data update period.
|
|
1513
|
+
|
|
1514
|
+
-EH or -EndHour, for end hour of next data update period if hourly controlled.
|
|
1515
|
+
|
|
1516
|
+
The dependencies of local indices are set via Info option -PI (-ParentIndex).
|
|
1517
|
+
If parent index of a local file record is not empty, the local file record
|
|
1518
|
+
for end date, and hour if hourly update, will be blocked until the end data
|
|
1519
|
+
date and hour of the parent local file record shows it is done for that end
|
|
1520
|
+
date and hour.
|
|
1521
|
+
|
|
1522
|
+
-EP or -EndPeriod, specifies the end time of a given update period determined
|
|
1523
|
+
from the given update frequency per -FQ (-Frequency). Default value of 0 means
|
|
1524
|
+
that the end time should be the end of update period, i.e., for march of 2007
|
|
1525
|
+
of a monthly update data record, the end time should be exactly '2007-03-31'.
|
|
1526
|
+
|
|
1527
|
+
-ET or -EndTime, used for remote file records to specify ending times.
|
|
1528
|
+
Combined with the time interval given per Info option -TI, it is used to set
|
|
1529
|
+
the ending time of creating multiple remote file names based on the
|
|
1530
|
+
temporal pattern of the file name in each update period. It is defaulted to 0,
|
|
1531
|
+
which means the last available time during a update period.
|
|
1532
|
+
|
|
1533
|
+
Ending time can be set as D:D:D:D:D:D:D:D:D:D:D:D for 12 offset values for
|
|
1534
|
+
setting ending days of each month in a year, if they are not the last days
|
|
1535
|
+
of each month of a monthly controlled update.
|
|
1536
|
+
|
|
1537
|
+
-FQ or -Frequency, data update and control frequencies are time internals in
|
|
1538
|
+
format of NU, N - integer value and U - a single letter unit measure, i.e.,
|
|
1539
|
+
1Y, 1M, 1W, 5D, 6H. Frequency of fraction of a month is supported in format
|
|
1540
|
+
of NU/F. For example, 1M/3 means the update frequency as one third of a month,
|
|
1541
|
+
so that 10 days for the first two periods (1 - 10 and 11 - 20) of each month,
|
|
1542
|
+
and 8 to 11 days for the third period (21 - End Of Month) depending on how many
|
|
1543
|
+
days in a month.
|
|
1544
|
+
|
|
1545
|
+
-FA or -FileArchived, specifies an archived file name for data on RDA
|
|
1546
|
+
Server. Archived file names are normally created automatically, but they
|
|
1547
|
+
can be specified otherwise per this option.
|
|
1548
|
+
|
|
1549
|
+
-GP or -GenericPattern (Alias: -GeneralPattern), generic pattern option for
|
|
1550
|
+
pattern values to be provided on command line. The first value is for
|
|
1551
|
+
replacing pattern '<P0>', the second value for '<P1>', and so on.
|
|
1552
|
+
|
|
1553
|
+
If 2 patterns are required for matching, <P0> and <P1>, n*2 generic pattern
|
|
1554
|
+
values are allowed on command line to loop n times for pattern matching. Here
|
|
1555
|
+
n is numeric value and n > 0.
|
|
1556
|
+
|
|
1557
|
+
-HN or -HostName, host machine names to set into update control records. One
|
|
1558
|
+
or multiple host names can be set into an update control record. The hostname
|
|
1559
|
+
information in update control records limits the associated update actions can,
|
|
1560
|
+
or cannot, be processed on the specified machines.
|
|
1561
|
+
|
|
1562
|
+
For example, set update control record of Control Index 1 of d277000 to
|
|
1563
|
+
control its associated update actions to be processed on rda-work only
|
|
1564
|
+
|
|
1565
|
+
dsupdt SC -CI 1 -HN rda-work
|
|
1566
|
+
|
|
1567
|
+
-HO or -HourOffset, works with Mode option -CN (-CheckNew) to provide a time zone
|
|
1568
|
+
offset value in hours to compare the timestamp of a server file to its local
|
|
1569
|
+
copy. For example, 2 means the server time zone is 2 hours ahead of the local time;
|
|
1570
|
+
while -2 means the server time zone is 2 hours behind the local time. It can be
|
|
1571
|
+
set into update control record via this Info option for action -SC (-SetControl).
|
|
1572
|
+
|
|
1573
|
+
-ID or -ControlID, a unique string to identify a Update control record.
|
|
1574
|
+
|
|
1575
|
+
-IF or -InputFile, for input file names; one or multiple file names may be
|
|
1576
|
+
given on command line. A input file name must starts with dataset number as
|
|
1577
|
+
in format of 'dsnnn.n.*' and the dataset number must match the dataset number
|
|
1578
|
+
given per option -DS (-Dataset). This restriction prevents specialists archiving
|
|
1579
|
+
data files accidentally into a wrong dataset. Input files are used to hold
|
|
1580
|
+
valid options and the associated values of Info options that need to be passed
|
|
1581
|
+
in for execution of 'dsupdt'.
|
|
1582
|
+
|
|
1583
|
+
In a input file, lines start with sign '#' are considered as comments;
|
|
1584
|
+
Option Names can be given either short, long or alias names. Action and Mode
|
|
1585
|
+
options are given in format of OptionName<!>. Single value Assignment is
|
|
1586
|
+
given in format of OptionName<=>OptionValue. One option is given on each line.
|
|
1587
|
+
Different setting sign of Action and Mode options can be provided by Info
|
|
1588
|
+
option -AO (-ActOption, default to <!>); and different equal sign of single
|
|
1589
|
+
value assignment can be provided by Info option -ES, (-EqualSign, default to
|
|
1590
|
+
'<=>'). Multi-value assignments can be given in columns delimited with
|
|
1591
|
+
separator specified per option -SP (-Separator, default to '<:>'). It starts
|
|
1592
|
+
with a column title line for multi-value option names and the rest holds
|
|
1593
|
+
values corresponding to each column titles. The value information stops at
|
|
1594
|
+
the end of the file or when a new column name line or another single value
|
|
1595
|
+
assignment appears. If the last column is a multi-line value field, an
|
|
1596
|
+
additional separator must be appended for each line, including the column
|
|
1597
|
+
title line to end lines properly.
|
|
1598
|
+
|
|
1599
|
+
-KF or -KeepFile, single letter options set in update control records for
|
|
1600
|
+
keeping files on local disks after data update finished:
|
|
1601
|
+
- Set S to keep server files retrieved remotely, it equivalents to Mode option -KS;
|
|
1602
|
+
- Set R to keep remote files after they are built into local files, it equivalents
|
|
1603
|
+
to Mode option -KR;
|
|
1604
|
+
- Set B to keep both, it equivalents to combination of Mode options -KS and -KR.
|
|
1605
|
+
- Set N for not keeping either.
|
|
1606
|
+
|
|
1607
|
+
-LF or -LocalFile, local file names for set update local file records and
|
|
1608
|
+
are used later for building up local data files and archiving them onto RDA
|
|
1609
|
+
Server. If there is any temporal patterns in the given file names, the
|
|
1610
|
+
patterns are replaced by the times evaluated from the end date/hour of
|
|
1611
|
+
the update periods.
|
|
1612
|
+
|
|
1613
|
+
A different local file name can be provided at the update run time if a local
|
|
1614
|
+
file index is provided.
|
|
1615
|
+
|
|
1616
|
+
-LI or -LocalIndex (Alias: -LocalFileIndx), indices for
|
|
1617
|
+
set update local file records. Locfile index can be provided for update of
|
|
1618
|
+
individual local file record.
|
|
1619
|
+
|
|
1620
|
+
-MC or -EMailControl, a single letter option for email controls set in update
|
|
1621
|
+
control records:
|
|
1622
|
+
- Set E to send emails only when error happens during update actions,
|
|
1623
|
+
it equivalents to Mode option -EE;
|
|
1624
|
+
- Set N to send no email at all, it equivalents to Mode option -NE;
|
|
1625
|
+
- Set S to send summary emails without detail, it equivalents to Mode option -SE;
|
|
1626
|
+
- Set A for always sending detail emails;
|
|
1627
|
+
- Set B for E + S, it equivalents to combination of Mode options -SE and -EE,
|
|
1628
|
+
which means send summary emails only when error happens.
|
|
1629
|
+
|
|
1630
|
+
-MR or -MissRemote, if set to 'Y', allows missing one of some remote
|
|
1631
|
+
files of the multiple remote files needed to build a single local file. Value
|
|
1632
|
+
'Y' of this option means differently if the value of option -VI
|
|
1633
|
+
(-ValidInterval) is not empty; that download effort is not given up for the
|
|
1634
|
+
remote data files until they are out of the valid periods defined by the given
|
|
1635
|
+
valid intervals.
|
|
1636
|
+
|
|
1637
|
+
-DI or -DueInterval, the data due interval between data end date/hour and the
|
|
1638
|
+
date/hour the data file file ready for update, which is set in local file record.
|
|
1639
|
+
The next update date/hour of a local file is evaluated by adding the due interval
|
|
1640
|
+
to the data end date/hour.
|
|
1641
|
+
|
|
1642
|
+
-OP or -Options (Alias: -DsarchOption), a string, upto 128 characters, passed
|
|
1643
|
+
to 'dsarch' for additional Info and Mode options to archiving data files.
|
|
1644
|
+
Group index or group can be passed to 'dsarch'. Temporal patterns can be
|
|
1645
|
+
setup into the group index or name values to dynamically identify group
|
|
1646
|
+
information for the data files being updated.
|
|
1647
|
+
|
|
1648
|
+
-PD or -PatternDelimiter, pattern delimiters, default to ["<", ">"],
|
|
1649
|
+
are used to hold temporal or generic patterns in file names and any other
|
|
1650
|
+
text fields. Normally the data end date/hour are used for the temporal
|
|
1651
|
+
patterns, but current date/hour will be used if the pattern is both started
|
|
1652
|
+
and ended with letter 'C', as in <CYYYY.MM.DD.HHC> for example. Beginning
|
|
1653
|
+
date/hour of update period are used if the pattern is both started and ended
|
|
1654
|
+
with letter 'B', as in <BYYYY.MM.DD.HHB> for example. If fractional month is
|
|
1655
|
+
used for update frequency, pattern as <M*M> is supported to get different file
|
|
1656
|
+
names for each fraction; '* == C' means up case letters: A, B, C, ...,
|
|
1657
|
+
'* == c' means lower case letters: a, b, c, ..., and '* == N' means numeric
|
|
1658
|
+
values: 1, 2, 3, ...
|
|
1659
|
+
|
|
1660
|
+
Generic patterns can be specified per option -GP (-GenericPattern) on command
|
|
1661
|
+
line while execute 'dsupdt' for update actions. For a given local file name
|
|
1662
|
+
as '<P0>.txt', the pattern value must be provided on the command line via
|
|
1663
|
+
option -GP (-GenericPattern) as the first option value; otherwise a fatal
|
|
1664
|
+
error happens. Multiple generic pattern matches can be provided the way.
|
|
1665
|
+
|
|
1666
|
+
-PI or -ParentIndex, parent update control or local indices for data update
|
|
1667
|
+
dependencies. Set parent indices for update control records, if the control
|
|
1668
|
+
records need the parent controls be processed first; set parent local index
|
|
1669
|
+
for local file records, if the the local file records need the parent local
|
|
1670
|
+
files be build and archived first.
|
|
1671
|
+
|
|
1672
|
+
-PR or -ProcessRemote, external command specified by individual specialists
|
|
1673
|
+
for additional processes to validate and convert remote files after they are
|
|
1674
|
+
successfully downloaded.
|
|
1675
|
+
|
|
1676
|
+
Options allowed passing information into the command include -LF, -RF, -SF,
|
|
1677
|
+
-DS, -ED, -EH, -SN/LN, and -LI. Each individual option is replaced by a real
|
|
1678
|
+
value at the time 'dsupdt' calls the command.
|
|
1679
|
+
|
|
1680
|
+
If you send email inside your process command and want the email being held
|
|
1681
|
+
until the update action of 'dsupdt' is finished successfully, you can use
|
|
1682
|
+
option -LI to pass the local file index to you command, and save a customized
|
|
1683
|
+
email into field of dlupdt.emnote in the local file record. If you want the
|
|
1684
|
+
email being held until multiple local file records being updated successfully,
|
|
1685
|
+
you can pass to your command the local file index value of the last local file
|
|
1686
|
+
record. The saved customized email will be sent out only the update action is
|
|
1687
|
+
completed successfully for all local file records.
|
|
1688
|
+
|
|
1689
|
+
-QS or -QsubOptions, (Alias: -PBSOptions), specifies options to execute dsupdt
|
|
1690
|
+
as a batch job via qsub on PBS nodes. The qsub options must be quoted when prsented
|
|
1691
|
+
on command line, such as, -QS '-l walltime=12:00:00'.
|
|
1692
|
+
|
|
1693
|
+
-RF or -RemoteFile, for remote file names to set update remote file records
|
|
1694
|
+
and are used later for download remote data files and building local files.
|
|
1695
|
+
If there is any temporal patterns in the given file names, the patterns are
|
|
1696
|
+
replaced by the times evaluated from the end date/hour of the update
|
|
1697
|
+
periods. Local file names are used if remote file names are not specified.
|
|
1698
|
+
|
|
1699
|
+
For simple situation that a remote file is the same as local file,
|
|
1700
|
+
information of the local file record is enough to process data update. A
|
|
1701
|
+
remote file update record is only required if the remote file is different
|
|
1702
|
+
from local file, including situations that multiple remote files are
|
|
1703
|
+
downloaded and tarred for a single local file, or a single remote file is
|
|
1704
|
+
available from multiple servers.
|
|
1705
|
+
|
|
1706
|
+
Multiple remote files can be provided in a single remote file record by joining
|
|
1707
|
+
them with separator '::', as Rfile1::Rfile2::Rfile3 for example.
|
|
1708
|
+
|
|
1709
|
+
For remote file name with a leading '!' it is treated as an executable command
|
|
1710
|
+
and its returned string is used as a dynamically created remote file name.
|
|
1711
|
+
|
|
1712
|
+
-RI or -RetryInterval, set the number of days or hours to retry the update
|
|
1713
|
+
control process when first try failed.
|
|
1714
|
+
|
|
1715
|
+
-RO or -ResetOrder (Alias: -Reorder), when present, resets the execute orders
|
|
1716
|
+
of the given local file lists as the orders the update records are given for
|
|
1717
|
+
actions -SL (-SetLocalFile). Another way of reordering the local files is to
|
|
1718
|
+
provide explicitly order index values per Info option -XO (-ExecOrder).
|
|
1719
|
+
|
|
1720
|
+
-SB or -SbatchOptions, (Alias: -SlurmOptions), specifies options to execute dsupdt
|
|
1721
|
+
as a batch job via sbatch on SLURM nodes. The sbatch options must be quoted when
|
|
1722
|
+
prsented on command line, such as, -QS '-t 24:00:00'.
|
|
1723
|
+
|
|
1724
|
+
-SF or -ServerFile, only used to set file names on Remote servers when they
|
|
1725
|
+
are different from the remote file names.
|
|
1726
|
+
|
|
1727
|
+
-SN or -Specialist, sets login names of specialists into RDADB for update
|
|
1728
|
+
records. It defaults to the login name of a specialist if not proved explicitly.
|
|
1729
|
+
At run time of a data update, specialist login name is validated against saved
|
|
1730
|
+
specialist names and the update process is blocked if them are not match.
|
|
1731
|
+
|
|
1732
|
+
-TI or -TimeInterval, used for remote file records to specify a time increase
|
|
1733
|
+
step. The interval is used to create multiple remote file names based on the
|
|
1734
|
+
temporal pattern of the file name in each update period; starting from the
|
|
1735
|
+
beginning time given per Info option -BT (-BeginTime) to the data end
|
|
1736
|
+
date/hour.
|
|
1737
|
+
|
|
1738
|
+
-UC or -UpdateControl, a string to hold multiple single letter options setting
|
|
1739
|
+
in update control records:
|
|
1740
|
+
- Include B for using begin times of update periods for temporal patterns, it
|
|
1741
|
+
equivalents to Mode option -BT;
|
|
1742
|
+
- Include C for allowing update actions for the current update period although
|
|
1743
|
+
the data end time is not due yet, it equivalents to Mode option -CP;
|
|
1744
|
+
- Include E to reset end date/hour if the file timestamp is newer than end date/hour
|
|
1745
|
+
plus one update frequency, it equivalents to Mode option -RE;
|
|
1746
|
+
- Include F for forcing update actions at least for one update period although
|
|
1747
|
+
data update is not due yet, it equivalents to Mode option -FU;
|
|
1748
|
+
- Include G for using GMT time zone as controlling time, it equivalents to Mode
|
|
1749
|
+
option -GZ;
|
|
1750
|
+
- Include M for update actions against multiple update periods, it equivalents
|
|
1751
|
+
to Mode option -MU;
|
|
1752
|
+
- Include N for checking if any files archived have been changed on remote
|
|
1753
|
+
servers, and executing update actions against only the files got changed,
|
|
1754
|
+
it equivalents to Mode option -CN;
|
|
1755
|
+
- Include O for update actions against the update periods with missing archives
|
|
1756
|
+
only, it equivalents to Mode option -MO;
|
|
1757
|
+
- Include Y for skipping February 29 in leap years while adding days to date,
|
|
1758
|
+
it equivalents to Mode option -NY;
|
|
1759
|
+
- Include Z for considering empty files as valid data files; it equivalents to
|
|
1760
|
+
Info option '-VS 0'.
|
|
1761
|
+
|
|
1762
|
+
-VI or -ValidInterval, sets the number of days or hours the remote files are
|
|
1763
|
+
valid on remote servers. If the time difference between the current date/time
|
|
1764
|
+
and the data date/time is longer than the time internal set in RDADB by this
|
|
1765
|
+
option, 'dsupdt' will not download.
|
|
1766
|
+
|
|
1767
|
+
-WD or -WorkDir, specifies a working directory to hold the temporary data files
|
|
1768
|
+
for the download/archive activities. The leading path of the working directory
|
|
1769
|
+
can be set as an environment variable $UPDTWKP. It is defaulted to
|
|
1770
|
+
'/glade/data02/dsswork' on machines that can write to /glade/data02. A working
|
|
1771
|
+
directory '$UPDTWKP/zji/icoads' means that the temporary downloaded data files
|
|
1772
|
+
are staged in '/glade/data02/dsswork/zji/icoads' during the update. Specify it
|
|
1773
|
+
differently or use a different environment variable name if you want to store
|
|
1774
|
+
the temporary data files some where else. Be aware that the default value for
|
|
1775
|
+
$UPDTWKP is used if you use a different environment variable name without
|
|
1776
|
+
defining it.
|
|
1777
|
+
|
|
1778
|
+
-XC or -ExecuteCommand (Alias: -ExecCmd), external command for additional
|
|
1779
|
+
process after successful update action for a specified update control
|
|
1780
|
+
configuration.
|
|
1781
|
+
|
|
1782
|
+
-XO or -ExecOrder, for executing order indices of local file records of a
|
|
1783
|
+
given a dataset. This Info option is ignored if Mode option -RO (-ResetOrder)
|
|
1784
|
+
is present.
|