@dmptool/utils 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +788 -0
- package/dist/cloudFormation.d.ts +8 -0
- package/dist/cloudFormation.js +54 -0
- package/dist/dynamo.d.ts +105 -0
- package/dist/dynamo.js +651 -0
- package/dist/eventBridge.d.ts +13 -0
- package/dist/eventBridge.js +48 -0
- package/dist/general.d.ts +56 -0
- package/dist/general.js +142 -0
- package/dist/index.d.ts +10 -0
- package/dist/index.js +27 -0
- package/dist/logger.d.ts +17 -0
- package/dist/logger.js +36 -0
- package/dist/maDMP.d.ts +41 -0
- package/dist/maDMP.js +982 -0
- package/dist/maDMPTypes.d.ts +273 -0
- package/dist/maDMPTypes.js +12 -0
- package/dist/rds.d.ts +11 -0
- package/dist/rds.js +108 -0
- package/dist/s3.d.ts +44 -0
- package/dist/s3.js +98 -0
- package/dist/ssm.d.ts +8 -0
- package/dist/ssm.js +33 -0
- package/package.json +66 -0
package/README.md
ADDED
|
@@ -0,0 +1,788 @@
|
|
|
1
|
+
# dmptool-aws
|
|
2
|
+
|
|
3
|
+
Functions that provide AWS functionality for the following DMPTool projects:
|
|
4
|
+
- Lambda Functions in [dmptool-infrastructure](https://github.com/CDLUC3/dmptool-infrastructure)
|
|
5
|
+
- Apollo Server in [dmsp_backend_prototype](https://github.com/CDLUC3/dmsp_backend_prototype)
|
|
6
|
+
- Narrative Generator Service in [dmptool-narrative-generator](https://github.com/CDLUC3/dmptool-narrative-generator)
|
|
7
|
+
|
|
8
|
+
## Installation
|
|
9
|
+
|
|
10
|
+
Add the DMP Utils types to your `package.json` by running `npm add @dmptool/utils`
|
|
11
|
+
|
|
12
|
+
See below for usage examples of each utility.
|
|
13
|
+
|
|
14
|
+
## Table of Contents
|
|
15
|
+
- [AWS CloudFormation Stack Output Access](#cloudformation-support)
|
|
16
|
+
- [AWS DynamoDB Table Access](#dynamodb-support)
|
|
17
|
+
- [General Helper Functions](#general-helper-functions)
|
|
18
|
+
- [EventBridge Event Publication](#eventbridge-support)
|
|
19
|
+
- [Logger Support (Pino with ECS formatting)](#logger-support-pino-with-ecs-formatting)
|
|
20
|
+
- [maDMP Support (serialization and deserialization)](#madmp-support-serialization-and-deserialization)
|
|
21
|
+
- [AWS RDS MySQL Database Access](#rds-mysql-support)
|
|
22
|
+
- [S3 Bucket Access](#s3-support)
|
|
23
|
+
- [SSM Parameter Store Access](#ssm-support)
|
|
24
|
+
|
|
25
|
+
## Development
|
|
26
|
+
|
|
27
|
+
This package is written in TypeScript and uses [Jest](https://jestjs.io/) for testing. Each utility has its own test file.
|
|
28
|
+
|
|
29
|
+
To add a new utility, create a new file in the `src` directory and add it to the `index.ts` file. Be sure to add a corresponding unit test file in the `__tests__` directory.
|
|
30
|
+
|
|
31
|
+
This package is published to DMP under the DMP Tool organization.
|
|
32
|
+
|
|
33
|
+
We use [semantic versioning](https://semver.org/) for versioning. If your changes are backwards compatible, then you can bump the patch version. If you make backwards-incompatible changes, then you can bump the minor version. If you make breaking changes, then you should bump the major version.
|
|
34
|
+
|
|
35
|
+
To publish a new version, update the version in the `package.json` file, then run `npm login` and `npm publish`. Be sure to notify the owners of the repositories listed at the top of this file about your changes.
|
|
36
|
+
|
|
37
|
+
### Dependencies
|
|
38
|
+
|
|
39
|
+
This package has the following dependencies:
|
|
40
|
+
- [@dmptool-types](https://github.com/CDLUC3/dmptool-types) For TypeScript types and Zod schemas to help with maDMP serialization/deserialization.
|
|
41
|
+
- [@elastic/ecs-pino-format](https://www.npmjs.com/package/@elastic/ecs-pino-format) For formatting logs for ECS (OpenSearch).
|
|
42
|
+
- [date-fns](https://www.npmjs.com/package/date-fns) For date manipulation.
|
|
43
|
+
- [jsonschema](https://www.npmjs.com/package/jsonschema) For validating maDMP JSON objects against the RDA Common Metadata Standard.
|
|
44
|
+
- [mysql2](https://www.npmjs.com/package/mysql2) For interacting with the RDS MySQL database.
|
|
45
|
+
- [pino](https://www.npmjs.com/package/pino) For logging.
|
|
46
|
+
- [pino-lambda](https://www.npmjs.com/package/pino-lambda) For setting up automatic request tracing in Pino logs.
|
|
47
|
+
|
|
48
|
+
This package also requires the following AWS dependencies. Note that the Lambda environment preinstalls these so you only need to include them in the `devDependencies` of your `package.json` for that environment.
|
|
49
|
+
- [@aws-sdk/client-cloudformation](https://www.npmjs.com/package/@aws-sdk/client-cloudformation) For interacting with CloudFormation stacks.
|
|
50
|
+
- [@aws-sdk/client-dynamodb](https://www.npmjs.com/package/@aws-sdk/client-dynamodb) For interacting with the RDS MySQL database.
|
|
51
|
+
- [@aws-sdk/client-s3](https://www.npmjs.com/package/@aws-sdk/client-s3) For interacting with S3 buckets.
|
|
52
|
+
- [@aws-sdk/s3-request-presigner](https://www.npmjs.com/package/@aws-sdk/s3-request-presigner) For generating pre-signed URLs for S3 objects.
|
|
53
|
+
- [@aws-sdk/client-sns](https://www.npmjs.com/package/@aws-sdk/client-sns) For publishing messages to SNS topics.
|
|
54
|
+
- [@aws-sdk/client-ssm](https://www.npmjs.com/package/@aws-sdk/client-ssm) For interacting with AWS Systems Manager Parameter Store.
|
|
55
|
+
- [@aws-sdk/util-dynamodb](https://www.npmjs.com/package/@aws-sdk/util-dynamodb) For converting DynamoDB items to JSON objects.
|
|
56
|
+
|
|
57
|
+
Many of these utilities require specific environment variables to be set. See each section below for specifics.
|
|
58
|
+
|
|
59
|
+
## CloudFormation Support
|
|
60
|
+
Provides access to CloudFormation stack outputs.
|
|
61
|
+
|
|
62
|
+
For example, our CloudFormation stack for the S3 buckets outputs the names of each bucket. This code allows a Lambda Function to access those bucket names.
|
|
63
|
+
|
|
64
|
+
Environment variable requirements:
|
|
65
|
+
- `AWS_REGION` The AWS region where the DynamoDB table is located
|
|
66
|
+
|
|
67
|
+
### Example usage
|
|
68
|
+
```typescript
|
|
69
|
+
import { getExport } from '@dmptool/dmptool-utils';
|
|
70
|
+
|
|
71
|
+
const tableName = await getExport('DynamoTableNames');
|
|
72
|
+
console.log(tableName);
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## DynamoDB Support
|
|
76
|
+
|
|
77
|
+
This code can be used to access maDMP data stored in the DynamoDB table.
|
|
78
|
+
|
|
79
|
+
It supports:
|
|
80
|
+
- Checking if a DMP exists (tombstoned DMPs are considered non-existent)
|
|
81
|
+
- Retrieving a DMP by ID and version
|
|
82
|
+
- Retrieving all versions timestamps for a DMP ID
|
|
83
|
+
- Creating a DMP
|
|
84
|
+
- Updating a DMP
|
|
85
|
+
- Tombstoning a DMP
|
|
86
|
+
- Deleting a DMP
|
|
87
|
+
|
|
88
|
+
The code handles the logic to determine the correct DynamoDB partition key (PK) and sort key (SK). It also handles the separation of the RDA Common Standard portion of the DMP JSON from the DMP Tool extensions and stores them separately in the DynamoDB table to help prevent us from performing large reads when it is not necessary.
|
|
89
|
+
|
|
90
|
+
A DMP PK looks like this: `DMP#doi.org/11.12345/A6B7C9D0`
|
|
91
|
+
A DMP SK for the current version of the RDA Common Standard looks like this: `VERSION#latest`
|
|
92
|
+
A DMP SK for the current version of the DMP Tool extensions looks like this: `EXTENSION#latest`
|
|
93
|
+
A DMP SK for a specific version of the RDA Common Standard looks like this: `VERSION#2025-11-21T13:41:32.000Z`
|
|
94
|
+
A DMP SK for a specific version of the DMP Tool extensions looks like this: `EXTENSION#2025-11-21T13:41:32.000Z`
|
|
95
|
+
|
|
96
|
+
These keys are attached to the DMP JSON when persisting it to DynamoDB and removed when returning it from DynamoDB.
|
|
97
|
+
|
|
98
|
+
Environment variable requirements:
|
|
99
|
+
- `AWS_REGION` The AWS region where the DynamoDB table is located
|
|
100
|
+
- `DOMAIN_NAME` The domain name of the application
|
|
101
|
+
- `DYNAMODB_TABLE_NAME` The name of the DynamoDB table
|
|
102
|
+
- `DYNAMO_MAX_ATTEMPTS` The maximum number of times to retry a DynamoDB operation (defaults to 3)
|
|
103
|
+
- `VERSION_GRACE_PERIOD` The number of seconds to wait before considering a change should generate a version snapshot (defaults to 7200000 => 2 hours)
|
|
104
|
+
|
|
105
|
+
## Example Usage:
|
|
106
|
+
```typescript
|
|
107
|
+
import { DMPToolDMPType } from '@dmptool/types';
|
|
108
|
+
import {
|
|
109
|
+
createDMP,
|
|
110
|
+
deleteDMP,
|
|
111
|
+
DMPExists,
|
|
112
|
+
DMP_LATEST_VERSION,
|
|
113
|
+
getDMPs,
|
|
114
|
+
getDMPVersions,
|
|
115
|
+
tombstoneDMP,
|
|
116
|
+
updateDMP
|
|
117
|
+
} from 'dmptool-dynamo';
|
|
118
|
+
|
|
119
|
+
process.env.AWS_REGION = 'eu-west-1';
|
|
120
|
+
process.env.DOMAIN_NAME = 'my-application.org';
|
|
121
|
+
process.env.DYNAMODB_TABLE_NAME = 'my-dynamo-table';
|
|
122
|
+
|
|
123
|
+
const dmpId = '123456789';
|
|
124
|
+
|
|
125
|
+
const dmpObj: DMPToolDMPType = {
|
|
126
|
+
dmp: {
|
|
127
|
+
title: 'Test DMP',
|
|
128
|
+
dmp_id: {
|
|
129
|
+
identifier: dmpId,
|
|
130
|
+
type: 'other'
|
|
131
|
+
},
|
|
132
|
+
created: '2021-01-01 03:11:23Z',
|
|
133
|
+
modified: '2021-01-01 02:23:11Z',
|
|
134
|
+
ethical_issues_exist: 'unknown',
|
|
135
|
+
language: 'eng',
|
|
136
|
+
contact: {
|
|
137
|
+
name: 'Test Contact',
|
|
138
|
+
mbox: 'tester@example.com',
|
|
139
|
+
contact_id: [{
|
|
140
|
+
identifier: '123456789',
|
|
141
|
+
type: 'other'
|
|
142
|
+
}]
|
|
143
|
+
},
|
|
144
|
+
dataset: [{
|
|
145
|
+
title: 'Test Dataset',
|
|
146
|
+
dataset_id: {
|
|
147
|
+
identifier: 'your-application.projects.123.dmp.12.outputs.1',
|
|
148
|
+
type: 'other'
|
|
149
|
+
},
|
|
150
|
+
personal_data: 'unknown',
|
|
151
|
+
sensitive_data: 'no',
|
|
152
|
+
}],
|
|
153
|
+
rda_schema_version: "1.2",
|
|
154
|
+
provenance: 'your-application',
|
|
155
|
+
status: 'draft',
|
|
156
|
+
privacy: 'private',
|
|
157
|
+
featured: 'no',
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// First make sure the DMP doesn't already exist
|
|
162
|
+
const exists = await DMPExists(dmpId);
|
|
163
|
+
if (exists) {
|
|
164
|
+
console.log('DMP already exists');
|
|
165
|
+
|
|
166
|
+
} else {
|
|
167
|
+
// Create the DMP
|
|
168
|
+
const created: DMPToolDMPType = await createDMP(dmpId, dmpObj);
|
|
169
|
+
if (!created) {
|
|
170
|
+
console.log('Failed to create DMP');
|
|
171
|
+
|
|
172
|
+
} else {
|
|
173
|
+
dmpObj.dmp.privacy = 'public';
|
|
174
|
+
dmpObj.dmp.modified = '2026-01-10T03:43:11Z';
|
|
175
|
+
|
|
176
|
+
// Update the DMP
|
|
177
|
+
const updated: DMPToolDMPType = await updateDMP(dmpObj);
|
|
178
|
+
if (!updated) {
|
|
179
|
+
console.log('Failed to update DMP');
|
|
180
|
+
|
|
181
|
+
} else {
|
|
182
|
+
// Fetch the DMP version timestamps (should only be two)
|
|
183
|
+
const versions = await getDMPVersions(dmpId);
|
|
184
|
+
console.log(versions);
|
|
185
|
+
|
|
186
|
+
// Fetch the latest version of the DMP
|
|
187
|
+
const latest = await getDMP(dmpId, DMP_LATEST_VERSION);
|
|
188
|
+
if (!latest) {
|
|
189
|
+
console.log('Failed to fetch latest version of DMP');
|
|
190
|
+
|
|
191
|
+
} else {
|
|
192
|
+
// If the DMP has a `registered` timestamp then it is published and can be tombstoned not deleted
|
|
193
|
+
// Since our example is not, we include this code here for reference only
|
|
194
|
+
|
|
195
|
+
// const tombstoned = await tombstoneDMP(dmpId);
|
|
196
|
+
// console.log( tombstoned
|
|
197
|
+
|
|
198
|
+
// Delete the DMP (can be done because the DMP is not published)
|
|
199
|
+
const deleted = await deleteDMP(dmpId);
|
|
200
|
+
console.log(deleted);
|
|
201
|
+
}
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
```
|
|
206
|
+
## EventBridge Support
|
|
207
|
+
|
|
208
|
+
This code can be used to publish events to the EventBridge.
|
|
209
|
+
|
|
210
|
+
Environment variable requirements:
|
|
211
|
+
- `AWS_REGION` - The AWS region where the Lambda Function is running
|
|
212
|
+
- `EVENTBRIDGE_BUS_NAME` - The ARN of the EventBridge Bus to publish events to
|
|
213
|
+
|
|
214
|
+
### Example usage
|
|
215
|
+
```typescript
|
|
216
|
+
import { publishMessage } from '@dmptool/utils';
|
|
217
|
+
|
|
218
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
219
|
+
|
|
220
|
+
const topicArn = 'arn:aws:sns:us-east-1:123456789012:my-topic';
|
|
221
|
+
|
|
222
|
+
// See the documentation for the AWS Lambda you are trying to invoke to determine what the
|
|
223
|
+
// `detail-type` and `detail` payload should look like.
|
|
224
|
+
const message = {
|
|
225
|
+
'detail-type': 'my-event',
|
|
226
|
+
detail: {
|
|
227
|
+
property1: 'value1',
|
|
228
|
+
property2: 'value2'
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const response = await publishMessage(
|
|
233
|
+
message,
|
|
234
|
+
topicArn
|
|
235
|
+
);
|
|
236
|
+
|
|
237
|
+
if (response.statusCode === 200) {
|
|
238
|
+
console.log('Message published successfully', response.body);
|
|
239
|
+
} else {
|
|
240
|
+
console.log('Error publishing message', response.body);
|
|
241
|
+
}
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
## General Helper Functions
|
|
245
|
+
|
|
246
|
+
Generic helper functions:
|
|
247
|
+
- `areEqual`: Compares two values for equality (including deep equality for objects and arrays)
|
|
248
|
+
- `convertMySQLDateTimeToRFC3339`: Converts a MySQL datetime string to RFC3339 format
|
|
249
|
+
- `currentDateAsString`: Returns the current date as a string in YYYY-MM-DD format
|
|
250
|
+
- `isNullOrUndefined`: Checks if a value is null or undefined
|
|
251
|
+
- `normaliseHttpProtocol`: Normalizes the protocol of a URL to either http or https
|
|
252
|
+
- `randomHex`: Generates a random hex string of a specified length
|
|
253
|
+
- `removeNullAndUndefinedFromObject`: Removes all null and undefined values from an object.
|
|
254
|
+
|
|
255
|
+
Environment variable requirements:
|
|
256
|
+
- NONE
|
|
257
|
+
|
|
258
|
+
### Example usage
|
|
259
|
+
```typescript
|
|
260
|
+
import {
|
|
261
|
+
areEqual,
|
|
262
|
+
convertMySQLDateTimeToRFC3339,
|
|
263
|
+
currentDateAsString,
|
|
264
|
+
isNullOrUndefined,
|
|
265
|
+
normaliseHttpProtocol,
|
|
266
|
+
randomHex,
|
|
267
|
+
removeNullAndUndefinedFromObject,
|
|
268
|
+
} from "dmptool-general";
|
|
269
|
+
|
|
270
|
+
console.log(areEqual("foo", "foo")); // Returns true
|
|
271
|
+
console.log(areEqual(123, "123")); // Returns false
|
|
272
|
+
console.log(areEqual("foo", undefined)); // Returns false
|
|
273
|
+
console.log(areEqual(["foo"], ["foo", "bar"])); // Returns false
|
|
274
|
+
console.log(areEqual({ a: "foo", c: "bar" }, { c: "bar", a: "foo" })); // Returns true
|
|
275
|
+
console.log(areEqual({ a: "foo", c: "bar" }, { a: "foo", c: { d: "bar" } })); // Returns false
|
|
276
|
+
|
|
277
|
+
console.log(convertMySQLDateTimeToRFC3339("2021-01-01 00:00:00")); // Returns "2021-01-01T00:00:00.000Z"
|
|
278
|
+
|
|
279
|
+
console.log(currentDateAsString()); // Returns "2021-01-01"
|
|
280
|
+
|
|
281
|
+
console.log(isNullOrUndefined(null)); // Returns true
|
|
282
|
+
|
|
283
|
+
console.log(normaliseHttpProtocol("http://www.example.com")); // Returns "https://www.example.com"
|
|
284
|
+
|
|
285
|
+
console.log(randomHex(16)); // Returns something like "a3f2c1b8e4d5f0a1"
|
|
286
|
+
|
|
287
|
+
console.log(removeNullAndUndefinedFromObject({ a: "foo", b: null, c: { d: "bar", e: undefined } }));
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
## Logger Support (Pino with ECS formatting)
|
|
291
|
+
|
|
292
|
+
This code can be used by Lambda Functions to provide access to a Pino logger formatted for ECS.
|
|
293
|
+
|
|
294
|
+
It provides a single `initializeLogger` function that can be used to create a Pino logger with standard formatting and a `LogLevel` enum that contains valid log levels.
|
|
295
|
+
|
|
296
|
+
Environment variable requirements:
|
|
297
|
+
- NONE
|
|
298
|
+
|
|
299
|
+
### Example usage
|
|
300
|
+
```typescript
|
|
301
|
+
import { Logger } from 'pino';
|
|
302
|
+
import { initializeLogger, LogLevel } from '@dmptool/utils';
|
|
303
|
+
|
|
304
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
305
|
+
|
|
306
|
+
const LOG_LEVEL = process.env.LOG_LEVEL?.toLowerCase() || 'info';
|
|
307
|
+
|
|
308
|
+
// Initialize the logger
|
|
309
|
+
const logger: Logger = initializeLogger('GenerateMaDMPRecordLambda', LogLevel[LOG_LEVEL]);
|
|
310
|
+
|
|
311
|
+
// Setup the LambdaRequestTracker for the logger
|
|
312
|
+
const withRequest = lambdaRequestTracker();
|
|
313
|
+
|
|
314
|
+
export const handler: Handler = async (event: EventBridgeEvent<string, EventBridgeDetails>, context: Context) => {
|
|
315
|
+
// Log the incoming event and context
|
|
316
|
+
logger.debug({ event, context }, 'Received event');
|
|
317
|
+
|
|
318
|
+
// Initialize the logger by setting up automatic request tracing.
|
|
319
|
+
withRequest(event, context);
|
|
320
|
+
|
|
321
|
+
logger.info({ log_level: LOG_LEVEL, foo: "bar" }, 'Hello World!');
|
|
322
|
+
}
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
## maDMP Support (serialization and deserialization)
|
|
326
|
+
|
|
327
|
+
This Lambda Layer contains code that fetches data about a Plan from the RDS MySQL database and converts it into a JSON object that conforms to the RDA Common Metadata Standard for DMPs with DMP Tool extensions.
|
|
328
|
+
|
|
329
|
+
Details about the RDA Common Metadata Standard can be found in the JSON examples folder of their [repository](https://github.com/RDA-DMP-Common/RDA-DMP-Common-Standard)
|
|
330
|
+
|
|
331
|
+
**Current RDA Common Metadata Standard Version:** v1.2
|
|
332
|
+
|
|
333
|
+
Environment variable requirements:
|
|
334
|
+
- `AWS_REGION` - The AWS region where the Lambda Function is running
|
|
335
|
+
- `ENV`: The AWS environment (e.g. `dev`, `stg`, `prd`)
|
|
336
|
+
- `APPLICATION_NAME`: The name of your application (NO spaces!, this is used to construct identifier namespaces)
|
|
337
|
+
- `DOMAIN_NAME`: The domain name of your application
|
|
338
|
+
|
|
339
|
+
### Notes
|
|
340
|
+
|
|
341
|
+
**DMP IDs:**
|
|
342
|
+
Every Plan in the RDS MySQL database has a `dmpId` defined. These values are DOIs once they have been registered/minted with DataCite/EZID.
|
|
343
|
+
|
|
344
|
+
If a Plan has a `registered` date, then the `dmp_id` in the JSON object will be a DOI (e.g. `{ "identifier": "https://doi.org/11.22222/C3PO}", "type": "doi" }`). The DOI will resolve to the DMP's landing page.
|
|
345
|
+
If not, then the `dmp_id` will be the URL to access the Plan in the DMP Tool (e.g. `https://your-domain.com/projects/123/dmp/12`).
|
|
346
|
+
|
|
347
|
+
**Privacy:**
|
|
348
|
+
The `privacy` property in the JSON object represents the privacy level set in the DMP Tool. This should be used to determine whether the caller has access to the entire DMP (e.g. the narrative). Please adhere to this value when accessing the DMP.
|
|
349
|
+
|
|
350
|
+
**Datasets:**
|
|
351
|
+
The `dataset` property in the JSON object represents the Research Outputs associated with the DMP. If the DMP has no Research Outputs, the JSON object qill contain a default generic Dataset (see minimal JSON example below). This is included because the RDA Common Standard requires a Dataset to be present in the JSON object.
|
|
352
|
+
|
|
353
|
+
**Other IDs:**
|
|
354
|
+
The `project_id` and `dataset_id` properties in the JSON object are constructed in a manner that uses namespacing to allow them to be unique across all DMP systems (e.g. DMP Tool, DSW, DMP Online, etc.). These ids also allow us to tie them back to the records in the RDS MySQL database.
|
|
355
|
+
|
|
356
|
+
For example `your-domain.com.projects.123.dmp.12` ties to Project `123` and Plan `12` in the RDS MySQL database.
|
|
357
|
+
|
|
358
|
+
In situations where an identifier would normally resolve to a repository record (e.g. ROR, ORCID, re3data, etc.) and no value is found in the RDS MySQL database, we construct one that is unique and can be tied back to the record in the RDS MySQL database. For example: `your-application.projects.123.dmp.12.members.1` would be a unique identifier for a member of Project `123`, Plan `12` and PlanMember `1` in the RDS MySQL database.
|
|
359
|
+
|
|
360
|
+
**Ethical Issues:**
|
|
361
|
+
The `ethical_issues_exist` property in the JSON object is only set to `unknown` if the Plan has no defined Research Outputs OR the Ressearch Outputs do not capture the `personal_data` or `sensitive_data` properties. Otherwise, those properties determine whether the DMP contains ethical issues.
|
|
362
|
+
|
|
363
|
+
**Narrative:**
|
|
364
|
+
The `narrative` property in the JSON object represents the Template, Sections, Question text and Answers to the DMP within the DMP Tool.
|
|
365
|
+
|
|
366
|
+
### Example usage
|
|
367
|
+
```typescript
|
|
368
|
+
import { DMPToolDMPType } from '@dmptool/types';
|
|
369
|
+
import { planToDMPCommonStandard } from '@dmptool/utils';
|
|
370
|
+
|
|
371
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
372
|
+
process.env.ENV = 'stg';
|
|
373
|
+
process.env.APPLICATION_NAME = 'your-application';
|
|
374
|
+
process.env.DOMAIN_NAME = 'your-domain.com';
|
|
375
|
+
|
|
376
|
+
const planId = '12345';
|
|
377
|
+
const dmp: DMPToolDMPType = await planToDMPCommonStandard(planId);
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
## Example of a minimal JSON object:
|
|
381
|
+
```
|
|
382
|
+
{
|
|
383
|
+
dmp: {
|
|
384
|
+
# RDA Common Standard properties:
|
|
385
|
+
title: 'Test DMP',
|
|
386
|
+
dmp_id: {
|
|
387
|
+
identifier: 'https://your-domain.com/projects/123/dmp/12',
|
|
388
|
+
type: 'other'
|
|
389
|
+
},
|
|
390
|
+
created: '2021-01-01 03:11:23Z',
|
|
391
|
+
modified: '2021-01-01 02:23:11Z',
|
|
392
|
+
ethical_issues_exist: 'unknown',
|
|
393
|
+
language: 'eng',
|
|
394
|
+
contact: {
|
|
395
|
+
name: 'Test Contact',
|
|
396
|
+
mbox: 'tester@example.com',
|
|
397
|
+
contact_id: [{
|
|
398
|
+
identifier: '123456789',
|
|
399
|
+
type: 'other'
|
|
400
|
+
}]
|
|
401
|
+
},
|
|
402
|
+
dataset: [{
|
|
403
|
+
title: 'Test Dataset',
|
|
404
|
+
dataset_id: {
|
|
405
|
+
identifier: 'your-application.projects.123.dmp.12.outputs.1',
|
|
406
|
+
type: 'other'
|
|
407
|
+
},
|
|
408
|
+
personal_data: 'unknown',
|
|
409
|
+
sensitive_data: 'no',
|
|
410
|
+
}],
|
|
411
|
+
|
|
412
|
+
# DMP Tool extension properties:
|
|
413
|
+
rda_schema_version: "1.2",
|
|
414
|
+
provenance: 'your-application',
|
|
415
|
+
privacy: 'private',
|
|
416
|
+
featured: 'no'
|
|
417
|
+
}
|
|
418
|
+
}
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
## Example of a complete JSON object:
|
|
422
|
+
```
|
|
423
|
+
{
|
|
424
|
+
dmp: {
|
|
425
|
+
# RDA Common Standard properties:
|
|
426
|
+
title: 'Test DMP',
|
|
427
|
+
description: 'This is a test DMP',
|
|
428
|
+
dmp_id: {
|
|
429
|
+
identifier: '123456789',
|
|
430
|
+
type: 'other'
|
|
431
|
+
},
|
|
432
|
+
created: '2021-01-01 03:11:23Z',
|
|
433
|
+
modified: '2021-01-01 02:23:11Z',
|
|
434
|
+
ethical_issues_exist: 'yes',
|
|
435
|
+
ethical_issues_description: 'This DMP contains ethical issues',
|
|
436
|
+
ethical_issues_report: 'https://example.com/ethical-issues-report',
|
|
437
|
+
language: 'eng',
|
|
438
|
+
contact: {
|
|
439
|
+
name: 'Test Contact',
|
|
440
|
+
mbox: 'tester@example.com',
|
|
441
|
+
contact_id: [{
|
|
442
|
+
identifier: 'https://orcid.org/0000-0000-0000-0000',
|
|
443
|
+
type: 'orcid'
|
|
444
|
+
}],
|
|
445
|
+
affiliation: [{
|
|
446
|
+
name: 'Test University',
|
|
447
|
+
affiliation_id: {
|
|
448
|
+
identifier: 'https://ror.org/01234567890',
|
|
449
|
+
type: 'ror'
|
|
450
|
+
}
|
|
451
|
+
}],
|
|
452
|
+
},
|
|
453
|
+
contributor: [{
|
|
454
|
+
name: 'Test Contact',
|
|
455
|
+
contributor_id: [{
|
|
456
|
+
identifier: 'https://orcid.org/0000-0000-0000-0000',
|
|
457
|
+
type: 'orcid'
|
|
458
|
+
}],
|
|
459
|
+
affiliation: [{
|
|
460
|
+
name: 'Test University',
|
|
461
|
+
affiliation_id: {
|
|
462
|
+
identifier: 'https://ror.org/01234567890',
|
|
463
|
+
type: 'ror'
|
|
464
|
+
}
|
|
465
|
+
}],
|
|
466
|
+
role: ['https://example.com/roles/investigation', 'https://example.com/roles/other']
|
|
467
|
+
}],
|
|
468
|
+
cost: [{
|
|
469
|
+
title: 'Budget Cost',
|
|
470
|
+
description: 'Description of budget costs',
|
|
471
|
+
value: 1234.56,
|
|
472
|
+
currency_code: 'USD'
|
|
473
|
+
}],
|
|
474
|
+
dataset: [{
|
|
475
|
+
title: 'Test Dataset',
|
|
476
|
+
type: 'dataset',
|
|
477
|
+
description: 'This is a test dataset',
|
|
478
|
+
dataset_id: {
|
|
479
|
+
identifier: 'your-application.projects.123.dmp.12.outputs.1',
|
|
480
|
+
type: 'other'
|
|
481
|
+
},
|
|
482
|
+
personal_data: 'unknown',
|
|
483
|
+
sensitive_data: 'no',
|
|
484
|
+
data_quality_assurance: ['Statement about data quality assurance'],
|
|
485
|
+
is_reused: false,
|
|
486
|
+
issued: '2026-01-03',
|
|
487
|
+
keyword: ['test', 'dataset'],
|
|
488
|
+
language: 'eng',
|
|
489
|
+
metadata: [{
|
|
490
|
+
description: 'Description of metadata',
|
|
491
|
+
language: 'eng',
|
|
492
|
+
metadata_standard_id: [{
|
|
493
|
+
identifier: 'https://example.com/metadata-standards/123',
|
|
494
|
+
type: 'url'
|
|
495
|
+
}]
|
|
496
|
+
}],
|
|
497
|
+
preservation_statement: 'Statement about preservation',
|
|
498
|
+
security_and_privacy: [{
|
|
499
|
+
title: 'Security and Privacy Statement',
|
|
500
|
+
description: 'Description of security and privacy statement'
|
|
501
|
+
}],
|
|
502
|
+
alternate_identifier: [{
|
|
503
|
+
identifier: 'https://example.com/dataset/123',
|
|
504
|
+
type: 'url'
|
|
505
|
+
}],
|
|
506
|
+
technical_resource: [{
|
|
507
|
+
name: 'Test Server',
|
|
508
|
+
description: 'This is a test server',
|
|
509
|
+
technical_resource_id: [{
|
|
510
|
+
identifier: 'https://example.com/server/123',
|
|
511
|
+
type: 'url'
|
|
512
|
+
}],
|
|
513
|
+
}],
|
|
514
|
+
distribution: [{
|
|
515
|
+
title: 'Test Distribution',
|
|
516
|
+
description: 'This is a test distribution',
|
|
517
|
+
access_url: 'https://example.com/dataset/123/distribution/123456789',
|
|
518
|
+
download_url: 'https://example.com/dataset/123/distribution/123456789/download',
|
|
519
|
+
byte_size: 123456789,
|
|
520
|
+
format: ['application/zip'],
|
|
521
|
+
data_access: 'open',
|
|
522
|
+
issued: '2026-01-03',
|
|
523
|
+
license: [{
|
|
524
|
+
license_ref: 'https://spdx.org/licenses/CC-BY-4.0.html',
|
|
525
|
+
start_date: '2026-01-03'
|
|
526
|
+
}],
|
|
527
|
+
host: {
|
|
528
|
+
title: 'Test Host',
|
|
529
|
+
description: 'This is a test host',
|
|
530
|
+
url: 'https://re3data.org/2784y97245792756789',
|
|
531
|
+
host_id: [{
|
|
532
|
+
identifier: 'https://re3data.org/2784y97245792756789',
|
|
533
|
+
type: 'url'
|
|
534
|
+
}],
|
|
535
|
+
availability: '99.99',
|
|
536
|
+
backup_frequency: 'weekly',
|
|
537
|
+
backup_type: 'tapes',
|
|
538
|
+
certified_with: 'coretrustseal',
|
|
539
|
+
geo_location: 'US',
|
|
540
|
+
pid_system: ['doi', 'ark'],
|
|
541
|
+
storage_type: 'LTO-8 tape',
|
|
542
|
+
support_versioning: 'yes'
|
|
543
|
+
}
|
|
544
|
+
}]
|
|
545
|
+
}],
|
|
546
|
+
related_identifier: [{
|
|
547
|
+
identifier: 'https://doi.org/10.1234/dmp.123456789',
|
|
548
|
+
relation_type: 'cites',
|
|
549
|
+
resource_type: 'dataset',
|
|
550
|
+
type: 'doi'
|
|
551
|
+
}],
|
|
552
|
+
alternate_identifier: [{
|
|
553
|
+
identifier: 'https://example.com/dmp/123456789',
|
|
554
|
+
type: 'url'
|
|
555
|
+
}],
|
|
556
|
+
},
|
|
557
|
+
project: [{
|
|
558
|
+
title: 'Test Project',
|
|
559
|
+
description: 'This is a test project',
|
|
560
|
+
project_id: [{
|
|
561
|
+
identifier: 'your-application.projects.123.dmp.12',
|
|
562
|
+
type: 'other'
|
|
563
|
+
}],
|
|
564
|
+
start: '2025-01-01',
|
|
565
|
+
end: '2028-01-31',
|
|
566
|
+
funding: [{
|
|
567
|
+
name: 'Funder Organization',
|
|
568
|
+
funding_status: 'granted',
|
|
569
|
+
funder_id: {
|
|
570
|
+
identifier: 'https://ror.org/0987654321',
|
|
571
|
+
type: 'ror'
|
|
572
|
+
},
|
|
573
|
+
grant_id: [{
|
|
574
|
+
identifier: '123456789',
|
|
575
|
+
type: 'other'
|
|
576
|
+
}]
|
|
577
|
+
}]
|
|
578
|
+
}],
|
|
579
|
+
|
|
580
|
+
# DMP Tool extension properties:
|
|
581
|
+
rda_schema_version: "1.2",
|
|
582
|
+
provenance: 'your-application',
|
|
583
|
+
privacy: 'private',
|
|
584
|
+
featured: 'no',
|
|
585
|
+
registered: '2026-01-01T10:32:45Z',
|
|
586
|
+
research_domain: {
|
|
587
|
+
name: 'biology',
|
|
588
|
+
research_domain_identifier: {
|
|
589
|
+
identifier: 'https://example.com/01234567',
|
|
590
|
+
type: 'url'
|
|
591
|
+
}
|
|
592
|
+
},
|
|
593
|
+
research_facility: [{
|
|
594
|
+
name: 'Super telescope',
|
|
595
|
+
type: 'observatory',
|
|
596
|
+
research_facility_identifier: {
|
|
597
|
+
identifier: 'https://example.com/01234567',
|
|
598
|
+
type: 'url'
|
|
599
|
+
}
|
|
600
|
+
}],
|
|
601
|
+
funding_opportunity: [{
|
|
602
|
+
# Used to tie the opportunity_identifier to a project[0].funding[?]
|
|
603
|
+
project_id: {
|
|
604
|
+
identifier: 'your-application.projects.123.dmp.12',
|
|
605
|
+
type: 'other'
|
|
606
|
+
},
|
|
607
|
+
# Used to tie the opportunity_identifier to a project[0].funding[?]
|
|
608
|
+
funder_id: {
|
|
609
|
+
identifier: 'https://ror.org/0987654321',
|
|
610
|
+
type: 'ror'
|
|
611
|
+
},
|
|
612
|
+
opportunity_identifier: {
|
|
613
|
+
identifier: 'https://example.com/01234567',
|
|
614
|
+
type: 'url'
|
|
615
|
+
}
|
|
616
|
+
}],
|
|
617
|
+
funding_project: [{
|
|
618
|
+
# Used to tie the opportunity_identifier to a project[0].funding[?]
|
|
619
|
+
project_id: {
|
|
620
|
+
identifier: 'your-application.projects.123.dmp.12',
|
|
621
|
+
type: 'other'
|
|
622
|
+
},
|
|
623
|
+
funder_id: {
|
|
624
|
+
identifier: 'https://ror.org/0987654321',
|
|
625
|
+
type: 'ror'
|
|
626
|
+
},
|
|
627
|
+
project_identifier: {
|
|
628
|
+
identifier: 'https://example.com/erbgierg',
|
|
629
|
+
type: 'url'
|
|
630
|
+
}
|
|
631
|
+
}],
|
|
632
|
+
version: [{
|
|
633
|
+
access_url: 'https://example.com/dmps/123456789?version=2026-01-01T10:32:45Z',
|
|
634
|
+
version: '2026-01-01T10:32:45Z',
|
|
635
|
+
}],
|
|
636
|
+
narrative: {
|
|
637
|
+
# URL to fetch the narrative from the narrative generator (PDF by default but MIME type negotiation is supported)
|
|
638
|
+
download_url: 'https://example.com/dmps/123456789/narrative',
|
|
639
|
+
template: {
|
|
640
|
+
id: 1234567,
|
|
641
|
+
title: 'Narrative Template',
|
|
642
|
+
description: 'This is a test template for a DMP narrative',
|
|
643
|
+
version: 'v1',
|
|
644
|
+
section: [{
|
|
645
|
+
id: 9876,
|
|
646
|
+
title: 'Section one',
|
|
647
|
+
description: 'The first section of the narrative',
|
|
648
|
+
order: 1,
|
|
649
|
+
question: [{
|
|
650
|
+
id: 1234,
|
|
651
|
+
text: 'What is the purpose of this DMP?',
|
|
652
|
+
order: 1,
|
|
653
|
+
answer: {
|
|
654
|
+
id: 543,
|
|
655
|
+
json: {
|
|
656
|
+
type: 'repositorySearch',
|
|
657
|
+
answer: [{
|
|
658
|
+
repositoryId: 'https://example.com/repository/123456789',
|
|
659
|
+
repositoryName: 'Example Repository',
|
|
660
|
+
}],
|
|
661
|
+
meta: {schemaVersion: '1.0'}
|
|
662
|
+
}
|
|
663
|
+
},
|
|
664
|
+
}]
|
|
665
|
+
}]
|
|
666
|
+
}
|
|
667
|
+
}
|
|
668
|
+
}
|
|
669
|
+
```
|
|
670
|
+
|
|
671
|
+
## RDS MySQL Support
|
|
672
|
+
|
|
673
|
+
This code can be used by to provide access to the RDS MySQL database.
|
|
674
|
+
|
|
675
|
+
It provides a simple `queryTable` function which can be used to query a table. Similar to the way we do so within the Apollo server backend code.
|
|
676
|
+
|
|
677
|
+
Environment variable requirements:
|
|
678
|
+
- `AWS_REGION` - The AWS region where the Lambda Function is running
|
|
679
|
+
- `RDS_HOST` The endpoint of the RDS instance
|
|
680
|
+
- `RDS_PORT` The port (defaults to 3306)
|
|
681
|
+
- `RDS_USER` The name of the user (defaults to "root")
|
|
682
|
+
- `RDS_PASSWORD` The user's password
|
|
683
|
+
- `RDS_DATABASE` The name of the database
|
|
684
|
+
|
|
685
|
+
### Example usage
|
|
686
|
+
```typescript
|
|
687
|
+
import { queryTable } from '@dmptool/utils';
|
|
688
|
+
|
|
689
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
690
|
+
process.env.RDS_HOST = 'some-rds-instance.us-east-1.rds.amazonaws.com';
|
|
691
|
+
process.env.RDS_PORT = '3306';
|
|
692
|
+
process.env.RDS_USER = 'my_user';
|
|
693
|
+
process.env.RDS_PASSWORD = 'open-sesame';
|
|
694
|
+
process.env.RDS_DATABASE = 'my_database';
|
|
695
|
+
|
|
696
|
+
const sql = 'SELECT * FROM some_table WHERE id = ?';
|
|
697
|
+
const id = 1234;
|
|
698
|
+
const resp = await queryTable(sql, [planId.toString()])
|
|
699
|
+
|
|
700
|
+
if (resp && Array.isArray(resp.results) && resp.results.length > 0) {
|
|
701
|
+
console.log('It worked!', resp.results[0]);
|
|
702
|
+
} else {
|
|
703
|
+
console.log('No results found');
|
|
704
|
+
}
|
|
705
|
+
```
|
|
706
|
+
|
|
707
|
+
## S3 Support
|
|
708
|
+
|
|
709
|
+
This code can be used to interact with objects in an S3 bucket.
|
|
710
|
+
|
|
711
|
+
It currently allows you to:
|
|
712
|
+
- List buckets
|
|
713
|
+
- Get an object from a bucket
|
|
714
|
+
- Put an object into a bucket
|
|
715
|
+
- Generate a pre-signed URL for an object in a bucket
|
|
716
|
+
|
|
717
|
+
Environment variable requirements:
|
|
718
|
+
- `AWS_REGION` - The AWS region where the Lambda Function is running
|
|
719
|
+
|
|
720
|
+
### Example usage
|
|
721
|
+
```typescript
|
|
722
|
+
import { getObject, getPresignedURL, listBuckets, putObject } from '@dmptool/utils';
|
|
723
|
+
|
|
724
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
725
|
+
|
|
726
|
+
const bucketName = 'my-bucket';
|
|
727
|
+
const objectKey = 'my-object.txt';
|
|
728
|
+
|
|
729
|
+
const fileName = 'my-file.json.gz'
|
|
730
|
+
const gzippedData = zlib.gzipSync(JSON.stringify({ testing: { foo: 'bar' } }));
|
|
731
|
+
|
|
732
|
+
// List the objects to verify that we're able to access the bucket)
|
|
733
|
+
const s3Objects = await listObjects(bucketName, '');
|
|
734
|
+
console.log('Objects in bucket:', s3Objects);
|
|
735
|
+
|
|
736
|
+
// First put the item into the bucket
|
|
737
|
+
const response = await putObject(
|
|
738
|
+
bucketName,
|
|
739
|
+
fileName,
|
|
740
|
+
gzippedData,
|
|
741
|
+
'application/json', 'gzip'
|
|
742
|
+
);
|
|
743
|
+
|
|
744
|
+
if (response) {
|
|
745
|
+
console.log('Object uploaded successfully');
|
|
746
|
+
|
|
747
|
+
// Get the object we just uploaded from the bucket
|
|
748
|
+
const object = await getObject(bucketName, objectKey);
|
|
749
|
+
console.log('Object fetched from bucket:', object);
|
|
750
|
+
|
|
751
|
+
// Generate a presigned URL to access the object from outside the VPC
|
|
752
|
+
const url = await getPresignedURL(bucketName, objectKey);
|
|
753
|
+
console.log('Presigned URL to fetch the Object:', url);
|
|
754
|
+
|
|
755
|
+
// Generate a presigned URL to put an object into the bucket from outside the VPC
|
|
756
|
+
const putURL = await getPresignedURL(bucketName, `2nd-${objectKey}`, true);
|
|
757
|
+
console.log('Presigned URL to put a new the Object into the bucket', putURL);
|
|
758
|
+
} else {
|
|
759
|
+
console.log('Failed to upload object');
|
|
760
|
+
}
|
|
761
|
+
```
|
|
762
|
+
|
|
763
|
+
## SSM Parameter Store Support
|
|
764
|
+
|
|
765
|
+
This code provides a simple `getSSMParameter` function which can be used to fetch an SSM Parameter.
|
|
766
|
+
|
|
767
|
+
Environment variable requirements:
|
|
768
|
+
- `AWS_REGION` - The AWS region where the Lambda Function is running
|
|
769
|
+
- `NODE_ENV` - The environment the Lambda Function is running in (e.g. `production`, `staging` or `development`)
|
|
770
|
+
|
|
771
|
+
The code will use that value to construct the appropriate prefix for the key. For example if you are running in the AWS development environment it will use `/uc3/dmp/tool/dev/` as the prefix.
|
|
772
|
+
|
|
773
|
+
### Example usage
|
|
774
|
+
```typescript
|
|
775
|
+
import { getSSMParameter } from '@dmptool/utils';
|
|
776
|
+
|
|
777
|
+
process.env.AWS_REGION = 'us-west-2';
|
|
778
|
+
|
|
779
|
+
const paramName = 'RdsDatabase';
|
|
780
|
+
|
|
781
|
+
const response = await getSSMParameter(paramName);
|
|
782
|
+
|
|
783
|
+
if (response) {
|
|
784
|
+
console.log('SSM Parameter fetched successfully', response);
|
|
785
|
+
} else {
|
|
786
|
+
console.log('Error fetching SSM Parameter');
|
|
787
|
+
}
|
|
788
|
+
```
|