@radicalimaging/static-wado-creator 0.6.4

package/README.md

@@ -0,0 +1,285 @@
+# `@radicalimaging/static-wado-creator`
+
+# static-wado-js
+
+Status: Beta - in development (as of Oct 25, 2021)
+
+The scope of this project is to convert DICOM P10 into a format into a DICOMweb compatible format - specifically JSON metadata, frames and bulkdata.  This library will enable the following:
+* Building of DICOMweb compliant services
+  * Ability to stream the data for "on the fly" use cases
+* Ability to pregenerate DICOMweb responses and store them so they can be served up with a standard HTTP server to implement a subset of DICOMweb WADO-RS, specifically:
+  * Retrieve instance metadata 
+  * Retrieve series level metadata
+  * Retrieve frames in an instance
+  * Retrieve bulk data
+  * Retrieve study, series or instance query
+* To explore an alternative archive format that is web friendly (no need for a DICOM parser)
+  * Should be able to recreate DICOM P10 file from this data (semantic equivalence or possibly bit for bit equivalent)
+  * Will make updates more efficient (e.g. updating patient name just requires updating metadata)
+  * More efficient image access (no need to scan through DICOM P10 to access frames, or store the offsets of each frame separately and seek to them) 
+
+The scope of this library is limited to:
+* Taking one or more DICOM files as input, writing Bulkdata and Pixeldata
+* Creating either deduplicated representations or instance level metadata
+* Combining deduplicated representations into sets of deduplicated representations
+* Using the deduplicated representations to generate study/series/instance query results and series metadata
+
+View the [design rationale](docs/design.md) for more information on the scope
+
+View [frequently asked questions](docs/faq.md)
+
+View [specification](docs/spec.md)
+
+## Pre-requisites
+View root pre-requisites section [pre-requisites](../../README.md#pre-requisites)
+
+## Development
+View root development section [development](../../README.md#development)
+
+## Running the CLI
+Install the CLI with either npm install -g .  or npm install -g @radicalimaging/static-wado-js
+
+## Usage
+Once you install the package the executables (i.e static-wado commands) are installed into to your PATH system, then you can use anywhere on your command line program.
+
+### Api
+There are 5 different commands: mkdicomweb, mkdicomwebinstances, mkdicomwebstudy, mkdicomwebdeduplicated, mkdicomwebdeduplicatedgroup that can be used (see usage bellow). It can be run as:
+
+```
+commnadName [options] [<input...>]
+```
+
+Arguments are:
+
+input
+ : List of directories/files separated by space.
+ Required for commands: **mkdicomweb**, **mkdicomwebdeduplicated**, **mkdicomwebinstances**. Optional for the others. This argument should be placed at end of command invoking.
+
+The options are:
+
+  -c, --clean
+  : Clean the study output directory for these instances (default: false)
+  
+  -C, --remove-deduplicated-instances
+  : Remove single instance deduplicated files after writing group files (default: false)
+  
+  -d, --deduplicate
+  : Write deduplicate instance level data (default: false)
+  
+  -g, --group 
+  : Write combined deduplicate data (default: true)
+  
+  -i, --instances
+  : Write instance metadata (default: false)
+  
+  -m, --maximum-inline-public-length <value>
+  : Maximum length of public binary data (default: 131074)
+  
+  -M, --maximum-inline-private-length <value>
+  : Maximum length of private binary data (default: 64)
+  
+  --no-recompress
+  : Force no recompression
+  [see](#to-recompress)
+
+  --no-recompress-thumb
+  : Force no recompression for thumbnail
+  [see](#to-recompress-thumb)
+  
+  -o, --dir <value>
+  : Set output directory (default: "~/dicomweb")
+  
+  --path-deduplicated <path>
+  : Set the deduplicate data directory path (relative to dir) (default: "deduplicated")
+  
+  --path-instances <path>
+  : Set the instances directory path (relative to dir) (default: "instances")
+  
+  -r, --recompress <listvalue...> 
+  : List of types to recompress separated by space (choices: "uncompressed", "jp2", "jpeglossless", "rle", default: "uncompressed jp2")
+  [see](#to-recompress)
+
+  --recompress-thumb <listvalue...> 
+  : List of types to recompress thumb separated by space (choices: "uncompressed", "jp2", "jpeglossless", "rle", default: "uncompressed jp2")
+  [see](#to-recompress-thumb)
+  
+  -s, --study
+  : Write study metadata - on provided instances only (TO FIX), (default: true)
+  
+  -t, --content-type <type>
+  : Destination type to compress to (choices: "jls", "lei", "jls-lossy" or DICOM Transfer Syntax UID - default: "jls")
+  
+  -T, --colour-content-type <type>
+  : Colour content type (default: null)
+  
+  -v, --verbose
+  : Write verbose output (default: false)
+  
+  -V, --version
+  : output the version number
+
+### Help command
+You can output help information by:
+```
+commandName help
+```
+
+```
+commandName -h
+```
+
+```
+commandName --help
+```
+
+### To create instances
+Run the tool:
+```
+mkdicomwebinstances <directoryOfP10Files>
+```
+
+### To create a full DICOMweb output structure
+Run the tool:
+```
+mkdicomweb <directoryOfP10Files>
+```
+
+### To run separated stages
+The mkdicomweb tool runs the three stages all together, on just the studies references.  This can instead be done on separate files by running:
+```
+mkdicomwebdeduplicated  <directoryOfP10Files>
+mkdicomwebdeduplicatedgroup
+mkdicomwebstudy
+```
+which creates a full study directory.  The first stage writes to ~/dicomweb/instances/<studyUID>/ data about each instance as it is read in.  The second stage then groups these files for more efficient compression into the ~/dicomweb/deduplicated/<studyUID>/   The last stage then creates the actual DICOMweb files.
+
+There is currently no notification of what studies have been updated between stages.  The intent is to write notifications to ~/dicomweb/notifications/<studyUID> of what operations need to be applied/updated.
+
+### To Serve Instances As a Web Server
+```
+cd ~/dicomweb
+npx http-server -p 5000 -g --cors
+```
+
+The -g option serves up compressed files ending in .gz as compressed http streams.
+
+### To Create DICOM part 10 from DICOMweb files
+TODO
+
+Run the tool mkdicomwebpart10 on the studyUID, and optionally on the series/instance UID's of interest to generate a local set of part 10 files.
+
+### To Update DICOM Metadata
+TODO
+
+Run the tool
+```
+mkdicomwebupdate -<delete/anonymize/patient/study/series/instance> <studyInstanceUID> (tag=newValue)* 
+```
+to delete the given item or to update the specified attribute contained in the given level.  Multiple mkdicomwebupdate commands may be run to perform updates on different attribute sets, or they may be grouped into a single file for bulk application.
+
+### To recompress
+It allows commands to recompress data/metadata prior writing it to local. Currently, if recompress is active the designed data types will be transcoded [see](#recompress-aliases-and-input-transfer-syntaxes).
+
+By default the recompression occurs for incoming types: uncompressed, jp2
+```
+mkdicomweb ./folderName
+```
+It will recompress any existing data whose transfer syntaxes are 1.2.840.10008.1.2.4.90, 1.2.840.10008.1.2.4.91, 1.2.840.10008.1.2, 1.2.840.10008.1.2.1 and 1.2.840.10008.1.2.2.
+
+Define incoming types for recompression of: uncompressed,jp2,rle
+```
+mkdicomweb  -r uncompressed jp2 rle ./folderName
+```
+It will recompress any existing data whose transfer syntaxes: 1.2.840.10008.1.2.4.90, 1.2.840.10008.1.2.4.91, 1.2.840.10008.1.2, 1.2.840.10008.1.2.1, 1.2.840.10008.1.2.2 and 1.2.840.10008.1.2.5 to transfer syntax 1.2.840.10008.1.2.4.80.
+
+By default the destination compression type for recompression: jls
+```
+mkdicomweb ./folderName
+```
+It will recompress any existing data whose transfer syntaxes are 1.2.840.10008.1.2.4.90, 1.2.840.10008.1.2.4.91, 1.2.840.10008.1.2, 1.2.840.10008.1.2.1 and 1.2.840.10008.1.2.2 to jls (transfer syntax 1.2.840.10008.1.2.4.80)
+
+Define destination type for uncompressed encoding: lei
+```
+mkdicomweb -t lei ./folderName
+```
+It will decode any existing data whose transfer syntaxes are 1.2.840.10008.1.2.4.90, 1.2.840.10008.1.2.4.91, 1.2.840.10008.1.2, 1.2.840.10008.1.2.1 and 1.2.840.10008.1.2.2 and store uncompressed as LEI (transfer syntax 1.2.840.10008.1.2.1)
+
+Force no recompression
+```
+mkdicomweb  -r uncompressed jp2 rle --no-recompress ./folderName
+```
+It will NOT recompress any existing data but instead it will keep them in the original encoding.
+
+Force no recompression
+```
+mkdicomweb --no-recompress ./folderName
+```
+It will NOT recompress any existing data but instead it will keep them in the original encoding.
+
+Obs: --no-recompress forces no compression at all. Use it if you want to disable even the default behavior.
+
+
+### Recompress aliases and input transfer syntaxes
+The table below shows the support for transfer syntax recompression(i.e use "... -r jp2" if you want to recompress from 1.2.840.10008.1.2.4.90 to 1.2.840.10008.1.2.4.80)
+
+| TransferSyntaxUID      	| Name                                                    	| Recompress alias 	|  Target TransferSyntaxUID |
+|------------------------	|---------------------------------------------------------	|------------------	|--------------------------	|
+| 1.2.840.10008.1.2      	| Implicit VR Endian                                      	| uncompressed     	| 1.2.840.10008.1.2         |
+| 1.2.840.10008.1.2.1    	| Explicit VR Little Endian                               	| uncompressed     	| 1.2.840.10008.1.2.1       |
+| 1.2.840.10008.1.2.2    	| Explicit VR Big Endian                                  	| uncompressed     	| 1.2.840.10008.1.2.1       |
+| 1.2.840.10008.1.2.4.57 	| JPEG Lossless, Nonhierarchical (Processes 14)           	| jpeglossless     	| 1.2.840.10008.1.2.4.80    |
+| 1.2.840.10008.1.2.4.70 	| JPEG Lossless, Nonhierarchical, First- Order Prediction 	| jpeglossless     	| 1.2.840.10008.1.2.4.80    |
+| 1.2.840.10008.1.2.4.90 	| JPEG 2000 Image Compression (Lossless Only)             	| jp2              	| 1.2.840.10008.1.2.4.80    |
+| 1.2.840.10008.1.2.4.91 	| JPEG 2000 Image Compression                             	| jp2              	| 1.2.840.10008.1.2.4.80    |
+| 1.2.840.10008.1.2.5    	| RLE Lossless                                            	| rle              	| 1.2.840.10008.1.2.4.80    |
+
+
+### To recompress thumb
+It tells thumbnail creation to use recompressed data or not.
+The list of types MUST be necessarily a subset of recompress list.
+
+Obs: --no-recompress-thumb forces to not use compression at all. Use it if you want to disable even the default behavior.
+
+By default the recompression occurs for incoming types: uncompressed, jp2. This will recompress the types mentioned and will generate the thumbnails with the result of data recompression.
+
+```
+mkdicomweb ./folderName
+```
+
+Force recompression thumb. This will recompress the types: uncompressed jp2 rle, thumbnails creation will use original data for all types except jp2 (which will use recompress data).
+```
+mkdicomweb  -r uncompressed jp2 rle --recompress-thumb jp2 ./folderName
+```
+
+Skipping not recompressed data. This will recompress the types: uncompressed jp2, thumbnails creation will use original data for all types (since recompress-thumb list does not intersect with recompress list).
+```
+mkdicomweb  -r uncompressed jp2 --recompress-thumb rle ./folderName
+```
+
+Force no recompression thumb. This will recompress the types: uncompressed jp2 rle, but for thumbnails creation will use original data.
+```
+mkdicomweb  -r uncompressed jp2 rle --no-recompress-thumb ./folderName
+```
+
+
+
+
+## Development
+
+## TODO (Looking for help here!!)
+
+* Create docker container to encapsulate build environment
+* Create test data (DICOM P10 and expected DICOMweb results) - Jordan working on this
+* Document API
+* Create CI + Publish to NPM
+* Fix Bugs
+    * Get bulkdata refs written properly
+* Enhance cli
+    * Add support for specifying bulkDataMinSize
+    * Add support for writing out DICOM P10 file
+* Enhance library
+    * Write out "info" file
+        * P10 Header
+        * Data needed to recreate original P10 instance
+* Create DICOMweb -> DICOM P10 tool
+

package/bin/mkdicomweb.js

@@ -0,0 +1,9 @@
+#!/usr/bin/env node
+
+const { mkdicomwebConfig } = require("../lib");
+const { configureProgram } = require("../lib/program");
+
+// Configure program commander
+configureProgram(mkdicomwebConfig).then(() => {
+  console.log("done");
+});

package/lib/DeleteStudy.js

@@ -0,0 +1,14 @@
+const { JSONReader } = require("@radicalimaging/static-wado-util");
+const JSONWriter = require("./writer/JSONWriter");
+const Tags = require("./dictionary/Tags");
+
+module.exports = (options) =>
+  async function (studyInstanceUid) {
+    console.log("Delete Study", studyInstanceUid);
+    const studyData = await this.scanStudy("studies", studyInstanceUid);
+    studyData.delete();
+    const allStudies = await JSONReader(options.directoryName, "studies/index.json.gz", []);
+    const studiesWithoutDeleted = allStudies.filter((study) => studyInstanceUid != Tags.getValue(study,Tags.StudyInstanceUID));
+    await JSONWriter(options.directoryName, "studies", studiesWithoutDeleted);
+    delete this.studyData;
+  };

package/lib/RejectInstance.js

@@ -0,0 +1,17 @@
+const dataExtractor = /^(.*studies)?[\\/]?([^/\\]+)[\\/](series[\\/])?([^/\\]+)([\\/]instances[\\/]([^/\\]*))?$/;
+
+module.exports = (options) =>
+  async function (rejectPath, reason) {
+    console.log("Reject instance", rejectPath, reason);
+    const extracted = rejectPath.match(dataExtractor);
+    if (!extracted) {
+      console.log("Unable to reject", rejectPath, "because it isn't in the format studies/<studyUID>/series/<seriesUID>");
+      return;
+    }
+    const [, , studyInstanceUid, , seriesInstanceUid, , sopInstanceUid] = extracted;
+    const studyData = await this.scanStudy("studies", studyInstanceUid);
+    if (options.verbose) console.log("Trying to reject", seriesInstanceUid, "because", reason);
+    await studyData.reject(seriesInstanceUid, sopInstanceUid, reason);
+    await studyData.writeDeduplicatedGroup();
+    return studyData.writeMetadata();
+  };

package/lib/StaticWado.js

@@ -0,0 +1,206 @@
+const dicomCodec = require("@cornerstonejs/dicom-codec");
+const { Stats, handleHomeRelative } = require("@radicalimaging/static-wado-util");
+const dicomParser = require("dicom-parser");
+const fs = require("fs");
+const path = require("path");
+const { dirScanner } = require("@radicalimaging/static-wado-util");
+const asyncIterableToBuffer = require("./operation/adapter/asyncIterableToBuffer");
+const getDataSet = require("./operation/getDataSet");
+const InstanceDeduplicate = require("./operation/InstanceDeduplicate");
+const DeduplicateWriter = require("./writer/DeduplicateWriter");
+const ImageFrameWriter = require("./writer/ImageFrameWriter");
+const CompleteStudyWriter = require("./writer/CompleteStudyWriter");
+const IdCreator = require("./util/IdCreator");
+const ScanStudy = require("./operation/ScanStudy");
+const HashDataWriter = require("./writer/HashDataWriter");
+const VideoWriter = require("./writer/VideoWriter");
+const { transcodeImageFrame, transcodeId, transcodeMetadata } = require("./operation/adapter/transcodeImage");
+const ThumbnailWriter = require("./writer/ThumbnailWriter");
+const ThumbnailService = require("./operation/ThumbnailService");
+const DeleteStudy = require("./DeleteStudy");
+const RejectInstance = require("./RejectInstance");
+
+function setStudyData(studyData) {
+  this.studyData = studyData;
+}
+
+class StaticWado {
+  constructor(configuration) {
+    const { rootDir = "~/dicomweb", pathDeduplicated = "deduplicated", pathInstances = "instances", verbose } = configuration;
+
+    dicomCodec.setConfig({ verbose });
+    const directoryName = handleHomeRelative(rootDir);
+
+    this.options = {
+      ...configuration,
+      directoryName,
+      deduplicatedRoot: path.join(directoryName, pathDeduplicated),
+      deduplicatedInstancesRoot: path.join(directoryName, pathInstances),
+      TransferSyntaxUID: "1.2.840.10008.1.2",
+    };
+    this.callback = {
+      uids: IdCreator(this.options),
+      bulkdata: HashDataWriter(this.options),
+      imageFrame: ImageFrameWriter(this.options),
+      videoWriter: VideoWriter(this.options),
+      thumbWriter: ThumbnailWriter(this.options),
+      completeStudy: CompleteStudyWriter(this.options),
+      metadata: InstanceDeduplicate(this.options),
+      deduplicated: DeduplicateWriter(this.options),
+      scanStudy: ScanStudy(this.options),
+      reject: RejectInstance(this.options),
+      delete: DeleteStudy(this.options),
+      setStudyData,
+    };
+  }
+
+  /**
+   * Rejects the given item, specified as studies/<studyUID>/series/<seriesUID>
+   * Does NOT remove any files, but adds a flag to the study indicating the deleted series and instances.
+   *
+   * @param {string} args listing the instances to remove (may remove entire series)
+   */
+  async reject(args) {
+    args.forEach((removal) => {
+      this.callback.reject(removal);
+    });
+  }
+
+  /**
+   * Deletes a study, as specified by the study instance UID by removing instance, deduplicated and studies files,
+   * as well as removing the object from the studies list.
+   *
+   * @param {string[]} args listing the studies to remove
+   */
+  async delete(args) {
+    args.forEach((studyUID) => this.callback.delete(studyUID));
+  }
+
+  /**
+   * Processes a set of DICOM files, where the starting point is a list of directory names or file instances.
+   * This is used for importing DICOM files.
+   *
+   * @param {*} files
+   * @param {*} callback
+   * @param {*} params
+   */
+  async processFiles(files, params) {
+    return dirScanner(files, {
+      ...params,
+      callback: async (file) => {
+        try {
+          const dicomp10stream = fs.createReadStream(file);
+          await this.importBinaryDicom(dicomp10stream, { ...params, file });
+          Stats.StudyStats.add("DICOM P10", "Parse DICOM P10 file");
+        } catch (e) {
+          console.error("Couldn't process", file, e);
+        }
+      },
+    });
+  }
+
+  /**
+   * Processes a study directory, matching up study instance UIDs.  Either processes the
+   * deduplicated group directory or the instances directory, or the notifications directory.
+   * @param {*} params
+   */
+  async processStudyDir(studyUids, params) {
+    return dirScanner(params[params.scanStudies], {
+      ...params,
+      recursive: false,
+      callback: (dir, name) => this.callback.scanStudy(dir, name),
+    });
+  }
+
+  async importBinaryDicom(dicomp10stream, params) {
+    // Read dicomp10 stream into buffer
+    const buffer = await asyncIterableToBuffer(dicomp10stream);
+
+    // Parse it
+    const dataSet = dicomParser.parseDicom(buffer, params);
+
+    const studyInstanceUid = dataSet.string("x0020000d");
+
+    if (!studyInstanceUid) {
+      console.log(`Can't import file ${params.file}`);
+      return undefined;
+    }
+
+    // Extract uids
+    const id = this.callback.uids(
+      {
+        studyInstanceUid,
+        seriesInstanceUid: dataSet.string("x0020000e"),
+        sopInstanceUid: dataSet.string("x00080018"),
+        transferSyntaxUid: dataSet.string("x00020010"),
+      },
+      params.file
+    );
+
+    const targetId = transcodeId(id, this.options);
+
+    let bulkDataIndex = 0;
+    let imageFrameIndex = 0;
+    const thumbnailService = new ThumbnailService();
+
+    const generator = {
+      bulkdata: async (bulkData, options) => {
+        const _bulkDataIndex = bulkDataIndex;
+        bulkDataIndex += 1;
+        return this.callback.bulkdata(targetId, _bulkDataIndex, bulkData, options);
+      },
+      imageFrame: async (originalImageFrame) => {
+        const { imageFrame: transcodedImageFrame, id: transcodedId } = await transcodeImageFrame(id, targetId, originalImageFrame, dataSet, this.options);
+
+        const currentImageFrameIndex = imageFrameIndex;
+        imageFrameIndex += 1;
+
+        thumbnailService.queueThumbnail(
+          {
+            imageFrame: originalImageFrame,
+            transcodedImageFrame,
+            transcodedId,
+            id,
+            frameIndex: currentImageFrameIndex,
+          },
+          this.options
+        );
+
+        return this.callback.imageFrame(transcodedId, currentImageFrameIndex, transcodedImageFrame);
+      },
+      videoWriter: async (_dataSet) => this.callback.videoWriter(id, _dataSet),
+    };
+
+    // convert to DICOMweb MetaData and BulkData
+    const result = await getDataSet(dataSet, generator, this.options);
+
+    const transcodedMeta = transcodeMetadata(result.metadata, id, this.options);
+    thumbnailService.generateThumbnails(id, dataSet, transcodedMeta, this.callback);
+
+    await this.callback.metadata(targetId, transcodedMeta);
+
+    // resolve promise with statistics
+    return {};
+  }
+
+  /**
+   * The mkdicomweb command first runs mkdicomwebinstances, writing out the deduplicated data, and then runs the
+   * mkdicomwebstudy command, creating the deduplicated data set.  This version, however, keeps the deduplicated
+   * data in memory by default on a study level, which avoids needing to run the load process.
+   */
+  async executeCommand(input) {
+    if (this.options.scanStudies) {
+      // Scan one of the study directories - in this case, files is a set of study directories
+      await this.processStudyDir(input, this.options);
+    } else {
+      await this.processFiles(input, this.options);
+    }
+    await this.close();
+  }
+
+  async close() {
+    await this.callback.completeStudy();
+    Stats.OverallStats.summarize("Completed Study Processing");
+  }
+}
+module.exports = StaticWado;

package/lib/createMain.js

@@ -0,0 +1,14 @@
+const StaticWado = require("./StaticWado");
+const adaptProgramOpts = require("./util/adaptProgramOpts");
+
+module.exports = function createMain(options, program) {
+  const finalOptions = adaptProgramOpts(options, {
+    ...this,
+    isInstance: false,
+    isGroup: true,
+    isDeduplicate: true,
+    isStudyData: true,
+  });
+  const importer = new StaticWado(finalOptions);
+  return importer.executeCommand(program.args);
+};

package/lib/deleteMain.js

@@ -0,0 +1,12 @@
+const StaticWado = require("./StaticWado");
+const adaptProgramOpts = require("./util/adaptProgramOpts");
+
+module.exports = function deleteMain(options, program) {
+  const finalOptions = adaptProgramOpts(options, {
+    ...this,
+    isGroup: true,
+    isStudyData: true,
+  });
+  const importer = new StaticWado(finalOptions);
+  return importer.delete(program.args);
+};

package/lib/dictionary/Tags.js

@@ -0,0 +1,111 @@
+const dataDictionary = require("./dataDictionary");
+const dcmjs = require("dcmjs");
+
+const { naturalizeDataset, denaturalizeDataset} = dcmjs.data.DicomMetaDictionary
+
+/** Find the actual tag for a private value */
+const findPrivate = (item, tagObject, create) => {
+    if( typeof tagObject === "string" ) return tagObject;
+    if( typeof tagObject === "number" ) return `00000000${tagObject.toString(16)}`.slice(-8);
+    const {creator,tag} = tagObject;
+    if( !creator ) return tag;
+    const start = tag.substring(0,4);
+    const end = tag.substring(6,8);
+    // Should technically go all the way up, but for now only test 0x30 items
+    let assignPosition;
+    for(let offset = 0x10; offset < 0x40; offset++) {
+      const testTag = `${start}00${offset.toString(16)}`
+      const testCreator = item[testTag];
+      if( testCreator===undefined && assignPosition===undefined ) assignPosition = offset;
+      if( testCreator && testCreator.Value && testCreator.Value[0]===creator ) {
+        return `${start}${offset.toString(16)}${end}`
+      }
+    }
+    if( create ) {
+      if( !assignPosition ) throw new Error(`Couldn't find any assign positions for ${creator} ${tag} in ${item}`);
+      const creatorTag = `${start}00${assignPosition.toString(16)}`
+      item[creatorTag] = { Value: [creator], vr: 'CS' };
+      return `${start}${assignPosition.toString(16)}${end}`
+    }
+  };
+
+const DeduppedCreator = "dedupped";
+  
+const Tags = {
+  // Raw tags have the x before them, not parsed yet
+  RawMinTag: "x00000000",
+  RawFirstBodyTag: "x00080000",
+  RawTransferSyntaxUID: "x00020010",
+
+  // This one isn't defined in the dataDictionary
+  AvailableTransferSyntaxUID: "00083002",
+
+  DeduppedCreator,
+
+  // The references to extract data included in this object, 1..n values
+  DeduppedRef: { creator: DeduppedCreator, tag: "00091010"},
+
+  // The hash value of THIS object
+  DeduppedHash: {creator: DeduppedCreator, tag: "00091011"},
+
+  // Type of hash instance
+  DeduppedType: {creator: DeduppedCreator, tag: "00091012"},
+  InstanceType: "instance",
+  InfoType: "info",
+
+  naturalizeDataset, denaturalizeDataset,
+  
+  setValue: (item, tag, value) => {
+    const actualTag = findPrivate(item,tag,true);
+    item[actualTag] = { Value: Array.isArray(value) ? value : [value] };
+    return actualTag;
+  },
+
+  setList: (item, tag, value) => {
+    const actualTag = findPrivate(item,tag,true);
+    item[actualTag] =  {Value: Array.isArray(value) ? value : [value]};
+    return actualTag;
+  },
+
+  pushList: (item, tag, value) => {
+    const actualTag = findPrivate(item,tag,true);
+    if( !item[actualTag] ) item[actualTag] = { Value: [] };
+    item[actualTag].Value.push(value);
+    return actualTag;
+  },
+
+  getValue: (item, tag) => {
+    const actualTag = findPrivate(item,tag);
+    if( !actualTag ) return undefined;
+    const value = item[actualTag];
+    if( value===undefined ) return undefined;
+    if( Array.isArray(value) ) return value.length==1 ? value[0] : (value.length===0 ? "" : value);
+    if( value.Value ) return value.Value.length==1 ? value.Value[0] : (value.Value.length===0 ? "" : value.Value);
+    return value;
+  },
+
+  getList: (item, tag) => {
+    const actualTag = findPrivate(item,tag);
+    if( !actualTag ) return undefined;
+    const value = item[actualTag];
+    if( value===undefined ) return undefined;
+    if( Array.isArray(value) ) return value;
+    if( value.Value ) return value.Value;
+    return [value];
+  },
+
+  removeValue: (item,tag) => {
+    const actualTag = findPrivate(item,tag);
+    if( actualTag ) delete item[actualTag];
+    // TODO, remove creator tag if not used any longer
+  },
+
+  findPrivate,
+};
+
+Object.keys(dataDictionary).forEach((key) => {
+  const value = dataDictionary[key];
+  Tags[value.name] = key;
+});
+
+module.exports = Tags;