@3sixty/data-processing-service 0.0.0-stage → 0.20.0-next.1

package/docs/reference/classes/DataProcessingService.md

@@ -0,0 +1,244 @@
+# Class: DataProcessingService
+
+Class for processing data from a source.
+
+## Implements
+
+- `IDataProcessingComponent`
+
+## Constructors
+
+### Constructor
+
+> **new DataProcessingService**(`options?`): `DataProcessingService`
+
+Create a new instance of DataExtractionService.
+
+#### Parameters
+
+##### options?
+
+[`IDataProcessingServiceConstructorOptions`](../interfaces/IDataProcessingServiceConstructorOptions.md)
+
+The options for the connector.
+
+#### Returns
+
+`DataProcessingService`
+
+#### Throws
+
+GeneralError If no extractor connector is registered.
+
+## Properties
+
+### CLASS\_NAME {#class_name}
+
+> `readonly` `static` **CLASS\_NAME**: `string`
+
+Runtime name for the class.
+
+## Methods
+
+### className() {#classname}
+
+> **className**(): `string`
+
+Returns the class name of the component.
+
+#### Returns
+
+`string`
+
+The class name of the component.
+
+#### Implementation of
+
+`IDataProcessingComponent.className`
+
+***
+
+### ruleGroupSet() {#rulegroupset}
+
+> **ruleGroupSet**(`ruleGroup`): `Promise`\<`void`\>
+
+Set an extraction rule group.
+
+#### Parameters
+
+##### ruleGroup
+
+`IRuleGroup`
+
+The rule group to store.
+
+#### Returns
+
+`Promise`\<`void`\>
+
+A promise that resolves when the rule group has been stored.
+
+#### Implementation of
+
+`IDataProcessingComponent.ruleGroupSet`
+
+***
+
+### ruleGroupGet() {#rulegroupget}
+
+> **ruleGroupGet**(`ruleGroupId`): `Promise`\<`IRuleGroup`\>
+
+Get a rule group for extraction.
+
+#### Parameters
+
+##### ruleGroupId
+
+`string`
+
+The id of the rule group to get.
+
+#### Returns
+
+`Promise`\<`IRuleGroup`\>
+
+The rule group.
+
+#### Implementation of
+
+`IDataProcessingComponent.ruleGroupGet`
+
+***
+
+### ruleGroupRemove() {#rulegroupremove}
+
+> **ruleGroupRemove**(`ruleGroupId`): `Promise`\<`void`\>
+
+Remove a rule group.
+
+#### Parameters
+
+##### ruleGroupId
+
+`string`
+
+The id of the rule group to remove.
+
+#### Returns
+
+`Promise`\<`void`\>
+
+A promise that resolves when the rule group has been removed.
+
+#### Implementation of
+
+`IDataProcessingComponent.ruleGroupRemove`
+
+***
+
+### extract() {#extract}
+
+> **extract**(`ruleGroupId`, `data`, `overrideExtractorType?`, `overrideMimeType?`): `Promise`\<`unknown`\>
+
+Extracts data from the provided input.
+
+#### Parameters
+
+##### ruleGroupId
+
+`string`
+
+The id of the rule group to use to extract data.
+
+##### data
+
+`Uint8Array`
+
+The data to extract from.
+
+##### overrideExtractorType?
+
+`string`
+
+An optional override for the extractor type.
+
+##### overrideMimeType?
+
+`string`
+
+An optional override for the mime type for conversion, will auto detect if not provided.
+
+#### Returns
+
+`Promise`\<`unknown`\>
+
+The extracted data.
+
+#### Implementation of
+
+`IDataProcessingComponent.extract`
+
+***
+
+### convert() {#convert}
+
+> **convert**(`data`, `overrideMimeType?`): `Promise`\<`unknown`\>
+
+Converts data from the provided input to a structured JSON document.
+
+#### Parameters
+
+##### data
+
+`Uint8Array`
+
+The data to convert.
+
+##### overrideMimeType?
+
+`string`
+
+An optional override for the mime type, will auto detect if empty.
+
+#### Returns
+
+`Promise`\<`unknown`\>
+
+The converted data.
+
+#### Implementation of
+
+`IDataProcessingComponent.convert`
+
+***
+
+### query() {#query}
+
+> **query**(`cursor?`, `limit?`): `Promise`\<\{ `entities`: `IRuleGroup`[]; `cursor?`: `string`; \}\>
+
+Query the rule group entries.
+
+#### Parameters
+
+##### cursor?
+
+`string`
+
+The cursor to request the next chunk of entities.
+
+##### limit?
+
+`number`
+
+Limit the number of entities to return.
+
+#### Returns
+
+`Promise`\<\{ `entities`: `IRuleGroup`[]; `cursor?`: `string`; \}\>
+
+All the entities for the storage matching the conditions,
+and a cursor which can be used to request more entities.
+
+#### Implementation of
+
+`IDataProcessingComponent.query`

package/docs/reference/classes/ExtractionRule.md

@@ -0,0 +1,45 @@
+# Class: ExtractionRule
+
+Class defining an extraction rule.
+
+## Constructors
+
+### Constructor
+
+> **new ExtractionRule**(): `ExtractionRule`
+
+#### Returns
+
+`ExtractionRule`
+
+## Properties
+
+### source {#source}
+
+> **source**: `string`
+
+The JSONPath expression identifying the source field in the input document.
+
+***
+
+### target {#target}
+
+> **target**: `string`
+
+The dotted path identifying where to store the extracted value in the output.
+
+***
+
+### retainPathDepth? {#retainpathdepth}
+
+> `optional` **retainPathDepth?**: `number`
+
+The number of path segments from the source location to preserve in the target path.
+
+***
+
+### coerce? {#coerce}
+
+> `optional` **coerce?**: `CoerceType`
+
+The type to coerce the extracted value to.

package/docs/reference/classes/ExtractionRuleGroup.md

@@ -0,0 +1,37 @@
+# Class: ExtractionRuleGroup
+
+Class defining an extraction rule group.
+
+## Constructors
+
+### Constructor
+
+> **new ExtractionRuleGroup**(): `ExtractionRuleGroup`
+
+#### Returns
+
+`ExtractionRuleGroup`
+
+## Properties
+
+### id {#id}
+
+> **id**: `string`
+
+The unique identifier for the rule group.
+
+***
+
+### label {#label}
+
+> **label**: `string`
+
+The human-readable label for the rule group.
+
+***
+
+### rules {#rules}
+
+> **rules**: [`ExtractionRule`](ExtractionRule.md)[]
+
+The extraction rules that belong to this group.

package/docs/reference/functions/dataProcessingConvert.md

@@ -0,0 +1,31 @@
+# Function: dataProcessingConvert()
+
+> **dataProcessingConvert**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`IDataProcessingConvertResponse`\>
+
+Convert data using the specified rule group.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingConvertRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`IDataProcessingConvertResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/functions/dataProcessingExtract.md

@@ -0,0 +1,31 @@
+# Function: dataProcessingExtract()
+
+> **dataProcessingExtract**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`IDataProcessingExtractResponse`\>
+
+Extract data using the specified rule group.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingExtractRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`IDataProcessingExtractResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/functions/dataProcessingGetRuleGroup.md

@@ -0,0 +1,31 @@
+# Function: dataProcessingGetRuleGroup()
+
+> **dataProcessingGetRuleGroup**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`IDataProcessingRuleGroupGetResponse`\>
+
+Get an extraction rule group.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingRuleGroupGetRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`IDataProcessingRuleGroupGetResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/functions/dataProcessingRemoveRuleGroup.md

@@ -0,0 +1,31 @@
+# Function: dataProcessingRemoveRuleGroup()
+
+> **dataProcessingRemoveRuleGroup**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`INoContentResponse`\>
+
+Remove an extraction rule group.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingRuleGroupRemoveRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`INoContentResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/functions/dataProcessingSetRuleGroup.md

@@ -0,0 +1,31 @@
+# Function: dataProcessingSetRuleGroup()
+
+> **dataProcessingSetRuleGroup**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`INoContentResponse`\>
+
+Store an extraction rule group.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingRuleGroupSetRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`INoContentResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/functions/generateRestRoutesDataProcessing.md

@@ -0,0 +1,25 @@
+# Function: generateRestRoutesDataProcessing()
+
+> **generateRestRoutesDataProcessing**(`baseRouteName`, `componentName`): `IRestRoute`\<`any`, `any`\>[]
+
+The REST routes for data processing.
+
+## Parameters
+
+### baseRouteName
+
+`string`
+
+Prefix to prepend to the paths.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes stored in the ComponentFactory.
+
+## Returns
+
+`IRestRoute`\<`any`, `any`\>[]
+
+The generated routes.

package/docs/reference/functions/initSchema.md

@@ -0,0 +1,9 @@
+# Function: initSchema()
+
+> **initSchema**(): `void`
+
+Registers entity schemas for the data extraction connector entity storage.
+
+## Returns
+
+`void`

package/docs/reference/functions/ruleGroupList.md

@@ -0,0 +1,31 @@
+# Function: ruleGroupList()
+
+> **ruleGroupList**(`httpRequestContext`, `componentName`, `request`): `Promise`\<`IDataProcessingRuleGroupListResponse`\>
+
+Get a list of the rule group entries.
+
+## Parameters
+
+### httpRequestContext
+
+`IHttpRequestContext`
+
+The request context for the API.
+
+### componentName
+
+`string`
+
+The name of the component to use in the routes.
+
+### request
+
+`IDataProcessingRuleGroupListRequest`
+
+The request.
+
+## Returns
+
+`Promise`\<`IDataProcessingRuleGroupListResponse`\>
+
+The response object with additional http response properties.

package/docs/reference/index.md

@@ -0,0 +1,27 @@
+# @3sixty/data-processing-service
+
+## Classes
+
+- [DataProcessingService](classes/DataProcessingService.md)
+- [ExtractionRule](classes/ExtractionRule.md)
+- [ExtractionRuleGroup](classes/ExtractionRuleGroup.md)
+
+## Interfaces
+
+- [IDataProcessingServiceConstructorOptions](interfaces/IDataProcessingServiceConstructorOptions.md)
+
+## Variables
+
+- [tagsDataProcessing](variables/tagsDataProcessing.md)
+- [restEntryPoints](variables/restEntryPoints.md)
+
+## Functions
+
+- [generateRestRoutesDataProcessing](functions/generateRestRoutesDataProcessing.md)
+- [dataProcessingSetRuleGroup](functions/dataProcessingSetRuleGroup.md)
+- [dataProcessingGetRuleGroup](functions/dataProcessingGetRuleGroup.md)
+- [dataProcessingRemoveRuleGroup](functions/dataProcessingRemoveRuleGroup.md)
+- [dataProcessingExtract](functions/dataProcessingExtract.md)
+- [dataProcessingConvert](functions/dataProcessingConvert.md)
+- [ruleGroupList](functions/ruleGroupList.md)
+- [initSchema](functions/initSchema.md)

package/docs/reference/interfaces/IDataProcessingServiceConstructorOptions.md

@@ -0,0 +1,25 @@
+# Interface: IDataProcessingServiceConstructorOptions
+
+The options for the data processing service constructor.
+
+## Properties
+
+### extractionRuleGroupStorageConnectorType? {#extractionrulegroupstorageconnectortype}
+
+> `optional` **extractionRuleGroupStorageConnectorType?**: `string`
+
+The type of the entity storage connector to use for extraction rule groups.
+
+#### Default
+
+```ts
+extraction-rule-group
+```
+
+***
+
+### defaultExtractorType? {#defaultextractortype}
+
+> `optional` **defaultExtractorType?**: `string`
+
+The default extractor type to use, if not provided uses the first in the factory.

package/docs/reference/variables/restEntryPoints.md

@@ -0,0 +1,5 @@
+# Variable: restEntryPoints
+
+> `const` **restEntryPoints**: `IRestRouteEntryPoint`[]
+
+The REST entry points for the data processing service.

package/docs/reference/variables/tagsDataProcessing.md

@@ -0,0 +1,5 @@
+# Variable: tagsDataProcessing
+
+> `const` **tagsDataProcessing**: `ITag`[]
+
+The tag to associate with the routes.

package/locales/en.json

@@ -0,0 +1,10 @@
+{
+	"error": {
+		"dataProcessingService": {
+			"noExtractorFound": "No extractors could be found in the factory",
+			"ruleGroupNotFound": "The rule group with id \"{notFoundId}\" could not be found",
+			"mimeTypeNotFound": "Unable to detect MIME type from data",
+			"converterNotFound": "Unable to find a converter for the MIME type \"{notFoundId}\""
+		}
+	}
+}

package/package.json

@@ -1,6 +1,60 @@
 {
-  "name": "@3sixty/data-processing-service",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+	"name": "@3sixty/data-processing-service",
+	"version": "0.20.0-next.1",
+	"description": "Service routes and orchestration logic for extraction and conversion workflows",
+	"repository": {
+		"type": "git",
+		"url": "git+https://github.com/3sixtyglobal/data-processing.git",
+		"directory": "packages/data-processing-service"
+	},
+	"author": "martyn.janes@iota.org",
+	"license": "Apache-2.0",
+	"type": "module",
+	"engines": {
+		"node": ">=24.0.0"
+	},
+	"dependencies": {
+		"@3sixty/api-models": "next",
+		"@3sixty/core": "next",
+		"@3sixty/data-processing-models": "0.20.0-next.1",
+		"@3sixty/entity": "next",
+		"@3sixty/entity-storage-models": "next",
+		"@3sixty/nameof": "next",
+		"@3sixty/web": "next"
+	},
+	"main": "./dist/es/index.js",
+	"types": "./dist/types/index.d.ts",
+	"exports": {
+		".": {
+			"types": "./dist/types/index.d.ts",
+			"import": "./dist/es/index.js",
+			"default": "./dist/es/index.js"
+		},
+		"./locales/*.json": "./locales/*.json"
+	},
+	"files": [
+		"dist/es",
+		"dist/types",
+		"locales",
+		"docs"
+	],
+	"keywords": [
+		"twin",
+		"trade",
+		"iota",
+		"framework",
+		"blockchain",
+		"data-processing",
+		"data",
+		"processing",
+		"transformation",
+		"conversion",
+		"service",
+		"microservice",
+		"business-logic"
+	],
+	"bugs": {
+		"url": "git+https://github.com/3sixtyglobal/data-processing/issues"
+	},
+	"homepage": "https://3sixty.global"
+}