@things-factory/dataset 6.0.25 → 6.0.28

package/README.md

@@ -1,8 +1,11 @@
 # dataset
+
 ## dataset relation diagram
-![ERD Diagram](./assets/diagram.jpg)
+
+![Dataset ERD Diagram](./assets/images/dataset-diagram.jpg)
+
 - data_sets 하나가 여러 data_items 를 갖습니다. data_sets와 data_items 조합으로 CCP 항목들을 정의합니다.
-- data_sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key_##)에 해당 값을 저장합니다.
+- data*sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key*##)에 해당 값을 저장합니다.
 - data_set_histories에는 data_sets의 변경 내역이 추가됩니다. data_sets의 id는 data_set_histories에 original_id로 추가됩니다. data_set_histories를 사용하려면 original_id 와 version이 필요합니다.
 - data_sensors에는 자동 수집되는 디바이스 정보를 정의합니다. data_sets를 참고하여 CCP를 관리하도록 합니다.
 - data_samples에는 입력받은 실제 데이터가 저장됩니다.
@@ -10,29 +13,38 @@
 - data_archives에는 다운로드 받기 위한 요청 정보 및 상태가 저장됩니다.
 
 ## Architecture for collecting data samples
-![Architecture for data-samples](./assets/data-samples.jpg)
+
+![Architecture for data-samples](./assets/images/data-samples.jpg)
 data_samples, data_oocs, data_set_histories의 테이블 변경 사항은 Database CDC로 Kafka에 전송 및 저장되고 토픽별로 다음의 lambda 들이 트리거되어 S3로 저장합니다. 이 S3 데이터는 Glue Crawler, Glue Data Catalog를 거쳐 Athena에서 조회할 수 있습니다. hive 스타일의 S3 파티션 정보들을 업데이트 하는 작업이 스케줄링되어 작동하고, 파티션 정보가 업데이트 된 S3파일들이 Athena에서 조회되므로 실시간으로 데이터가 반영되지는 않습니다.
+
 - [func-data-samples-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-samples-to-s3)
 - [func-data-oocs-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-oocs-to-s3)
 - [func-data-set-histories-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-set-histories-to-s3)
 
-개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-* 로 시작하는 lambda들을 사용합니다.
+개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-\* 로 시작하는 lambda들을 사용합니다.
+
 ## data-report process
-![data-report process](./assets/data-report.jpg)
+
+![data-report process](./assets/images/data-report.jpg)
 
 'shiny' uses the next lambda. But 'jasper' is not using this lambda now. Consider using the same data service.
+
 - [func-transform-data-samples](https://github.com/operatochef/serverless/tree/main/func-transform-data-samples)
 
 ## Partition Keys
+
 At the early stage, partition keys are desinged for dynamic partitioning for Athena. But It will be required so many AWS Glue crawlers also by each 'data-sets'.
 
-Partition keys are related for S3 request limitations. 
-> __3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket__
+Partition keys are related for S3 request limitations.
+
+> **3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket**
+
 - https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
 
 ### :warning: This concept will be deprecated.
 
-Now partition keys are fixed  
+Now partition keys are fixed
+
 - domain
 - datasetid
 - date: local time date
@@ -40,19 +52,22 @@ Now partition keys are fixed
 - workshift
 
 ### Glue Catalog Table Indices
+
 Partitions can be indices. You can create new indices with combination of partitions.
 It affects query performance and finding partitions.
-![partitions and indices](./assets/glue-table-indices.png)
+![partitions and indices](./assets/images/glue-table-indices-for-dataset.png)
 
 ## Timestamp of Data Samples
+
 sample data could have an own timestamp if it is from a sensor data.
 'collected_at' uses this timestamp.
 Manual type of data is used 'Date.now() and new Date()'
 Graphql might affect timezone of server os for Date type field.
 'TZ=UTC node' can help to solve this when server starts.
-Or use ``` process.env.TZ = 'UTC' ```.
+Or use `process.env.TZ = 'UTC'`.
 Now applied in 'create-data-samples.ts'
 
 ## Data Archive
+
 제출용 데이터를 위해 data_samples를 다운로드 하는 기능입니다. csv.gz 형태로 다운로드 가능한 임시 url을 제공합니다. 해당 기능을 위해 [func-data-set-download](https://github.com/operatochef/serverless/tree/main/func-data-set-download) lambda를 사용합니다. 이는 수동 수집용 데이터에 적합합니다. 자동 수집 데이터는 lambda로 처리하기에는 많은 메모리가 필요할 수 있습니다. 따라서, 데이타셋 유형별 처리가 필요할 것 같습니다.
-Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.
+Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.

package/client/{bootstrap.js → bootstrap.ts}

@@ -1,3 +1,7 @@
+import './components/data-collect-activity-view' /* refered by the activity template (activity-data-collect) on server */
+import './components/ooc-review-activity-view' /* refered by the activity template (activity-ooc-review) on server */
+import './components/ooc-resolve-activity-view' /* refered by the activity template (activity-ooc-resolve) on server */
+
 import '@operato/app/filter-renderer.js' /* register resource-object filter renderer */
 
 import { OxGristEditorCode } from '@operato/app/grist-editor/ox-grist-editor-code.js'
@@ -6,7 +10,7 @@ import {
   registerEditor as registerGristEditor,
   registerRenderer as registerGristRenderer
 } from '@operato/data-grist'
-import { OxGristEditorDataItemSpec } from '@operato/dataset/grist-editor'
+import { OxGristEditorDataItemSpec } from '@operato/dataset/grist-editor/ox-grist-editor-data-item-spec.js'
 import { OxGristEditorPartitionKeys } from '@operato/grist-editor/ox-grist-editor-partition-keys.js'
 
 export default function bootstrap() {

package/client/components/data-collect-activity-view.ts

@@ -0,0 +1,100 @@
+import '@operato/dataset/ox-data-entry-form.js'
+
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { ScrollbarStyles } from '@operato/styles'
+import { OxDataEntryForm } from '@operato/dataset/ox-data-entry-form.js'
+import { DataSet } from '@operato/dataset'
+
+@customElement('data-collect-activity-view')
+export class DataCollectActivityView extends localize(i18next)(LitElement) {
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
+
+        background-color: #fff;
+      }
+
+      ox-data-entry-form {
+        flex: 1;
+        padding: 10px;
+        overflow: auto;
+      }
+    `
+  ]
+
+  @property({ type: Object }) input?: any
+  @property({ type: Object }) output?: any
+
+  @state() dataSet?: DataSet & { id: string }
+
+  @query('ox-data-entry-form') entryForm!: OxDataEntryForm
+
+  render() {
+    return html`
+      <ox-data-entry-form
+        .dataSet=${this.dataSet}
+        .value=${this.output}
+        @change=${this._updateDataItems}
+      ></ox-data-entry-form>
+    `
+  }
+
+  async _updateDataItems() {
+    this.output = this.entryForm.value
+
+    this.dispatchEvent(
+      new CustomEvent('change', {
+        detail: this.output
+      })
+    )
+  }
+
+  updated(changes) {
+    if (changes.has('input')) {
+      this.fetchDataSet()
+    }
+  }
+
+  async fetchDataSet() {
+    const id = this.input.dataSetId
+
+    if (id) {
+      const response = await client.query({
+        query: gql`
+          query ($id: String!) {
+            dataSet(id: $id) {
+              id
+              name
+              description
+              useCase
+              dataItems {
+                name
+                description
+                active
+                tag
+                type
+                unit
+                options
+                quota
+                spec
+              }
+            }
+          }
+        `,
+        variables: {
+          id
+        }
+      })
+
+      this.dataSet = response.data.dataSet
+    }
+  }
+}

package/client/components/{data-entry-form.js → data-entry-form.ts}

@@ -2,47 +2,43 @@ import '@operato/dataset/ox-data-entry-form.js'
 
 import gql from 'graphql-tag'
 import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
 
 import { client } from '@operato/graphql'
 import { i18next, localize } from '@operato/i18n'
 import { ScrollbarStyles } from '@operato/styles'
+import { OxDataEntryForm } from '@operato/dataset/ox-data-entry-form.js'
+import { DataSet } from '@operato/dataset'
 
+@customElement('data-entry-form')
 export class DataEntryForm extends localize(i18next)(LitElement) {
-  static get properties() {
-    return {
-      dataSet: Object
-    }
-  }
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
 
-  static get styles() {
-    return [
-      ScrollbarStyles,
-      css`
-        :host {
-          display: flex;
-          flex-direction: column;
+        background-color: #fff;
+      }
 
-          background-color: #fff;
-        }
+      ox-data-entry-form {
+        flex: 1;
+        padding: 10px;
+        overflow: auto;
+      }
 
-        ox-data-entry-form {
-          flex: 1;
-          padding: 10px;
-          overflow: auto;
-        }
+      .button-container {
+        display: flex;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
 
-        .button-container {
-          display: flex;
-          margin-left: auto;
-          padding: var(--padding-default);
-        }
-      `
-    ]
-  }
+  @property({ type: Object }) dataSet?: DataSet & { id: string }
 
-  get entryForm() {
-    return this.renderRoot.querySelector('ox-data-entry-form')
-  }
+  @query('ox-data-entry-form') entryForm!: OxDataEntryForm
 
   render() {
     return html`
@@ -54,10 +50,13 @@ export class DataEntryForm extends localize(i18next)(LitElement) {
   }
 
   async _updateDataItems() {
+    //@ts-ignore TODO replace with following line and confirm
     const data = this.entryForm.buildValue()
+    // const data = this.entryForm.value
+
     const dataSample = {
       dataSet: {
-        id: this.dataSet.id
+        id: this.dataSet?.id
       },
       data
     }
@@ -83,5 +82,3 @@ export class DataEntryForm extends localize(i18next)(LitElement) {
     }
   }
 }
-
-window.customElements.define('data-entry-form', DataEntryForm)

package/client/components/ooc-resolve-activity-view.ts

@@ -0,0 +1,174 @@
+import '@operato/dataset/ox-data-sample-view.js'
+
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { ScrollbarStyles } from '@operato/styles'
+
+@customElement('ooc-resolve-activity-view')
+class OocResolveActivityView extends localize(i18next)(LitElement) {
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
+
+        background-color: #fff;
+      }
+
+      div[content] {
+        flex: 1;
+        flex-direction: column;
+
+        display: flex;
+        overflow: auto;
+      }
+
+      ox-data-sample-view {
+        flex: 1;
+        padding: var(--padding-wide);
+        overflow: auto;
+      }
+
+      div[instruction] {
+        display: flex;
+        flex-direction: column;
+
+        padding: var(--padding-wide);
+      }
+
+      div[instruction] div {
+        display: flex;
+      }
+
+      div[instruction] div[content] {
+        display: flex;
+        min-height: 50px;
+      }
+
+      label[action] {
+        display: flex;
+        flex-direction: column;
+
+        padding: var(--padding-wide);
+      }
+
+      label[action] div {
+        display: flex;
+      }
+
+      mwc-icon {
+        color: var(--status-danger-color);
+      }
+
+      textarea {
+        border: var(--input-field-border);
+        border-radius: var(--input-border-radius);
+        padding: var(--input-field-padding);
+        font: var(--input-field-font);
+      }
+
+      .button-container {
+        display: flex;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
+
+  @property({ type: Object }) input?: any
+  @property({ type: Object }) output?: any
+
+  @state() dataOoc?: any
+
+  @query('textarea') actionTextArea!: HTMLTextAreaElement
+
+  render() {
+    const action = this.output?.action
+
+    return html`
+      <div content>
+        <ox-data-sample-view .dataSample=${this.dataOoc}></ox-data-sample-view>
+      </div>
+
+      <div instruction>
+        <div><mwc-icon>build_circle</mwc-icon> <span>corrective instruction</span></div>
+        <div content>${this.input?.instruction}</div>
+      </div>
+
+      <label action>
+        <div><mwc-icon>build_circle</mwc-icon> <span>corrective action</span></div>
+        <textarea
+          placeholder="시정 및 예방을 위한 조치 이행 사항을 입력해주세요."
+          .value=${action || ''}
+          @change=${this.onChangeAction}
+        ></textarea>
+      </label>
+    `
+  }
+
+  onChangeAction(e: Event) {
+    this.output ||= {}
+    this.output.action = (e.target as HTMLTextAreaElement).value
+
+    this.dispatchEvent(
+      new CustomEvent('change', {
+        detail: this.output
+      })
+    )
+  }
+
+  updated(changes) {
+    if (changes.has('input')) {
+      this.fetchDataOoc()
+    }
+  }
+
+  async fetchDataOoc() {
+    const id = this.input.dataOocId
+
+    if (id) {
+      const response = await client.query({
+        query: gql`
+          query ($id: String!) {
+            dataOoc(id: $id) {
+              id
+              name
+              description
+              ooc
+              oos
+              state
+              type
+              useCase
+              data
+              judgment
+              dataItems {
+                name
+                description
+                active
+                tag
+                type
+                unit
+                options
+                quota
+                spec
+              }
+              workDate
+              workShift
+              collectedAt
+            }
+          }
+        `,
+        variables: {
+          id
+        }
+      })
+
+      this.dataOoc = response.data.dataOoc
+    }
+  }
+}

package/client/components/ooc-review-activity-view.ts

@@ -0,0 +1,153 @@
+import '@operato/dataset/ox-data-sample-view.js'
+
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { ScrollbarStyles } from '@operato/styles'
+
+@customElement('ooc-review-activity-view')
+class OocReviewActivityView extends localize(i18next)(LitElement) {
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
+
+        background-color: #fff;
+      }
+
+      div[content] {
+        flex: 1;
+        flex-direction: column;
+
+        display: flex;
+        overflow: auto;
+      }
+
+      ox-data-sample-view {
+        flex: 1;
+        padding: var(--padding-wide);
+        overflow: auto;
+      }
+
+      label[instruction] {
+        display: flex;
+        flex-direction: column;
+
+        padding: var(--padding-wide);
+      }
+
+      label[instruction] div {
+        display: flex;
+      }
+
+      mwc-icon {
+        color: var(--status-danger-color);
+      }
+
+      textarea {
+        border: var(--input-field-border);
+        border-radius: var(--input-border-radius);
+        padding: var(--input-field-padding);
+        font: var(--input-field-font);
+      }
+
+      .button-container {
+        display: flex;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
+
+  @property({ type: Object }) input?: any
+  @property({ type: Object }) output?: any
+
+  @state() dataOoc?: any
+
+  @query('textarea') instructionTextArea!: HTMLTextAreaElement
+
+  render() {
+    const instruction = this.output?.instruction
+
+    return html`
+      <div content>
+        <ox-data-sample-view .dataSample=${this.dataOoc}></ox-data-sample-view>
+      </div>
+
+      <label instruction>
+        <div><mwc-icon>build_circle</mwc-icon> <span>corrective instruction</span></div>
+        <textarea
+          placeholder="시정 및 예방을 위한 조치 지시 사항을 입력해주세요."
+          .value=${instruction || ''}
+          @change=${this.onChangeInstruction}
+        ></textarea>
+      </label>
+    `
+  }
+
+  onChangeInstruction(e: Event) {
+    this.output ||= {}
+    this.output.instruction = (e.target as HTMLTextAreaElement).value
+
+    this.dispatchEvent(
+      new CustomEvent('change', {
+        detail: this.output
+      })
+    )
+  }
+
+  updated(changes) {
+    if (changes.has('input')) {
+      this.fetchDataOoc()
+    }
+  }
+
+  async fetchDataOoc() {
+    const id = this.input?.dataOocId
+
+    if (id) {
+      const response = await client.query({
+        query: gql`
+          query ($id: String!) {
+            dataOoc(id: $id) {
+              id
+              name
+              description
+              ooc
+              oos
+              state
+              type
+              useCase
+              data
+              judgment
+              dataItems {
+                name
+                description
+                active
+                tag
+                type
+                unit
+                options
+                quota
+                spec
+              }
+              workDate
+              workShift
+              collectedAt
+            }
+          }
+        `,
+        variables: {
+          id
+        }
+      })
+
+      this.dataOoc = response.data.dataOoc
+    }
+  }
+}