@things-factory/dataset 6.0.25 → 6.0.27

package/README.md

@@ -1,8 +1,11 @@
 # dataset
+
 ## dataset relation diagram
-![ERD Diagram](./assets/diagram.jpg)
+
+![Dataset ERD Diagram](./assets/images/dataset-diagram.jpg)
+
 - data_sets 하나가 여러 data_items 를 갖습니다. data_sets와 data_items 조합으로 CCP 항목들을 정의합니다.
-- data_sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key_##)에 해당 값을 저장합니다.
+- data*sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key*##)에 해당 값을 저장합니다.
 - data_set_histories에는 data_sets의 변경 내역이 추가됩니다. data_sets의 id는 data_set_histories에 original_id로 추가됩니다. data_set_histories를 사용하려면 original_id 와 version이 필요합니다.
 - data_sensors에는 자동 수집되는 디바이스 정보를 정의합니다. data_sets를 참고하여 CCP를 관리하도록 합니다.
 - data_samples에는 입력받은 실제 데이터가 저장됩니다.
@@ -10,29 +13,38 @@
 - data_archives에는 다운로드 받기 위한 요청 정보 및 상태가 저장됩니다.
 
 ## Architecture for collecting data samples
-![Architecture for data-samples](./assets/data-samples.jpg)
+
+![Architecture for data-samples](./assets/images/data-samples.jpg)
 data_samples, data_oocs, data_set_histories의 테이블 변경 사항은 Database CDC로 Kafka에 전송 및 저장되고 토픽별로 다음의 lambda 들이 트리거되어 S3로 저장합니다. 이 S3 데이터는 Glue Crawler, Glue Data Catalog를 거쳐 Athena에서 조회할 수 있습니다. hive 스타일의 S3 파티션 정보들을 업데이트 하는 작업이 스케줄링되어 작동하고, 파티션 정보가 업데이트 된 S3파일들이 Athena에서 조회되므로 실시간으로 데이터가 반영되지는 않습니다.
+
 - [func-data-samples-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-samples-to-s3)
 - [func-data-oocs-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-oocs-to-s3)
 - [func-data-set-histories-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-set-histories-to-s3)
 
-개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-* 로 시작하는 lambda들을 사용합니다.
+개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-\* 로 시작하는 lambda들을 사용합니다.
+
 ## data-report process
-![data-report process](./assets/data-report.jpg)
+
+![data-report process](./assets/images/data-report.jpg)
 
 'shiny' uses the next lambda. But 'jasper' is not using this lambda now. Consider using the same data service.
+
 - [func-transform-data-samples](https://github.com/operatochef/serverless/tree/main/func-transform-data-samples)
 
 ## Partition Keys
+
 At the early stage, partition keys are desinged for dynamic partitioning for Athena. But It will be required so many AWS Glue crawlers also by each 'data-sets'.
 
-Partition keys are related for S3 request limitations. 
-> __3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket__
+Partition keys are related for S3 request limitations.
+
+> **3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket**
+
 - https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
 
 ### :warning: This concept will be deprecated.
 
-Now partition keys are fixed  
+Now partition keys are fixed
+
 - domain
 - datasetid
 - date: local time date
@@ -40,19 +52,22 @@ Now partition keys are fixed
 - workshift
 
 ### Glue Catalog Table Indices
+
 Partitions can be indices. You can create new indices with combination of partitions.
 It affects query performance and finding partitions.
-![partitions and indices](./assets/glue-table-indices.png)
+![partitions and indices](./assets/images/glue-table-indices-for-dataset.png)
 
 ## Timestamp of Data Samples
+
 sample data could have an own timestamp if it is from a sensor data.
 'collected_at' uses this timestamp.
 Manual type of data is used 'Date.now() and new Date()'
 Graphql might affect timezone of server os for Date type field.
 'TZ=UTC node' can help to solve this when server starts.
-Or use ``` process.env.TZ = 'UTC' ```.
+Or use `process.env.TZ = 'UTC'`.
 Now applied in 'create-data-samples.ts'
 
 ## Data Archive
+
 제출용 데이터를 위해 data_samples를 다운로드 하는 기능입니다. csv.gz 형태로 다운로드 가능한 임시 url을 제공합니다. 해당 기능을 위해 [func-data-set-download](https://github.com/operatochef/serverless/tree/main/func-data-set-download) lambda를 사용합니다. 이는 수동 수집용 데이터에 적합합니다. 자동 수집 데이터는 lambda로 처리하기에는 많은 메모리가 필요할 수 있습니다. 따라서, 데이타셋 유형별 처리가 필요할 것 같습니다.
-Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.
+Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.

package/client/{bootstrap.js → bootstrap.ts}

@@ -1,3 +1,5 @@
+import './components/data-ooc-activity-view' /* refered by the activity template (activity-ooc) on server */
+
 import '@operato/app/filter-renderer.js' /* register resource-object filter renderer */
 
 import { OxGristEditorCode } from '@operato/app/grist-editor/ox-grist-editor-code.js'

package/client/components/{data-entry-form.js → data-entry-form.ts}

@@ -2,47 +2,43 @@ import '@operato/dataset/ox-data-entry-form.js'
 
 import gql from 'graphql-tag'
 import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
 
 import { client } from '@operato/graphql'
 import { i18next, localize } from '@operato/i18n'
 import { ScrollbarStyles } from '@operato/styles'
+import { OxDataEntryForm } from '@operato/dataset/ox-data-entry-form.js'
+import { DataSet } from '@operato/dataset'
 
+@customElement('data-entry-form')
 export class DataEntryForm extends localize(i18next)(LitElement) {
-  static get properties() {
-    return {
-      dataSet: Object
-    }
-  }
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
 
-  static get styles() {
-    return [
-      ScrollbarStyles,
-      css`
-        :host {
-          display: flex;
-          flex-direction: column;
+        background-color: #fff;
+      }
 
-          background-color: #fff;
-        }
+      ox-data-entry-form {
+        flex: 1;
+        padding: 10px;
+        overflow: auto;
+      }
 
-        ox-data-entry-form {
-          flex: 1;
-          padding: 10px;
-          overflow: auto;
-        }
+      .button-container {
+        display: flex;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
 
-        .button-container {
-          display: flex;
-          margin-left: auto;
-          padding: var(--padding-default);
-        }
-      `
-    ]
-  }
+  @property({ type: Object }) dataSet?: DataSet & { id: string }
 
-  get entryForm() {
-    return this.renderRoot.querySelector('ox-data-entry-form')
-  }
+  @query('ox-data-entry-form') entryForm!: OxDataEntryForm
 
   render() {
     return html`
@@ -54,10 +50,13 @@ export class DataEntryForm extends localize(i18next)(LitElement) {
   }
 
   async _updateDataItems() {
+    //@ts-ignore TODO replace with following line and confirm
     const data = this.entryForm.buildValue()
+    // const data = this.entryForm.value
+
     const dataSample = {
       dataSet: {
-        id: this.dataSet.id
+        id: this.dataSet?.id
       },
       data
     }
@@ -83,5 +82,3 @@ export class DataEntryForm extends localize(i18next)(LitElement) {
     }
   }
 }
-
-window.customElements.define('data-entry-form', DataEntryForm)

package/client/components/data-ooc-activity-view.ts

@@ -0,0 +1,153 @@
+import '@operato/dataset/ox-data-sample-view.js'
+
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { ScrollbarStyles } from '@operato/styles'
+
+@customElement('data-ooc-activity-view')
+class DataOocActivityView extends localize(i18next)(LitElement) {
+  static styles = [
+    ScrollbarStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
+
+        background-color: #fff;
+      }
+
+      div[content] {
+        flex: 1;
+        flex-direction: column;
+
+        display: flex;
+        overflow: auto;
+      }
+
+      ox-data-ooc-view {
+        flex: 1;
+        padding: var(--padding-wide);
+        overflow: auto;
+      }
+
+      label[comment] {
+        display: flex;
+        flex-direction: column;
+
+        padding: var(--padding-wide);
+      }
+
+      label[comment] div {
+        display: flex;
+      }
+
+      mwc-icon {
+        color: var(--status-danger-color);
+      }
+
+      textarea {
+        border: var(--input-field-border);
+        border-radius: var(--input-border-radius);
+        padding: var(--input-field-padding);
+        font: var(--input-field-font);
+      }
+
+      .button-container {
+        display: flex;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
+
+  @property({ type: Object }) input?: any
+  @property({ type: Object }) output?: any
+
+  @state() dataOoc?: any
+
+  @query('textarea') commentTextArea!: HTMLTextAreaElement
+
+  render() {
+    const comment = this.output?.comment
+
+    return html`
+      <div content>
+        <ox-data-sample-view .dataSample=${this.dataOoc}></ox-data-sample-view>
+      </div>
+
+      <label comment>
+        <div><mwc-icon>build_circle</mwc-icon> <span>correction activity</span></div>
+        <textarea
+          placeholder="조치 내용을 입력해주세요."
+          .value=${comment || ''}
+          @change=${this.onChangeComment}
+        ></textarea>
+      </label>
+    `
+  }
+
+  onChangeComment(e: Event) {
+    this.output = this.output || {}
+    this.output.comment = (e.target as HTMLTextAreaElement).value
+
+    this.dispatchEvent(
+      new CustomEvent('change', {
+        detail: this.output
+      })
+    )
+  }
+
+  updated(changes) {
+    if (changes.has('input')) {
+      this.fetchDataOoc()
+    }
+  }
+
+  async fetchDataOoc() {
+    const id = this.input.dataOocId
+
+    if (id) {
+      const response = await client.query({
+        query: gql`
+          query ($id: String!) {
+            dataOoc(id: $id) {
+              id
+              name
+              description
+              ooc
+              oos
+              state
+              type
+              useCase
+              data
+              judgment
+              dataItems {
+                name
+                description
+                active
+                tag
+                type
+                unit
+                options
+                quota
+                spec
+              }
+              workDate
+              workShift
+              collectedAt
+            }
+          }
+        `,
+        variables: {
+          id
+        }
+      })
+
+      this.dataOoc = response.data.dataOoc
+    }
+  }
+}

package/client/pages/data-archive/{data-archive-list-page.js → data-archive-list-page.ts}

@@ -2,51 +2,79 @@ import '@operato/data-grist'
 
 import gql from 'graphql-tag'
 import { css, html } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
 import { connect } from 'pwa-helpers/connect-mixin'
 
+import {
+  getEditor,
+  getRenderer,
+  ColumnConfig,
+  DataGrist,
+  FetchOption,
+  SortersControl,
+  GristRecord
+} from '@operato/data-grist'
 import { client } from '@operato/graphql'
 import { i18next, localize } from '@operato/i18n'
 import { openPopup } from '@operato/layout'
 import { PageView, store } from '@operato/shell'
-import { CommonGristStyles, ScrollbarStyles } from '@operato/styles'
+import { CommonButtonStyles, CommonGristStyles, ScrollbarStyles } from '@operato/styles'
 import { isMobileDevice } from '@operato/utils'
+import { OxPopup, PopupHandle } from '@operato/popup'
 
 import './data-archive-request-popup'
 
+@customElement('data-archive-list-page')
 export class DataArchiveListPage extends connect(store)(localize(i18next)(PageView)) {
-  static get properties() {
-    return {
-      active: String,
-      gristConfig: Object,
-      mode: String
-    }
-  }
+  static styles = [
+    ScrollbarStyles,
+    CommonGristStyles,
+    css`
+      :host {
+        display: flex;
+        flex-direction: column;
 
-  static get styles() {
-    return [
-      ScrollbarStyles,
-      CommonGristStyles,
-      css`
-        :host {
-          display: flex;
-          flex-direction: column;
+        overflow: hidden;
+      }
 
-          overflow: hidden;
-        }
+      ox-grist {
+        overflow-y: auto;
+        flex: 1;
+      }
+    `
+  ]
 
-        ox-grist {
-          overflow-y: auto;
-          flex: 1;
-        }
-      `
-    ]
-  }
+  @state() gristConfig: any
+  @state() mode: 'CARD' | 'GRID' | 'LIST' = isMobileDevice() ? 'CARD' : 'GRID'
+
+  @query('ox-grist') private grist!: DataGrist
+  @query('#sorter-control') private sortersControl!: OxPopup
+
+  private popup?: PopupHandle
 
   get context() {
     return {
-      title: i18next.t('title.data-archive list'),
+      search: {
+        handler: (search: string) => {
+          this.grist.searchText = search
+        },
+        placeholder: i18next.t('title.data-archive list'),
+        value: this.grist.searchText
+      },
+      filter: {
+        handler: () => {
+          this.grist.toggleHeadroom()
+        }
+      },
       help: 'dataset/data-archive',
-      actions: [],
+      actions: [
+        {
+          ...CommonButtonStyles.save,
+          title: i18next.t('button.request-archive'),
+          action: this.openArchivePopup.bind(this),
+          icon: 'archive'
+        }
+      ],
       exportable: {
         name: i18next.t('title.data-archive list'),
         data: this._exportableData.bind(this)
@@ -66,7 +94,7 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
       >
         <div slot="headroom">
           <div id="filters">
-            <ox-filters-form autofocus></ox-filters-form>
+            <ox-filters-form autofocus without-search></ox-filters-form>
           </div>
 
           <div id="sorters">
@@ -74,7 +102,7 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
             <mwc-icon
               @click=${e => {
                 const target = e.currentTarget
-                this.renderRoot.querySelector('#sorter-control').open({
+                this.sortersControl.open({
                   right: 0,
                   top: target.offsetTop + target.offsetHeight
                 })
@@ -91,23 +119,11 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
             <mwc-icon @click=${() => (this.mode = 'LIST')} ?active=${mode == 'LIST'}>format_list_bulleted</mwc-icon>
             <mwc-icon @click=${() => (this.mode = 'CARD')} ?active=${mode == 'CARD'}>apps</mwc-icon>
           </div>
-
-          <mwc-button
-            dense
-            raised
-            label=${i18next.t('button.request')}
-            @click=${this.openArchivePopup.bind(this)}
-            icon="archive"
-          ></mwc-button>
         </div>
       </ox-grist>
     `
   }
 
-  get grist() {
-    return this.renderRoot.querySelector('ox-grist')
-  }
-
   async pageInitialized(lifecycle) {
     const today = new Date().toISOString().split('T')[0]
 
@@ -190,25 +206,25 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
   }
 
   closePopupAndRefesh() {
-    this.popup && this.popup.close()
+    this.popup?.close()
     this.grist.fetch()
   }
 
   async openArchivePopup() {
     this.popup = openPopup(
-      html`
-      <data-archive-request-popup
-        @requested="${this.closePopupAndRefesh.bind(this)}"
-        @created="${this.closePopupAndRefesh.bind(this)}"
-      ></data-archive-request-popup>`, 
+      html` <data-archive-request-popup
+        @requested=${this.closePopupAndRefesh.bind(this)}
+        @created=${this.closePopupAndRefesh.bind(this)}
+      ></data-archive-request-popup>`,
       {
         backdrop: true,
         size: 'small',
         title: i18next.t('title.data-archive request popup')
-      })
+      }
+    )
   }
 
-  async fetchHandler({ page, limit, sortings = [], filters = [] }) {
+  async fetchHandler({ page, limit, sortings = [], filters = [] }: FetchOption) {
     const response = await client.query({
       query: gql`
         query ($filters: [Filter!], $pagination: Pagination, $sortings: [Sorting!]) {
@@ -248,7 +264,7 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
   }
 
   _exportableData() {
-    let records = []
+    let records = [] as GristRecord[]
     if (this.grist.selected && this.grist.selected.length > 0) {
       records = this.grist.selected
     } else {
@@ -260,7 +276,7 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
       .map(column => {
         return column.imex === true
           ? {
-              header: column.header.renderer(),
+              header: column.header.renderer(column),
               key: column.name,
               width: column.width,
               type: column.type
@@ -286,5 +302,3 @@ export class DataArchiveListPage extends connect(store)(localize(i18next)(PageVi
     return { header: headerSetting, data: data }
   }
 }
-
-window.customElements.define('data-archive-list-page', DataArchiveListPage)

package/client/pages/data-archive/{data-archive-request-popup.js → data-archive-request-popup.ts}

@@ -1,38 +1,41 @@
-import { SingleColumnFormStyles } from '@things-factory/form-ui'
-import { i18next, localize } from '@things-factory/i18n-base'
-import { client, CustomAlert } from '@things-factory/shell'
 import gql from 'graphql-tag'
-import { css, html, LitElement } from 'lit-element'
+import { css, html, LitElement } from 'lit'
+import { customElement, property, query, state } from 'lit/decorators.js'
+
+import { i18next, localize } from '@operato/i18n'
+import { client } from '@operato/graphql'
+import { SingleColumnFormStyles } from '@things-factory/form-ui'
+
 import moment from 'moment-timezone'
+import Swal from 'sweetalert2'
 
+@customElement('data-archive-request-popup')
 class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
-  static get properties() {
-    return {
-      dataSetTypes: Array
-    }
-  }
+  static styles = [
+    SingleColumnFormStyles,
+    css`
+      :host {
+        padding: 10px;
+        display: flex;
+        flex-direction: column;
+        overflow-x: overlay;
+        background-color: var(--main-section-background-color);
+      }
 
-  static get styles() {
-    return [
-      SingleColumnFormStyles,
-      css`
-        :host {
-          padding: 10px;
-          display: flex;
-          flex-direction: column;
-          overflow-x: overlay;
-          background-color: var(--main-section-background-color);
-        }
-        .button-container {
-          padding: var(--button-container-padding);
-          margin: var(--button-container-margin);
-          text-align: var(--button-container-align);
-          background-color: var(--button-container-background);
-          height: var(--button-container-height);
-        }
-      `
-    ]
-  }
+      form {
+        overflow: auto;
+      }
+
+      .button-container {
+        display: flex;
+        margin-top: auto;
+        margin-left: auto;
+        padding: var(--padding-default);
+      }
+    `
+  ]
+
+  @state() dataSetTypes: any
 
   render() {
     return html`
@@ -44,13 +47,13 @@ class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
           <input type="month" name="endDate" .value=${moment().format('YYYY-MM')} />
           <label>${i18next.t('label.data-set-type')}</label>
           <select name="type" required>
-            ${(this.dataSetTypes || []).map(t => html`<option value="${t && t.value}">${t && t.display}</option>`)}
+            ${(this.dataSetTypes || []).map(t => html`<option value=${t && t.value}>${t && t.display}</option>`)}
           </select>
         </fieldset>
       </form>
 
       <div class="button-container">
-        <mwc-button raised @click="${this.requestArchive}" label="${i18next.t('button.submit')}"></mwc-button>
+        <mwc-button raised @click=${this.requestArchive} label=${String(i18next.t('button.submit'))}></mwc-button>
       </div>
     `
   }
@@ -65,9 +68,10 @@ class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
   serializeFormData() {
     const obj = {}
 
-    Array.from(this.shadowRoot.querySelectorAll('form#input-form input, select')).forEach(field => {
-      if (!field.hasAttribute('hidden') && field.value) {
-        obj[field.name] = field.type === 'checkbox' ? field.checked : field.value
+    Array.from(this.shadowRoot!.querySelectorAll('form#input-form input, select')).forEach((field: Element) => {
+      const input = field as HTMLInputElement
+      if (!input.hasAttribute('hidden') && input.value) {
+        obj[input.name] = input.type === 'checkbox' ? input.checked : input.value
       }
     })
 
@@ -116,12 +120,12 @@ class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
 
       this._generateArchiveAndDownloadUrl(dataArchive)
 
-      await CustomAlert({
+      await Swal.fire({
         type: 'info',
         title: i18next.t('title.ready'),
         text: i18next.t('text.data-archive waits'),
         confirmButton: { text: i18next.t('button.confirm') }
-      })
+      } as any)
     } else {
       console.error(response.errors)
       this.showToast(i18next.t('text.failed'))
@@ -171,5 +175,3 @@ class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
     ]
   }
 }
-
-window.customElements.define('data-archive-request-popup', DataArchiveRequestPopup)