@things-factory/dataset 5.0.14 → 5.0.16

package/README.md

@@ -1,6 +1,28 @@
 # dataset
-## Architecture for data-samples
+## dataset relation diagram
+![ERD Diagram](./assets/diagram.jpg)
+- data_sets 하나가 여러 data_items 를 갖습니다. data_sets와 data_items 조합으로 CCP 항목들을 정의합니다.
+- data_sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key_##)에 해당 값을 저장합니다.
+- data_set_histories에는 data_sets의 변경 내역이 추가됩니다. data_sets의 id는 data_set_histories에 original_id로 추가됩니다. data_set_histories를 사용하려면 original_id 와 version이 필요합니다.
+- data_sensors에는 자동 수집되는 디바이스 정보를 정의합니다. data_sets를 참고하여 CCP를 관리하도록 합니다.
+- data_samples에는 입력받은 실제 데이터가 저장됩니다.
+- data_oocs에는 data_sets의 CCP 스펙을 벗어나는 경우, data_samples의 데이터를 복사하여 추가됩니다.
+- data_archives에는 다운로드 받기 위한 요청 정보 및 상태가 저장됩니다.
+
+## Architecture for collecting data samples
 ![Architecture for data-samples](./assets/data-samples.jpg)
+data_samples, data_oocs, data_set_histories의 테이블 변경 사항은 Database CDC로 Kafka에 전송 및 저장되고 토픽별로 다음의 lambda 들이 트리거되어 S3로 저장합니다. 이 S3 데이터는 Glue Crawler, Glue Data Catalog를 거쳐 Athena에서 조회할 수 있습니다. hive 스타일의 S3 파티션 정보들을 업데이트 하는 작업이 스케줄링되어 작동하고, 파티션 정보가 업데이트 된 S3파일들이 Athena에서 조회되므로 실시간으로 데이터가 반영되지는 않습니다.
+- [func-data-samples-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-samples-to-s3)
+- [func-data-oocs-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-oocs-to-s3)
+- [func-data-set-histories-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-set-histories-to-s3)
+
+개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-* 로 시작하는 lambda들을 사용합니다.
+## data-report process
+![data-report process](./assets/data-report.jpg)
+
+'shiny' uses the next lambda. But 'jasper' is not using this lambda now. Consider using the same data service.
+- [func-transform-data-samples](https://github.com/operatochef/serverless/tree/main/func-transform-data-samples)
+
 ## Partition Keys
 At the early stage, partition keys are desinged for dynamic partitioning for Athena. But It will be required so many AWS Glue crawlers also by each 'data-sets'.
 
@@ -17,18 +39,20 @@ Now partition keys are fixed
 - workdate: working date
 - workshift
 
-## PostgreSQL query for dataset
-```
-select 
-	data::json#>>'{product,0}' as product,
-	data::json#>>'{operation,0}' as operation 
-from data_samples ds 
-where 1=1
-and data_set_id = 'f1e394cb-9ace-4686-af14-817c619ae512'
-and data_set_version = 3
-and data::json#>>'{operation,0}' = '세척'
-order by created_at desc;
-```
-
-Reference.
-- [1] https://www.postgresql.org/docs/11/functions-json.html
+### Glue Catalog Table Indices
+Partitions can be indices. You can create new indices with combination of partitions.
+It affects query performance and finding partitions.
+![partitions and indices](./assets/glue-table-indices.png)
+
+## Timestamp of Data Samples
+sample data could have an own timestamp if it is from a sensor data.
+'collected_at' uses this timestamp.
+Manual type of data is used 'Date.now() and new Date()'
+Graphql might affect timezone of server os for Date type field.
+'TZ=UTC node' can help to solve this when server starts.
+Or use ``` process.env.TZ = 'UTC' ```.
+Now applied in 'create-data-samples.ts'
+
+## Data Archive
+제출용 데이터를 위해 data_samples를 다운로드 하는 기능입니다. csv.gz 형태로 다운로드 가능한 임시 url을 제공합니다. 해당 기능을 위해 [func-data-set-download](https://github.com/operatochef/serverless/tree/main/func-data-set-download) lambda를 사용합니다. 이는 수동 수집용 데이터에 적합합니다. 자동 수집 데이터는 lambda로 처리하기에는 많은 메모리가 필요할 수 있습니다. 따라서, 데이타셋 유형별 처리가 필요할 것 같습니다.
+Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.

package/client/pages/data-archive/data-archive-list-page.js

@@ -0,0 +1,290 @@
+import '@operato/data-grist'
+
+import gql from 'graphql-tag'
+import { css, html } from 'lit'
+import { connect } from 'pwa-helpers/connect-mixin'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { openPopup } from '@operato/layout'
+import { PageView, store } from '@operato/shell'
+import { CommonGristStyles, ScrollbarStyles } from '@operato/styles'
+import { isMobileDevice } from '@operato/utils'
+
+import './data-archive-request-popup'
+
+export class DataArchiveListPage extends connect(store)(localize(i18next)(PageView)) {
+  static get properties() {
+    return {
+      active: String,
+      gristConfig: Object,
+      mode: String
+    }
+  }
+
+  static get styles() {
+    return [
+      ScrollbarStyles,
+      CommonGristStyles,
+      css`
+        :host {
+          display: flex;
+          flex-direction: column;
+
+          overflow: hidden;
+        }
+
+        ox-grist {
+          overflow-y: auto;
+          flex: 1;
+        }
+      `
+    ]
+  }
+
+  get context() {
+    return {
+      title: i18next.t('title.data-archive list'),
+      help: 'dataset/data-archive',
+      actions: [],
+      exportable: {
+        name: i18next.t('title.data-archive list'),
+        data: this._exportableData.bind(this)
+      }
+    }
+  }
+
+  render() {
+    const mode = this.mode || (isMobileDevice() ? 'LIST' : 'GRID')
+
+    return html`
+      <ox-grist
+        .mode=${mode}
+        .config=${this.gristConfig}
+        .fetchHandler=${this.fetchHandler.bind(this)}
+        ?url-params-sensitive=${false /* this.active */}
+      >
+        <div slot="headroom">
+          <div id="filters">
+            <ox-filters-form autofocus></ox-filters-form>
+          </div>
+
+          <div id="sorters">
+            Sort
+            <mwc-icon
+              @click=${e => {
+                const target = e.currentTarget
+                this.renderRoot.querySelector('#sorter-control').open({
+                  right: 0,
+                  top: target.offsetTop + target.offsetHeight
+                })
+              }}
+              >expand_more</mwc-icon
+            >
+            <ox-popup id="sorter-control">
+              <ox-sorters-control> </ox-sorters-control>
+            </ox-popup>
+          </div>
+
+          <div id="modes">
+            <mwc-icon @click=${() => (this.mode = 'GRID')} ?active=${mode == 'GRID'}>grid_on</mwc-icon>
+            <mwc-icon @click=${() => (this.mode = 'LIST')} ?active=${mode == 'LIST'}>format_list_bulleted</mwc-icon>
+            <mwc-icon @click=${() => (this.mode = 'CARD')} ?active=${mode == 'CARD'}>apps</mwc-icon>
+          </div>
+
+          <mwc-button
+            dense
+            raised
+            label=${i18next.t('button.request')}
+            @click=${this.openArchivePopup.bind(this)}
+            icon="archive"
+          ></mwc-button>
+        </div>
+      </ox-grist>
+    `
+  }
+
+  get grist() {
+    return this.renderRoot.querySelector('ox-grist')
+  }
+
+  async pageInitialized(lifecycle) {
+    const today = new Date().toISOString().split('T')[0]
+
+    this.gristConfig = {
+      list: { fields: ['updater', 'updatedAt'] },
+      columns: [
+        { type: 'gutter', gutterName: 'sequence' },
+        // { type: 'gutter', gutterName: 'row-selector', multiple: true },
+        {
+          type: 'resource-object',
+          name: 'creator',
+          header: i18next.t('field.creator'),
+          sortable: true,
+          width: 120,
+          imex: true
+        },
+        {
+          type: 'datetime',
+          name: 'createdAt',
+          header: i18next.t('field.created_at'),
+          sortable: true,
+          width: 180,
+          imex: true
+        },
+        {
+          type: 'datetime',
+          name: 'updatedAt',
+          header: i18next.t('field.updated_at'),
+          sortable: true,
+          width: 180,
+          imex: true
+        },
+        {
+          type: 'json5',
+          name: 'requestParams',
+          header: i18next.t('field.request-params'),
+          record: {
+            editable: false
+          },
+          width: 200,
+          imex: true
+        },
+        {
+          type: 'string',
+          name: 'downloadUrl',
+          label: true,
+          header: i18next.t('field.download-url'),
+          record: {
+            editable: false
+          },
+          width: 240,
+          imex: true
+        },
+        {
+          type: 'string',
+          name: 'status',
+          label: true,
+          header: i18next.t('field.status'),
+          record: {
+            editable: false
+          },
+          sortable: true,
+          width: 120,
+          imex: true
+        }
+      ],
+      rows: {
+        appendable: false,
+        selectable: {
+          multiple: true
+        }
+      },
+      sorters: [
+        {
+          name: 'createdAt',
+          desc: true
+        }
+      ]
+    }
+  }
+
+  closePopupAndRefesh() {
+    this.popup && this.popup.close()
+    this.grist.fetch()
+  }
+
+  async openArchivePopup() {
+    this.popup = openPopup(
+      html`
+      <data-archive-request-popup
+        @requested="${this.closePopupAndRefesh.bind(this)}"
+        @created="${this.closePopupAndRefesh.bind(this)}"
+      ></data-archive-request-popup>`, 
+      {
+        backdrop: true,
+        size: 'small',
+        title: i18next.t('title.data-archive request popup')
+      })
+  }
+
+  async fetchHandler({ page, limit, sortings = [], filters = [] }) {
+    const response = await client.query({
+      query: gql`
+        query ($filters: [Filter!], $pagination: Pagination, $sortings: [Sorting!]) {
+          responses: dataArchives(filters: $filters, pagination: $pagination, sortings: $sortings) {
+            items {
+              id
+              type
+              requestParams
+              downloadUrl
+              status
+              creator {
+                id
+                name
+              }
+              updater {
+                id
+                name
+              }
+              updatedAt
+              createdAt
+            }
+            total
+          }
+        }
+      `,
+      variables: {
+        filters,
+        pagination: { page, limit },
+        sortings
+      }
+    })
+
+    return {
+      total: response.data.responses.total || 0,
+      records: response.data.responses.items || []
+    }
+  }
+
+  _exportableData() {
+    let records = []
+    if (this.grist.selected && this.grist.selected.length > 0) {
+      records = this.grist.selected
+    } else {
+      records = this.grist.data.records
+    }
+
+    var headerSetting = this.grist.compiledConfig.columns
+      .filter(column => column.type !== 'gutter' && column.record !== undefined && column.imex !== undefined)
+      .map(column => {
+        return column.imex === true
+          ? {
+              header: column.header.renderer(),
+              key: column.name,
+              width: column.width,
+              type: column.type
+            }
+          : column.imex
+      })
+
+    var data = records.map(item => {
+      return {
+        id: item.id,
+        ...this.gristConfig.columns
+          .filter(column => column.type !== 'gutter' && column.record !== undefined && column.imex !== undefined)
+          .reduce((record, column) => {
+            const key = column.imex === true ? column.name : column.imex.key
+            record[key] = key
+              .split('.')
+              .reduce((obj, key) => (obj && obj[key] !== 'undefined' ? obj[key] : undefined), item)
+            return record
+          }, {})
+      }
+    })
+
+    return { header: headerSetting, data: data }
+  }
+}
+
+window.customElements.define('data-archive-list-page', DataArchiveListPage)

package/client/pages/data-archive/data-archive-request-popup.js

@@ -0,0 +1,180 @@
+import { SingleColumnFormStyles } from '@things-factory/form-ui'
+import '@things-factory/grist-ui'
+import { i18next, localize } from '@things-factory/i18n-base'
+import { client, CustomAlert } from '@things-factory/shell'
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit-element'
+import moment from 'moment-timezone'
+
+class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
+  static get properties() {
+    return {
+      dataSetTypes: Array
+    }
+  }
+
+  static get styles() {
+    return [
+      SingleColumnFormStyles,
+      css`
+        :host {
+          padding: 10px;
+          display: flex;
+          flex-direction: column;
+          overflow-x: overlay;
+          background-color: var(--main-section-background-color);
+        }
+        .button-container {
+          padding: var(--button-container-padding);
+          margin: var(--button-container-margin);
+          text-align: var(--button-container-align);
+          background-color: var(--button-container-background);
+          height: var(--button-container-height);
+        }
+      `
+    ]
+  }
+
+  render() {
+    return html`
+      <form id="input-form" name="generation" class="single-column-form" @submit=${e => this.onSubmit(e)}>
+        <fieldset>
+          <label>${i18next.t('label.start-date')}</label>
+          <input type="month" name="startDate" required/>
+          <label>${i18next.t('label.end-date')}</label>
+          <input type="month" name="endDate" .value=${moment().format('YYYY-MM')}/>
+          <label>${i18next.t('label.data-set-type')}</label>
+          <select name="type" required>
+            ${(this.dataSetTypes || []).map(
+              t => html`<option value="${t && t.value}">${t && t.display}</option>`
+            )}
+          </select> 
+        </fieldset>
+      </form>
+
+      <div class="button-container">
+        <mwc-button raised @click="${this.requestArchive}" label="${i18next.t('button.submit')}"></mwc-button>
+      </div>
+    `
+  }
+
+  async firstUpdated() {
+    
+  }
+
+  async onSubmit(e) {
+    e.preventDefault()
+    this.requestArchive()
+  }
+
+
+  serializeFormData() {
+    const obj = {}
+    
+    Array.from(this.shadowRoot.querySelectorAll('form#input-form input, select')).forEach(field => {
+      if (!field.hasAttribute('hidden') && field.value) {
+        obj[field.name] = field.type === 'checkbox' ? field.checked : field.value
+      }
+    })
+
+    if (obj['startDate']) {
+      obj['startDate'] += '-01'
+    }
+
+    if (obj['endDate']) {
+      const endDate = moment(obj['endDate']).endOf('month')
+      const today = moment()
+      const format = 'YYYY-MM-DD'
+
+      obj['endDate'] = endDate > today ? today.format(format) : endDate.format(format)
+    }
+
+    return obj
+  }
+
+  /** request download url. */
+  async requestArchive() {
+    // #1 request download url
+    const dataArchive = {
+      requestParams: this.serializeFormData(),
+      status: 'requested'
+    }
+
+    const response = await client.mutate({
+      mutation: gql`
+        mutation createDataArchive($dataArchive: NewDataArchive!) {
+          createDataArchive(dataArchive: $dataArchive) {
+            id
+            status
+          }
+        }
+      `,
+      variables: { dataArchive }
+    })
+
+    if (!response.errors) {
+      this.dispatchEvent(new CustomEvent('requested', {}))
+
+      const { createDataArchive: { id } } = response.data
+      dataArchive['id'] = id
+
+      this._generateArchiveAndDownloadUrl(dataArchive)
+
+      await CustomAlert({
+        type: 'info',
+        title: i18next.t('title.ready'),
+        text: i18next.t('text.data-archive waits'),
+        confirmButton: { text: i18next.t('button.confirm') }
+      })
+      
+    } else {
+      console.error(response.errors)
+      this.showToast(i18next.t('text.failed'))
+    }
+  }
+
+  async _generateArchiveAndDownloadUrl(dataArchive) {
+    const response = await client.mutate({
+      mutation: gql`
+        mutation generatePresignedUrl($patch: DataArchivePatch!) {
+          generatePresignedUrl(patch: $patch) {
+            id
+            downloadUrl
+            status
+          }
+        }
+      `,
+      variables: { patch: dataArchive }
+    })
+
+    if (!response.errors) {     
+      this.showToast(i18next.t('title.data-archive downloads ready'))
+      this.dispatchEvent(new CustomEvent('created', {}))
+    } else {
+      console.error(response.errors)
+      this.showToast(i18next.t('text.failed'))
+    }
+  }
+
+  showToast(message) {
+    document.dispatchEvent(new CustomEvent('notify', { detail: { message } }))
+  }
+
+  constructor() {
+    super()
+
+    this.dataSetTypes = [
+      {},
+      {
+        display: i18next.t('text.manually collected'),
+        value: 'manual'
+      },
+      {
+        display: i18next.t('text.automatically collected'),
+        value: 'automatic'
+      }
+    ]
+  }
+}
+
+window.customElements.define('data-archive-request-popup', DataArchiveRequestPopup)

package/client/pages/data-key-set/data-key-set-list-page.js

@@ -12,7 +12,40 @@ import { notify, openPopup } from '@operato/layout'
 import { navigate, PageView, store } from '@operato/shell'
 import { CommonButtonStyles, CommonGristStyles, ScrollbarStyles } from '@operato/styles'
 import { isMobileDevice } from '@operato/utils'
+import { encodeUrlParams } from '@things-factory/utils'
 
+import { getEditor, getRenderer } from '@operato/data-grist'
+
+const REPORT_TYPES = [
+  {
+    display: '',
+    value: ''
+  },
+  {
+    display: 'Generated',
+    value: 'generated'
+  },
+  {
+    display: 'Embed',
+    value: 'embed'
+  },
+  {
+    display: 'Page',
+    value: 'page'
+  },
+  {
+    display: 'External URL',
+    value: 'external'
+  },
+  {
+    display: 'Jasper',
+    value: 'jasper'
+  },
+  {
+    display: 'Shiny',
+    value: 'shiny'
+  }
+]
 export class DataKeySetListPage extends connect(store)(localize(i18next)(PageView)) {
   static get properties() {
     return {
@@ -123,7 +156,7 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
               if (!record.id) return
               const popup = openPopup(html` <data-key-item-list .dataKeySet=${record}></data-key-item-list> `, {
                 backdrop: true,
-                help: 'data-set/ui/data-key-item-list',
+                help: 'dataset/ui/data-key-item-list',
                 size: 'large',
                 title: i18next.t('title.data-key-item list')
               })
@@ -136,13 +169,45 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
         {
           type: 'gutter',
           gutterName: 'button',
-          icon: 'newspaper',
+          icon: 'checklist',
           handlers: {
             click: (columns, data, column, record, rowIndex) => {
               navigate(`data-sample-search/${record.id}`)
             }
           }
         },
+        {
+          type: 'gutter',
+          gutterName: 'button',
+          icon: 'newspaper',
+          handlers: {
+            click: (columns, data, column, record, rowIndex) => {
+              try {
+                const { id:dataKeySetId, reportType, reportView } = record
+                switch (reportType) {
+                  case 'generated':
+                    break
+                  case 'page':
+                    break
+                  case 'embed':
+                    break
+                  case 'external':
+                    reportView && window.open(reportView, '_blank')
+                    break
+                  case 'jasper':
+                  case 'shiny':
+                    const urlParams = encodeUrlParams({ dataKeySetId, reportType, reportView })
+                    navigate(`data-report-samples/?${urlParams}`)
+                    break
+                  default:
+                    break
+                }
+              } catch(ex) {
+                console.log(ex)
+              }
+            }
+          }
+        },
         {
           type: 'string',
           name: 'name',
@@ -176,6 +241,34 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
           sortable: true,
           width: 60
         },
+        {
+          type: 'select',
+          name: 'reportType',
+          label: true,
+          header: i18next.t('field.report-type'),
+          record: {
+            editable: true,
+            options: REPORT_TYPES
+          },
+          width: 80
+        },
+        {
+          type: 'string',
+          name: 'reportView',
+          header: i18next.t('field.report-view'),
+          record: {
+            editable: true,
+            editor: function (value, column, record, rowIndex, field) {
+              var type = record.reportType !== 'custom' ? 'string' : 'script'
+              return getEditor(type)(value, column, record, rowIndex, field)
+            },
+            renderer: function (value, column, record, rowIndex, field) {
+              var type = record.reportType !== 'custom' ? 'string' : 'string'
+              return getRenderer(type)(value, column, record, rowIndex, field)
+            }
+          },
+          width: 140
+        },
         {
           type: 'resource-object',
           name: 'updater',
@@ -226,6 +319,8 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
               name
               description
               active
+              reportType
+              reportView
               dataKeyItems {
                 name
                 description
@@ -315,7 +410,7 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
 
   async exportHandler() {
     const exportTargets = this.grist.selected.length ? this.grist.selected : this.grist.dirtyData.records
-    const targetFieldSet = new Set(['id', 'name', 'description', 'active'])
+    const targetFieldSet = new Set(['id', 'name', 'description', 'active', 'reportType', 'reportView'])
 
     return exportTargets.map(dataKeySet => {
       let tempObj = {}