@things-factory/dataset 6.0.0-alpha.6 → 6.0.0-alpha.8

package/README.md

@@ -1,11 +1,27 @@
 # dataset
-## Architecture for data-samples
+## dataset relation diagram
+![ERD Diagram](./assets/diagram.jpg)
+- data_sets 하나가 여러 data_items 를 갖습니다. data_sets와 data_items 조합으로 CCP 항목들을 정의합니다.
+- data_sets는 또한 data_key_sets를 참고할 수 있습니다. data_key_sets에서는 data_samples, data_oocs의 조회 인덱스를 위한 key(key_01~key_05)를 정의합니다. data_samples, data_oocs 데이타 생성 시점에, data 컬럼 값으로 부터 별도 컬럼(key_##)에 해당 값을 저장합니다.
+- data_set_histories에는 data_sets의 변경 내역이 추가됩니다. data_sets의 id는 data_set_histories에 original_id로 추가됩니다. data_set_histories를 사용하려면 original_id 와 version이 필요합니다.
+- data_sensors에는 자동 수집되는 디바이스 정보를 정의합니다. data_sets를 참고하여 CCP를 관리하도록 합니다.
+- data_samples에는 입력받은 실제 데이터가 저장됩니다.
+- data_oocs에는 data_sets의 CCP 스펙을 벗어나는 경우, data_samples의 데이터를 복사하여 추가됩니다.
+- data_archives에는 다운로드 받기 위한 요청 정보 및 상태가 저장됩니다.
+
+## Architecture for collecting data samples
 ![Architecture for data-samples](./assets/data-samples.jpg)
+data_samples, data_oocs, data_set_histories의 테이블 변경 사항은 Database CDC로 Kafka에 전송 및 저장되고 토픽별로 다음의 lambda 들이 트리거되어 S3로 저장합니다. 이 S3 데이터는 Glue Crawler, Glue Data Catalog를 거쳐 Athena에서 조회할 수 있습니다. hive 스타일의 S3 파티션 정보들을 업데이트 하는 작업이 스케줄링되어 작동하고, 파티션 정보가 업데이트 된 S3파일들이 Athena에서 조회되므로 실시간으로 데이터가 반영되지는 않습니다.
+- [func-data-samples-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-samples-to-s3)
+- [func-data-oocs-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-oocs-to-s3)
+- [func-data-set-histories-to-s3](https://github.com/operatochef/serverless/tree/main/func-data-set-histories-to-s3)
+
+개발용으로는 reference-app을 사용하고 있으며, 여기서는 func-dev-* 로 시작하는 lambda들을 사용합니다.
 ## data-report process
 ![data-report process](./assets/data-report.jpg)
 
 'shiny' uses the next lambda. But 'jasper' is not using this lambda now. Consider using the same data service.
-- https://github.com/operatochef/serverless/tree/main/func-transform-data-samples
+- [func-transform-data-samples](https://github.com/operatochef/serverless/tree/main/func-transform-data-samples)
 
 ## Partition Keys
 At the early stage, partition keys are desinged for dynamic partitioning for Athena. But It will be required so many AWS Glue crawlers also by each 'data-sets'.
@@ -27,7 +43,8 @@ Now partition keys are fixed
 Partitions can be indices. You can create new indices with combination of partitions.
 It affects query performance and finding partitions.
 ![partitions and indices](./assets/glue-table-indices.png)
-## Data Samples
+
+## Timestamp of Data Samples
 sample data could have an own timestamp if it is from a sensor data.
 'collected_at' uses this timestamp.
 Manual type of data is used 'Date.now() and new Date()'
@@ -36,19 +53,6 @@ Graphql might affect timezone of server os for Date type field.
 Or use ``` process.env.TZ = 'UTC' ```.
 Now applied in 'create-data-samples.ts'
 
-## PostgreSQL query for dataset
-This is not using now.
-```
-select 
-	data::json#>>'{product,0}' as product,
-	data::json#>>'{operation,0}' as operation 
-from data_samples ds 
-where 1=1
-and data_set_id = 'f1e394cb-9ace-4686-af14-817c619ae512'
-and data_set_version = 3
-and data::json#>>'{operation,0}' = '세척'
-order by created_at desc;
-```
-
-Reference.
-- [1] https://www.postgresql.org/docs/11/functions-json.html
+## Data Archive
+제출용 데이터를 위해 data_samples를 다운로드 하는 기능입니다. csv.gz 형태로 다운로드 가능한 임시 url을 제공합니다. 해당 기능을 위해 [func-data-set-download](https://github.com/operatochef/serverless/tree/main/func-data-set-download) lambda를 사용합니다. 이는 수동 수집용 데이터에 적합합니다. 자동 수집 데이터는 lambda로 처리하기에는 많은 메모리가 필요할 수 있습니다. 따라서, 데이타셋 유형별 처리가 필요할 것 같습니다.
+Glue Job을 이용한 처리를 진행해보았으나, 시간이 너무 오래걸리는 문제가 있었습니다. 해당 내용은 AWS ibex계정의 Glue Jobs - 'reference_app_samples_archive' 리소스를 참고바랍니다.

package/client/pages/data-archive/data-archive-list-page.js

@@ -0,0 +1,290 @@
+import '@operato/data-grist'
+
+import gql from 'graphql-tag'
+import { css, html } from 'lit'
+import { connect } from 'pwa-helpers/connect-mixin'
+
+import { client } from '@operato/graphql'
+import { i18next, localize } from '@operato/i18n'
+import { openPopup } from '@operato/layout'
+import { PageView, store } from '@operato/shell'
+import { CommonGristStyles, ScrollbarStyles } from '@operato/styles'
+import { isMobileDevice } from '@operato/utils'
+
+import './data-archive-request-popup'
+
+export class DataArchiveListPage extends connect(store)(localize(i18next)(PageView)) {
+  static get properties() {
+    return {
+      active: String,
+      gristConfig: Object,
+      mode: String
+    }
+  }
+
+  static get styles() {
+    return [
+      ScrollbarStyles,
+      CommonGristStyles,
+      css`
+        :host {
+          display: flex;
+          flex-direction: column;
+
+          overflow: hidden;
+        }
+
+        ox-grist {
+          overflow-y: auto;
+          flex: 1;
+        }
+      `
+    ]
+  }
+
+  get context() {
+    return {
+      title: i18next.t('title.data-archive list'),
+      help: 'dataset/data-archive',
+      actions: [],
+      exportable: {
+        name: i18next.t('title.data-archive list'),
+        data: this._exportableData.bind(this)
+      }
+    }
+  }
+
+  render() {
+    const mode = this.mode || (isMobileDevice() ? 'LIST' : 'GRID')
+
+    return html`
+      <ox-grist
+        .mode=${mode}
+        .config=${this.gristConfig}
+        .fetchHandler=${this.fetchHandler.bind(this)}
+        ?url-params-sensitive=${false /* this.active */}
+      >
+        <div slot="headroom">
+          <div id="filters">
+            <ox-filters-form autofocus></ox-filters-form>
+          </div>
+
+          <div id="sorters">
+            Sort
+            <mwc-icon
+              @click=${e => {
+                const target = e.currentTarget
+                this.renderRoot.querySelector('#sorter-control').open({
+                  right: 0,
+                  top: target.offsetTop + target.offsetHeight
+                })
+              }}
+              >expand_more</mwc-icon
+            >
+            <ox-popup id="sorter-control">
+              <ox-sorters-control> </ox-sorters-control>
+            </ox-popup>
+          </div>
+
+          <div id="modes">
+            <mwc-icon @click=${() => (this.mode = 'GRID')} ?active=${mode == 'GRID'}>grid_on</mwc-icon>
+            <mwc-icon @click=${() => (this.mode = 'LIST')} ?active=${mode == 'LIST'}>format_list_bulleted</mwc-icon>
+            <mwc-icon @click=${() => (this.mode = 'CARD')} ?active=${mode == 'CARD'}>apps</mwc-icon>
+          </div>
+
+          <mwc-button
+            dense
+            raised
+            label=${i18next.t('button.request')}
+            @click=${this.openArchivePopup.bind(this)}
+            icon="archive"
+          ></mwc-button>
+        </div>
+      </ox-grist>
+    `
+  }
+
+  get grist() {
+    return this.renderRoot.querySelector('ox-grist')
+  }
+
+  async pageInitialized(lifecycle) {
+    const today = new Date().toISOString().split('T')[0]
+
+    this.gristConfig = {
+      list: { fields: ['updater', 'updatedAt'] },
+      columns: [
+        { type: 'gutter', gutterName: 'sequence' },
+        // { type: 'gutter', gutterName: 'row-selector', multiple: true },
+        {
+          type: 'resource-object',
+          name: 'creator',
+          header: i18next.t('field.creator'),
+          sortable: true,
+          width: 120,
+          imex: true
+        },
+        {
+          type: 'datetime',
+          name: 'createdAt',
+          header: i18next.t('field.created_at'),
+          sortable: true,
+          width: 180,
+          imex: true
+        },
+        {
+          type: 'datetime',
+          name: 'updatedAt',
+          header: i18next.t('field.updated_at'),
+          sortable: true,
+          width: 180,
+          imex: true
+        },
+        {
+          type: 'json5',
+          name: 'requestParams',
+          header: i18next.t('field.request-params'),
+          record: {
+            editable: false
+          },
+          width: 200,
+          imex: true
+        },
+        {
+          type: 'string',
+          name: 'downloadUrl',
+          label: true,
+          header: i18next.t('field.download-url'),
+          record: {
+            editable: false
+          },
+          width: 240,
+          imex: true
+        },
+        {
+          type: 'string',
+          name: 'status',
+          label: true,
+          header: i18next.t('field.status'),
+          record: {
+            editable: false
+          },
+          sortable: true,
+          width: 120,
+          imex: true
+        }
+      ],
+      rows: {
+        appendable: false,
+        selectable: {
+          multiple: true
+        }
+      },
+      sorters: [
+        {
+          name: 'createdAt',
+          desc: true
+        }
+      ]
+    }
+  }
+
+  closePopupAndRefesh() {
+    this.popup && this.popup.close()
+    this.grist.fetch()
+  }
+
+  async openArchivePopup() {
+    this.popup = openPopup(
+      html`
+      <data-archive-request-popup
+        @requested="${this.closePopupAndRefesh.bind(this)}"
+        @created="${this.closePopupAndRefesh.bind(this)}"
+      ></data-archive-request-popup>`, 
+      {
+        backdrop: true,
+        size: 'small',
+        title: i18next.t('title.data-archive request popup')
+      })
+  }
+
+  async fetchHandler({ page, limit, sortings = [], filters = [] }) {
+    const response = await client.query({
+      query: gql`
+        query ($filters: [Filter!], $pagination: Pagination, $sortings: [Sorting!]) {
+          responses: dataArchives(filters: $filters, pagination: $pagination, sortings: $sortings) {
+            items {
+              id
+              type
+              requestParams
+              downloadUrl
+              status
+              creator {
+                id
+                name
+              }
+              updater {
+                id
+                name
+              }
+              updatedAt
+              createdAt
+            }
+            total
+          }
+        }
+      `,
+      variables: {
+        filters,
+        pagination: { page, limit },
+        sortings
+      }
+    })
+
+    return {
+      total: response.data.responses.total || 0,
+      records: response.data.responses.items || []
+    }
+  }
+
+  _exportableData() {
+    let records = []
+    if (this.grist.selected && this.grist.selected.length > 0) {
+      records = this.grist.selected
+    } else {
+      records = this.grist.data.records
+    }
+
+    var headerSetting = this.grist.compiledConfig.columns
+      .filter(column => column.type !== 'gutter' && column.record !== undefined && column.imex !== undefined)
+      .map(column => {
+        return column.imex === true
+          ? {
+              header: column.header.renderer(),
+              key: column.name,
+              width: column.width,
+              type: column.type
+            }
+          : column.imex
+      })
+
+    var data = records.map(item => {
+      return {
+        id: item.id,
+        ...this.gristConfig.columns
+          .filter(column => column.type !== 'gutter' && column.record !== undefined && column.imex !== undefined)
+          .reduce((record, column) => {
+            const key = column.imex === true ? column.name : column.imex.key
+            record[key] = key
+              .split('.')
+              .reduce((obj, key) => (obj && obj[key] !== 'undefined' ? obj[key] : undefined), item)
+            return record
+          }, {})
+      }
+    })
+
+    return { header: headerSetting, data: data }
+  }
+}
+
+window.customElements.define('data-archive-list-page', DataArchiveListPage)

package/client/pages/data-archive/data-archive-request-popup.js

@@ -0,0 +1,180 @@
+import { SingleColumnFormStyles } from '@things-factory/form-ui'
+import '@things-factory/grist-ui'
+import { i18next, localize } from '@things-factory/i18n-base'
+import { client, CustomAlert } from '@things-factory/shell'
+import gql from 'graphql-tag'
+import { css, html, LitElement } from 'lit-element'
+import moment from 'moment-timezone'
+
+class DataArchiveRequestPopup extends localize(i18next)(LitElement) {
+  static get properties() {
+    return {
+      dataSetTypes: Array
+    }
+  }
+
+  static get styles() {
+    return [
+      SingleColumnFormStyles,
+      css`
+        :host {
+          padding: 10px;
+          display: flex;
+          flex-direction: column;
+          overflow-x: overlay;
+          background-color: var(--main-section-background-color);
+        }
+        .button-container {
+          padding: var(--button-container-padding);
+          margin: var(--button-container-margin);
+          text-align: var(--button-container-align);
+          background-color: var(--button-container-background);
+          height: var(--button-container-height);
+        }
+      `
+    ]
+  }
+
+  render() {
+    return html`
+      <form id="input-form" name="generation" class="single-column-form" @submit=${e => this.onSubmit(e)}>
+        <fieldset>
+          <label>${i18next.t('label.start-date')}</label>
+          <input type="month" name="startDate" required/>
+          <label>${i18next.t('label.end-date')}</label>
+          <input type="month" name="endDate" .value=${moment().format('YYYY-MM')}/>
+          <label>${i18next.t('label.data-set-type')}</label>
+          <select name="type" required>
+            ${(this.dataSetTypes || []).map(
+              t => html`<option value="${t && t.value}">${t && t.display}</option>`
+            )}
+          </select> 
+        </fieldset>
+      </form>
+
+      <div class="button-container">
+        <mwc-button raised @click="${this.requestArchive}" label="${i18next.t('button.submit')}"></mwc-button>
+      </div>
+    `
+  }
+
+  async firstUpdated() {
+    
+  }
+
+  async onSubmit(e) {
+    e.preventDefault()
+    this.requestArchive()
+  }
+
+
+  serializeFormData() {
+    const obj = {}
+    
+    Array.from(this.shadowRoot.querySelectorAll('form#input-form input, select')).forEach(field => {
+      if (!field.hasAttribute('hidden') && field.value) {
+        obj[field.name] = field.type === 'checkbox' ? field.checked : field.value
+      }
+    })
+
+    if (obj['startDate']) {
+      obj['startDate'] += '-01'
+    }
+
+    if (obj['endDate']) {
+      const endDate = moment(obj['endDate']).endOf('month')
+      const today = moment()
+      const format = 'YYYY-MM-DD'
+
+      obj['endDate'] = endDate > today ? today.format(format) : endDate.format(format)
+    }
+
+    return obj
+  }
+
+  /** request download url. */
+  async requestArchive() {
+    // #1 request download url
+    const dataArchive = {
+      requestParams: this.serializeFormData(),
+      status: 'requested'
+    }
+
+    const response = await client.mutate({
+      mutation: gql`
+        mutation createDataArchive($dataArchive: NewDataArchive!) {
+          createDataArchive(dataArchive: $dataArchive) {
+            id
+            status
+          }
+        }
+      `,
+      variables: { dataArchive }
+    })
+
+    if (!response.errors) {
+      this.dispatchEvent(new CustomEvent('requested', {}))
+
+      const { createDataArchive: { id } } = response.data
+      dataArchive['id'] = id
+
+      this._generateArchiveAndDownloadUrl(dataArchive)
+
+      await CustomAlert({
+        type: 'info',
+        title: i18next.t('title.ready'),
+        text: i18next.t('text.data-archive waits'),
+        confirmButton: { text: i18next.t('button.confirm') }
+      })
+      
+    } else {
+      console.error(response.errors)
+      this.showToast(i18next.t('text.failed'))
+    }
+  }
+
+  async _generateArchiveAndDownloadUrl(dataArchive) {
+    const response = await client.mutate({
+      mutation: gql`
+        mutation generatePresignedUrl($patch: DataArchivePatch!) {
+          generatePresignedUrl(patch: $patch) {
+            id
+            downloadUrl
+            status
+          }
+        }
+      `,
+      variables: { patch: dataArchive }
+    })
+
+    if (!response.errors) {     
+      this.showToast(i18next.t('title.data-archive downloads ready'))
+      this.dispatchEvent(new CustomEvent('created', {}))
+    } else {
+      console.error(response.errors)
+      this.showToast(i18next.t('text.failed'))
+    }
+  }
+
+  showToast(message) {
+    document.dispatchEvent(new CustomEvent('notify', { detail: { message } }))
+  }
+
+  constructor() {
+    super()
+
+    this.dataSetTypes = [
+      {},
+      {
+        display: i18next.t('text.manually collected'),
+        value: 'manual'
+      },
+      {
+        display: i18next.t('text.automatically collected'),
+        value: 'automatic'
+      }
+    ]
+  }
+}
+
+window.customElements.define('data-archive-request-popup', DataArchiveRequestPopup)

package/client/pages/data-key-set/data-key-set-list-page.js

@@ -156,7 +156,7 @@ export class DataKeySetListPage extends connect(store)(localize(i18next)(PageVie
               if (!record.id) return
               const popup = openPopup(html` <data-key-item-list .dataKeySet=${record}></data-key-item-list> `, {
                 backdrop: true,
-                help: 'data-set/ui/data-key-item-list',
+                help: 'dataset/ui/data-key-item-list',
                 size: 'large',
                 title: i18next.t('title.data-key-item list')
               })

package/client/pages/data-report/data-report-list-page.js

@@ -137,8 +137,8 @@ export class DataReportListPage extends connect(store)(localize(i18next)(PageVie
 
   get context() {
     return {
-      title: i18next.t('title.data-entry list'),
-      help: 'dataset/data-entry-list'
+      title: i18next.t('title.data-report list'),
+      help: 'dataset/data-report-list'
     }
   }
 

package/client/pages/data-set/data-set-list-page.js

@@ -225,7 +225,7 @@ export class DataSetListPage extends connect(store)(localize(i18next)(PageView))
               if (!record.id) return
               const popup = openPopup(html` <data-item-list .dataSet=${record}></data-item-list> `, {
                 backdrop: true,
-                help: 'data-set/ui/data-item-list',
+                help: 'dataset/ui/data-item-list',
                 size: 'large',
                 title: i18next.t('title.data-item list')
               })

package/client/route.js

@@ -31,6 +31,10 @@ export default function route(page) {
     case 'data-report-list':
       import('./pages/data-report/data-report-list-page.js')
       return page
+  
+    case 'data-archive-list':
+      import('./pages/data-archive/data-archive-list-page.js')
+      return page
 
     // case 'jasper-report-samples':
     //   import('./pages/data-report/jasper-report-samples-page.js')

package/config/config.development.js

@@ -1,4 +1,15 @@
 module.exports = {
+    dataArchive: {
+        dataset: {
+            endpoint: {
+                host: 'localhost',
+                port: 80
+            },
+            datasource: {
+                database: ''
+            }
+        }
+    },
     dataReport: {
         jasper: {
             endpoint: {

package/config/config.production.js

@@ -1,4 +1,15 @@
 module.exports = {
+    dataArchive: {
+        dataset: {
+            endpoint: {
+                host: 'localhost',
+                port: 80
+            },
+            datasource: {
+                database: ''
+            }
+        }
+    },
     dataReport: {
         jasper: {
             endpoint: {

package/dist-server/controllers/jasper-report.js

@@ -4,11 +4,12 @@ exports.renderJasperReport = void 0;
 const tslib_1 = require("tslib");
 const form_data_1 = tslib_1.__importDefault(require("form-data"));
 const node_fetch_1 = tslib_1.__importDefault(require("node-fetch"));
+const config_resolver_1 = require("../utils/config-resolver");
 const attachment_base_1 = require("@things-factory/attachment-base");
 const aws_base_1 = require("@things-factory/aws-base");
 const env_1 = require("@things-factory/env");
 const dataReportConfig = env_1.config.get('dataReport');
-const { jasper: { endpoint: { protocol: PROTOCOL, host: HOST, port: PORT }, datasource: { database: DATABASE } } } = dataReportConfig || {
+const { jasper: { endpoint: ENDPOINT, datasource: { database: DATABASE } } } = dataReportConfig || {
     jasper: {
         endpoint: {},
         datasource: {}
@@ -151,9 +152,8 @@ async function renderJasperReport(context) {
         formData.append('jsonString', JSON.stringify(parsedData));
         formData.append('parameters', JSON.stringify(parameters));
         const { reportView } = query;
-        const subpath = reportView ? reportView.split('/').filter(x => x).join('/') : '';
-        const reportUrl = `${PROTOCOL || 'http'}://${HOST}:${PORT}/${subpath}`;
-        const response = await (0, node_fetch_1.default)(reportUrl, {
+        const url = (0, config_resolver_1.getEndpointUrl)(ENDPOINT, reportView);
+        const response = await (0, node_fetch_1.default)(url, {
             method: 'POST',
             body: formData
         });