@coopenomics/parser 2.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Anthony Fu <https://github.com/antfu>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,146 @@
+# COOPARSER.
+
+[![npm version][npm-version-src]][npm-version-href]
+[![npm downloads][npm-downloads-src]][npm-downloads-href]
+[![bundle][bundle-src]][bundle-href]
+[![JSDocs][jsdocs-src]][jsdocs-href]
+[![License][license-src]][license-href]
+
+Пакет производит распаковку блоков, сохраняя действия и дельты таблиц и выдавая их по API. Состоит из двух модулей: парсера и API. Парсер считывает данные из блокчейна и помещает их в базу. API получает данные по запросу и возвращает их с пагинацией.
+
+## Установка
+```
+pnpm install
+```
+
+### Конфигурационный файл .env
+```
+NODE_ENV=development
+- Определяет среду выполнения приложения.
+
+API=http://127.0.0.1:8888
+- Определяет URL-адрес API, к которому будет осуществляться доступ.
+
+SHIP=ws://127.0.0.1:8080
+- Определяет URL-адрес WebSocket-соединения, используемого для связи с другими узлами.
+
+MONGO_EXPLORER_URI=mongodb://127.0.0.1:27017/cooperative
+- Определяет URI-адрес MongoDB, используемый для подключения к базе данных.
+
+START_BLOCK=1
+- Определяет номер блока, с которого начинается парсинг блокчейна.
+
+FINISH_BLOCK=0xFFFFFFFF
+- Определяет номер блока, на котором заканчивается парсинг блокчейна. В данном случае, установлено значение "0xFFFFFFFF", что означает, что парсинг будет продолжаться до последнего доступного блока.
+
+PORT=4000
+- Определяет порт, на котором будет запущен сервер приложения.
+
+ACTIVATE_PARSER=0
+- Определяет флаг активации парсера. Если значение равно "1", парсер будет активирован.
+```
+
+### Конфигурация парсера
+В конфиге src/config.ts находится массив таблиц и действий, на которые парсер осуществит подписку.
+
+```
+export const subsribedTables = [
+  { code: 'registrator', table: 'users', 'scope': 'registrator' },
+  { code: 'soviet', table: 'participants' },
+]
+```
+- подписка будет осуществлена на изменения таблиц указанных контрактов. Параметр scope - не обязательный. Без его указания любые scope будут попадать в базу данных.
+
+```
+export const subsribedActions = [
+  { code: 'soviet', action: 'votefor' },
+  { code: 'soviet', action: 'voteagainst' },
+]
+```
+- подписка будет осуществлена на действия указанных контрактов.
+
+Парсер может быть расширен любыми кастомными действиями, которые будут выполняться перед добавлением записи в базу данных. Для этого, для таблиц и действий соответственно, в папках src/ActionParser/Actions и src/DeltaParser/Deltas необходимо создать файлы с методами обработки и добавить их к src/ActionParser/Actions или src/DeltaParser/DeltaFactory.
+
+### Запуск
+```
+pnpm start
+```
+
+## API
+
+### Получение таблиц
+Конечная точка предоставляет информацию о изменении (дельтах) таблиц между блоками.
+
+```
+
+let params = {
+  page: 1,
+  limit: 10,
+  filter: {  } - любые параметры фильтрации таблицы, включая данные в полях
+};
+
+axios.get('http://localhost:4000/get-tables', { params })
+  .then(response => {
+    console.log(response.data);
+    //  {
+    //    results: array,
+    //    page: number,
+    //    limit: number
+    //  };
+  })
+```
+
+### Получение действий
+Конечная точка предоставляет информацию о действиях, произошедших между блоками.
+
+```
+let params = {
+  page: 1,
+  limit: 10,
+  filter: {  } // любые параметры фильтрации действий, включая данные в полях
+};
+
+axios.get('http://localhost:4000/get-actions', { params })
+  .then(response => {
+    console.log(response.data);
+    //  {
+    //    results: array,
+    //    page: number,
+    //    limit: number
+    //  };
+  })
+  .catch(error => {
+    console.error(error);
+  });
+```
+
+### Получение текущего блока
+Конечная точка предоставляет информацию о текущем блоке. Эта информация используется при формировании кооперативных документов.
+
+```
+axios.get('http://localhost:4000/get-current-block')
+  .then(response => {
+    console.log(response.data);
+    // number
+  })
+  .catch(error => {
+    console.error(error);
+  });
+```
+
+## Лицензия
+
+[MIT](./LICENSE) License © 2024-PRESENT [CBS VOSKHOD](https://github.com/copenomics)
+
+<!-- Badges -->
+
+[npm-version-src]: https://img.shields.io/npm/v/cooparser?style=flat&colorA=080f12&colorB=1fa669
+[npm-version-href]: https://npmjs.com/package/cooparser
+[npm-downloads-src]: https://img.shields.io/npm/dm/cooparser?style=flat&colorA=080f12&colorB=1fa669
+[npm-downloads-href]: https://npmjs.com/package/cooparser
+[bundle-src]: https://img.shields.io/bundlephobia/minzip/cooparser?style=flat&colorA=080f12&colorB=1fa669&label=minzip
+[bundle-href]: https://bundlephobia.com/result?p=cooparser
+[license-src]: https://img.shields.io/github/license/copenomics/cooparser.svg?style=flat&colorA=080f12&colorB=1fa669
+[license-href]: https://github.com/copenomics/cooparser/blob/main/LICENSE
+[jsdocs-src]: https://img.shields.io/badge/jsdocs-reference-080f12?style=flat&colorA=080f12&colorB=1fa669
+[jsdocs-href]: https://www.jsdocs.io/package/cooparser

package/dist/index.cjs

@@ -0,0 +1,386 @@
+'use strict';
+
+const express = require('express');
+require('express-async-errors');
+const mongodb = require('mongodb');
+const dotenv = require('dotenv');
+const eosioShipReader = require('@blockmatic/eosio-ship-reader');
+const fetch = require('node-fetch');
+const Redis = require('ioredis');
+
+function _interopDefaultCompat (e) { return e && typeof e === 'object' && 'default' in e ? e.default : e; }
+
+const express__default = /*#__PURE__*/_interopDefaultCompat(express);
+const dotenv__default = /*#__PURE__*/_interopDefaultCompat(dotenv);
+const fetch__default = /*#__PURE__*/_interopDefaultCompat(fetch);
+const Redis__default = /*#__PURE__*/_interopDefaultCompat(Redis);
+
+dotenv__default.config();
+function getEnvVar(key) {
+  const envVar = process.env[key];
+  if (envVar === void 0)
+    throw new Error(`Env variable ${key} is required`);
+  return envVar;
+}
+const node_env = getEnvVar("NODE_ENV");
+const eosioApi = getEnvVar("API");
+const shipApi = getEnvVar("SHIP");
+const mongoUri = `${getEnvVar("MONGO_EXPLORER_URI")}${node_env === "test" ? "-test" : ""}`;
+const startBlock = getEnvVar("START_BLOCK");
+const finishBlock = getEnvVar("FINISH_BLOCK");
+const redisPort = getEnvVar("REDIS_PORT");
+const redisHost = getEnvVar("REDIS_HOST");
+const redisPassword = getEnvVar("REDIS_PASSWORD");
+const redisStreamLimit = Number(getEnvVar("REDIS_STREAM_LIMIT"));
+const subsribedTables = [
+  // документы
+  { code: "draft", table: "drafts" },
+  { code: "draft", table: "translations" },
+  // совет
+  { code: "soviet", table: "decisions" },
+  { code: "soviet", table: "boards" },
+  { code: "soviet", table: "participants" },
+  // registrator.joincoop
+  { code: "soviet", table: "joincoops" },
+  // регистратор
+  { code: "registrator", table: "accounts" },
+  { code: "registrator", table: "orgs" }
+];
+const subsribedActions = [
+  { code: "eosio.token", action: "transfer", notify: true },
+  { code: "registrator", action: "confirmreg", notify: true },
+  { code: "soviet", action: "votefor" },
+  { code: "soviet", action: "voteagainst" },
+  { code: "soviet", action: "newsubmitted" },
+  { code: "soviet", action: "newresolved" },
+  { code: "soviet", action: "newdecision" },
+  // // registrator.joincoop
+  { code: "soviet", action: "joincoop" },
+  { code: "soviet", action: "joincoopdec" },
+  { code: "soviet", action: "updateboard", notify: true },
+  { code: "soviet", action: "createboard", notify: true }
+];
+
+var __defProp = Object.defineProperty;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __publicField = (obj, key, value) => {
+  __defNormalProp(obj, typeof key !== "symbol" ? key + "" : key, value);
+  return value;
+};
+class Database {
+  constructor() {
+    __publicField(this, "client");
+    __publicField(this, "db");
+    __publicField(this, "actions");
+    __publicField(this, "deltas");
+    __publicField(this, "sync");
+    this.client = new mongodb.MongoClient(mongoUri);
+  }
+  async connect() {
+    await this.client.connect();
+    this.db = this.client.db();
+    this.actions = this.db.collection("actions");
+    this.deltas = this.db.collection("deltas");
+    this.sync = this.db.collection("sync");
+  }
+  async saveActionToDB(action) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.actions.insertOne(action);
+  }
+  async saveDeltaToDB(delta) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.deltas.insertOne(delta);
+  }
+  async getDelta(filter) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const result = await this.deltas.findOne(filter);
+    return result;
+  }
+  async getTables(filter, page = 1, limit = 10) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const pipeline = [
+      { $match: filter },
+      { $sort: { block_num: -1 } },
+      // Сортировка по primary_key и block_num
+      { $group: { _id: "$primary_key", doc: { $first: "$$ROOT" } } },
+      { $replaceRoot: { newRoot: "$doc" } },
+      { $sort: { block_num: -1 } },
+      { $skip: (page - 1) * limit },
+      // Применяется внутри пайплайна
+      { $limit: limit }
+      // Применяется внутри пайплайна
+    ];
+    const result = await this.deltas.aggregate(pipeline).toArray();
+    return {
+      results: result,
+      page,
+      limit
+    };
+  }
+  async getActions(filter, page = 1, limit = 10) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const query = filter || {};
+    const result = await this.actions.aggregate([
+      { $match: query },
+      { $sort: { block_num: -1 } },
+      // Сортировка по primary_key и block_num
+      { $group: { _id: "$global_sequence", doc: { $first: "$$ROOT" } } },
+      { $replaceRoot: { newRoot: "$doc" } },
+      { $sort: { block_num: -1 } },
+      { $skip: (page - 1) * limit },
+      { $limit: limit }
+    ]).toArray();
+    return {
+      results: result,
+      page,
+      limit
+    };
+  }
+  async getAction(filter) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const result = await this.actions.findOne(filter);
+    return result ? result.value : null;
+  }
+  async getCurrentBlock() {
+    if (!this.sync)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const currentBlockDocument = await this.sync.findOne({ key: "currentBlock" });
+    return currentBlockDocument ? currentBlockDocument.block_num : 0;
+  }
+  async updateCurrentBlock(block_num) {
+    if (!this.sync)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.sync.updateOne({ key: "currentBlock" }, { $set: { block_num } }, { upsert: true });
+  }
+  async purgeAfterBlock(since_block) {
+    if (!this.actions || !this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.actions.deleteMany({ block_num: { $gt: since_block } });
+    await this.deltas.deleteMany({ block_num: { $gt: since_block } });
+  }
+}
+const db = new Database();
+async function init() {
+  return db.connect().then(() => {
+    console.log("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u0438\u043D\u0438\u0446\u0438\u0430\u043B\u0438\u0437\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+  });
+}
+
+const redis = new Redis__default({
+  port: Number(redisPort),
+  host: redisHost,
+  password: redisPassword
+  // другие опции при необходимости
+});
+const streamName = "notifications";
+async function publishEvent(type, event) {
+  const message = JSON.stringify({ type, event });
+  await redis.xadd(streamName, "*", "event", message);
+  await redis.xtrim(streamName, "MAXLEN", "~", redisStreamLimit);
+}
+
+class AnyAnyActionParser {
+  async process(db, action) {
+    db.saveActionToDB(action);
+  }
+}
+
+class ActionParserFactory {
+  static create(accountName, actionName) {
+    switch (`${accountName}::${actionName}`) {
+      case "*::*":
+        return null;
+      default:
+        return new AnyAnyActionParser();
+    }
+  }
+}
+
+async function ActionsParser(db, reader) {
+  const { actions$ } = reader;
+  actions$.subscribe(async (action) => {
+    console.log(`
+ACTION - account: ${action.account}, name: ${action.name}, authorization: ${JSON.stringify(action.authorization)}, data: ${JSON.stringify(action.data)}`);
+    const parser = ActionParserFactory.create(action.account, action.name);
+    const source = subsribedActions.find((el) => el.action === action.name && el.code === action.account);
+    if (parser) {
+      await parser.process(db, action);
+      if (source?.notify)
+        await publishEvent("action", action);
+    }
+  });
+  console.log("\u041F\u043E\u0434\u043F\u0438\u0441\u043A\u0430 \u043D\u0430 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044F \u0430\u043A\u0442\u0438\u0432\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+}
+
+async function BlockParser(db, reader) {
+  const { blocks$, errors$, close$ } = reader;
+  errors$.subscribe(async (error) => {
+    console.log("\n\u041E\u0448\u0438\u0431\u043A\u0430 \u0434\u0435\u0441\u0435\u0440\u0438\u0430\u043B\u0438\u0437\u0430\u0446\u0438\u0438: ", error);
+  });
+  close$.subscribe(async (error) => {
+    console.error("\n\u041E\u0448\u0438\u0431\u043A\u0430 \u0441\u043E\u0435\u0434\u0438\u043D\u0435\u043D\u0438\u044F: ", error);
+    console.log("\u0412\u044B\u043A\u043B\u044E\u0447\u0435\u043D\u0438\u0435 \u0447\u0435\u0440\u0435\u0437 10 \u0441\u0435\u043A\u0443\u043D\u0434");
+    setTimeout(() => process.exit(1), 1e4);
+  });
+  blocks$.subscribe(async (block) => {
+    db.updateCurrentBlock(block.block_num);
+  });
+}
+
+class DeltaParser {
+  async process(db, delta) {
+    db.saveDeltaToDB(delta);
+  }
+}
+
+class DeltaParserFactory {
+  static create(code, scope, table) {
+    switch (`${code}::${scope}::${table}`) {
+      default:
+        return new DeltaParser();
+    }
+  }
+}
+
+async function DeltasParser(db, reader) {
+  const { rows$ } = reader;
+  rows$.subscribe(async (delta) => {
+    console.log(`
+DELTA - code: ${delta.code}, scope: ${delta.scope}, table: ${delta.table}, primary_key: ${delta.primary_key}, data: ${JSON.stringify(delta.value)}`);
+    const source = subsribedTables.find((el) => el.code === delta.code && el.table === delta.table);
+    const parser = DeltaParserFactory.create(delta.code, delta.scope, delta.table);
+    if (parser) {
+      await parser.process(db, delta);
+      if (source?.notify)
+        await publishEvent("delta", delta);
+    }
+  });
+  console.log("\u041F\u043E\u0434\u043F\u0438\u0441\u043A\u0430 \u043D\u0430 \u0434\u0435\u043B\u044C\u0442\u044B \u0430\u043A\u0442\u0438\u0432\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+}
+
+const getInfo = () => fetch__default(`${eosioApi}/v1/chain/get_info`).then((res) => res.json());
+function fetchAbi(account_name) {
+  return fetch__default(`${eosioApi}/v1/chain/get_abi`, {
+    method: "POST",
+    body: JSON.stringify({
+      account_name
+    })
+  }).then(async (res) => {
+    const response = await res.json();
+    return {
+      account_name,
+      abi: response.abi
+    };
+  });
+}
+
+const table_rows_whitelist = () => subsribedTables;
+const actions_whitelist = () => subsribedActions;
+console.log(subsribedTables);
+console.log(subsribedActions);
+async function loadReader(db) {
+  let currentBlock = await db.getCurrentBlock();
+  const info = await getInfo();
+  if (currentBlock === 0)
+    currentBlock = Number(startBlock);
+  console.log("\u0421\u0442\u0430\u0440\u0442\u0443\u0435\u043C \u0441 \u0431\u043B\u043E\u043A\u0430: ", currentBlock);
+  console.log("\u0417\u0430\u0432\u0435\u0440\u0448\u0438\u043C \u043D\u0430 \u0431\u043B\u043E\u043A\u0435: ", finishBlock);
+  console.log("\u0412\u044B\u0441\u043E\u0442\u0430 \u0446\u0435\u043F\u043E\u0447\u043A\u0438: ", info.head_block_num);
+  console.log("\u041E\u0447\u0438\u0449\u0430\u0435\u043C \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044F \u0438 \u0434\u0435\u043B\u044C\u0442\u044B \u043F\u043E\u0441\u043B\u0435 \u0431\u043B\u043E\u043A\u0430: ", currentBlock);
+  await db.purgeAfterBlock(currentBlock);
+  const unique_contract_names = [...new Set(table_rows_whitelist().map((row) => row.code)), ...new Set(actions_whitelist().map((row) => row.code))];
+  const abisArr = await Promise.all(unique_contract_names.map((account_name) => fetchAbi(account_name)));
+  const contract_abis = () => {
+    const numap = /* @__PURE__ */ new Map();
+    abisArr.forEach(({ account_name, abi }) => numap.set(account_name, abi));
+    return numap;
+  };
+  const delta_whitelist = () => [
+    "account_metadata",
+    "contract_table",
+    "contract_row",
+    "contract_index64",
+    "resource_usage",
+    "resource_limits_state"
+  ];
+  const eosioReaderConfig = {
+    ws_url: shipApi,
+    rpc_url: eosioApi,
+    ds_threads: 2,
+    ds_experimental: false,
+    delta_whitelist,
+    table_rows_whitelist,
+    actions_whitelist,
+    contract_abis,
+    request: {
+      start_block_num: currentBlock,
+      end_block_num: Number(finishBlock),
+      // info.head_block_num,
+      max_messages_in_flight: 50,
+      have_positions: [],
+      irreversible_only: true,
+      fetch_block: true,
+      fetch_traces: true,
+      fetch_deltas: true
+    },
+    auto_start: true
+  };
+  return await eosioShipReader.createEosioShipReader(eosioReaderConfig);
+}
+
+class Parser {
+  async start() {
+    const reader = await loadReader(db);
+    try {
+      BlockParser(db, reader);
+      ActionsParser(db, reader);
+      DeltasParser(db, reader);
+    } catch (e) {
+      console.error("\u041E\u0448\u0438\u0431\u043A\u0430: ", e);
+      this.start();
+    }
+  }
+}
+
+const app = express__default();
+app.use(express__default.json());
+app.use(express__default.urlencoded({ extended: true }));
+const port = process.env.PORT || 4e3;
+const parser = new Parser();
+init().then(() => {
+  app.listen(port, () => {
+    console.log(`API \u043E\u0431\u043E\u0437\u0440\u0435\u0432\u0430\u0442\u0435\u043B\u044F \u0437\u0430\u043F\u0443\u0449\u0435\u043D\u043E \u043D\u0430 http://localhost:${port}`);
+    if (process.env.ACTIVATE_PARSER === "1")
+      parser.start();
+  });
+});
+app.get("/get-tables", async (req, res) => {
+  const page = Number(req.query.page) || 1;
+  const limit = Number(req.query.limit) || 10;
+  const filter = req.query.filter ? JSON.parse(req.query.filter) : {};
+  const result = await db.getTables(filter, page, limit);
+  res.json(result);
+});
+app.get("/get-actions", async (req, res) => {
+  const page = Number(req.query.page) || 1;
+  const limit = Number(req.query.limit) || 10;
+  const filter = req.query.filter ? JSON.parse(req.query.filter) : {};
+  const result = await db.getActions(filter, page, limit);
+  res.json(result);
+});
+app.get("/get-current-block", async (req, res) => {
+  const result = await db.getCurrentBlock();
+  res.json(result);
+});
+app.use((err, req, res, _next) => {
+  console.error("\u0433\u043B\u043E\u0431\u0430\u043B\u044C\u043D\u0430\u044F \u043E\u0448\u0438\u0431\u043A\u0430: ", err);
+  res.status(500).send(err.message);
+});
+
+exports.parser = parser;

package/dist/index.d.cts

@@ -0,0 +1,7 @@
+declare class Parser {
+    start(): Promise<void>;
+}
+
+declare const parser: Parser;
+
+export { parser };

package/dist/index.d.mts

@@ -0,0 +1,7 @@
+declare class Parser {
+    start(): Promise<void>;
+}
+
+declare const parser: Parser;
+
+export { parser };

package/dist/index.d.ts

@@ -0,0 +1,7 @@
+declare class Parser {
+    start(): Promise<void>;
+}
+
+declare const parser: Parser;
+
+export { parser };

package/dist/index.mjs

@@ -0,0 +1,377 @@
+import express from 'express';
+import 'express-async-errors';
+import { MongoClient } from 'mongodb';
+import dotenv from 'dotenv';
+import { createEosioShipReader } from '@blockmatic/eosio-ship-reader';
+import fetch from 'node-fetch';
+import Redis from 'ioredis';
+
+dotenv.config();
+function getEnvVar(key) {
+  const envVar = process.env[key];
+  if (envVar === void 0)
+    throw new Error(`Env variable ${key} is required`);
+  return envVar;
+}
+const node_env = getEnvVar("NODE_ENV");
+const eosioApi = getEnvVar("API");
+const shipApi = getEnvVar("SHIP");
+const mongoUri = `${getEnvVar("MONGO_EXPLORER_URI")}${node_env === "test" ? "-test" : ""}`;
+const startBlock = getEnvVar("START_BLOCK");
+const finishBlock = getEnvVar("FINISH_BLOCK");
+const redisPort = getEnvVar("REDIS_PORT");
+const redisHost = getEnvVar("REDIS_HOST");
+const redisPassword = getEnvVar("REDIS_PASSWORD");
+const redisStreamLimit = Number(getEnvVar("REDIS_STREAM_LIMIT"));
+const subsribedTables = [
+  // документы
+  { code: "draft", table: "drafts" },
+  { code: "draft", table: "translations" },
+  // совет
+  { code: "soviet", table: "decisions" },
+  { code: "soviet", table: "boards" },
+  { code: "soviet", table: "participants" },
+  // registrator.joincoop
+  { code: "soviet", table: "joincoops" },
+  // регистратор
+  { code: "registrator", table: "accounts" },
+  { code: "registrator", table: "orgs" }
+];
+const subsribedActions = [
+  { code: "eosio.token", action: "transfer", notify: true },
+  { code: "registrator", action: "confirmreg", notify: true },
+  { code: "soviet", action: "votefor" },
+  { code: "soviet", action: "voteagainst" },
+  { code: "soviet", action: "newsubmitted" },
+  { code: "soviet", action: "newresolved" },
+  { code: "soviet", action: "newdecision" },
+  // // registrator.joincoop
+  { code: "soviet", action: "joincoop" },
+  { code: "soviet", action: "joincoopdec" },
+  { code: "soviet", action: "updateboard", notify: true },
+  { code: "soviet", action: "createboard", notify: true }
+];
+
+var __defProp = Object.defineProperty;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __publicField = (obj, key, value) => {
+  __defNormalProp(obj, typeof key !== "symbol" ? key + "" : key, value);
+  return value;
+};
+class Database {
+  constructor() {
+    __publicField(this, "client");
+    __publicField(this, "db");
+    __publicField(this, "actions");
+    __publicField(this, "deltas");
+    __publicField(this, "sync");
+    this.client = new MongoClient(mongoUri);
+  }
+  async connect() {
+    await this.client.connect();
+    this.db = this.client.db();
+    this.actions = this.db.collection("actions");
+    this.deltas = this.db.collection("deltas");
+    this.sync = this.db.collection("sync");
+  }
+  async saveActionToDB(action) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.actions.insertOne(action);
+  }
+  async saveDeltaToDB(delta) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.deltas.insertOne(delta);
+  }
+  async getDelta(filter) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const result = await this.deltas.findOne(filter);
+    return result;
+  }
+  async getTables(filter, page = 1, limit = 10) {
+    if (!this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const pipeline = [
+      { $match: filter },
+      { $sort: { block_num: -1 } },
+      // Сортировка по primary_key и block_num
+      { $group: { _id: "$primary_key", doc: { $first: "$$ROOT" } } },
+      { $replaceRoot: { newRoot: "$doc" } },
+      { $sort: { block_num: -1 } },
+      { $skip: (page - 1) * limit },
+      // Применяется внутри пайплайна
+      { $limit: limit }
+      // Применяется внутри пайплайна
+    ];
+    const result = await this.deltas.aggregate(pipeline).toArray();
+    return {
+      results: result,
+      page,
+      limit
+    };
+  }
+  async getActions(filter, page = 1, limit = 10) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const query = filter || {};
+    const result = await this.actions.aggregate([
+      { $match: query },
+      { $sort: { block_num: -1 } },
+      // Сортировка по primary_key и block_num
+      { $group: { _id: "$global_sequence", doc: { $first: "$$ROOT" } } },
+      { $replaceRoot: { newRoot: "$doc" } },
+      { $sort: { block_num: -1 } },
+      { $skip: (page - 1) * limit },
+      { $limit: limit }
+    ]).toArray();
+    return {
+      results: result,
+      page,
+      limit
+    };
+  }
+  async getAction(filter) {
+    if (!this.actions)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const result = await this.actions.findOne(filter);
+    return result ? result.value : null;
+  }
+  async getCurrentBlock() {
+    if (!this.sync)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    const currentBlockDocument = await this.sync.findOne({ key: "currentBlock" });
+    return currentBlockDocument ? currentBlockDocument.block_num : 0;
+  }
+  async updateCurrentBlock(block_num) {
+    if (!this.sync)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.sync.updateOne({ key: "currentBlock" }, { $set: { block_num } }, { upsert: true });
+  }
+  async purgeAfterBlock(since_block) {
+    if (!this.actions || !this.deltas)
+      throw new Error("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u043D\u0435 \u043F\u043E\u0434\u043A\u043B\u044E\u0447\u0435\u043D\u0430");
+    await this.actions.deleteMany({ block_num: { $gt: since_block } });
+    await this.deltas.deleteMany({ block_num: { $gt: since_block } });
+  }
+}
+const db = new Database();
+async function init() {
+  return db.connect().then(() => {
+    console.log("\u0411\u0430\u0437\u0430 \u0434\u0430\u043D\u043D\u044B\u0445 \u0438\u043D\u0438\u0446\u0438\u0430\u043B\u0438\u0437\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+  });
+}
+
+const redis = new Redis({
+  port: Number(redisPort),
+  host: redisHost,
+  password: redisPassword
+  // другие опции при необходимости
+});
+const streamName = "notifications";
+async function publishEvent(type, event) {
+  const message = JSON.stringify({ type, event });
+  await redis.xadd(streamName, "*", "event", message);
+  await redis.xtrim(streamName, "MAXLEN", "~", redisStreamLimit);
+}
+
+class AnyAnyActionParser {
+  async process(db, action) {
+    db.saveActionToDB(action);
+  }
+}
+
+class ActionParserFactory {
+  static create(accountName, actionName) {
+    switch (`${accountName}::${actionName}`) {
+      case "*::*":
+        return null;
+      default:
+        return new AnyAnyActionParser();
+    }
+  }
+}
+
+async function ActionsParser(db, reader) {
+  const { actions$ } = reader;
+  actions$.subscribe(async (action) => {
+    console.log(`
+ACTION - account: ${action.account}, name: ${action.name}, authorization: ${JSON.stringify(action.authorization)}, data: ${JSON.stringify(action.data)}`);
+    const parser = ActionParserFactory.create(action.account, action.name);
+    const source = subsribedActions.find((el) => el.action === action.name && el.code === action.account);
+    if (parser) {
+      await parser.process(db, action);
+      if (source?.notify)
+        await publishEvent("action", action);
+    }
+  });
+  console.log("\u041F\u043E\u0434\u043F\u0438\u0441\u043A\u0430 \u043D\u0430 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044F \u0430\u043A\u0442\u0438\u0432\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+}
+
+async function BlockParser(db, reader) {
+  const { blocks$, errors$, close$ } = reader;
+  errors$.subscribe(async (error) => {
+    console.log("\n\u041E\u0448\u0438\u0431\u043A\u0430 \u0434\u0435\u0441\u0435\u0440\u0438\u0430\u043B\u0438\u0437\u0430\u0446\u0438\u0438: ", error);
+  });
+  close$.subscribe(async (error) => {
+    console.error("\n\u041E\u0448\u0438\u0431\u043A\u0430 \u0441\u043E\u0435\u0434\u0438\u043D\u0435\u043D\u0438\u044F: ", error);
+    console.log("\u0412\u044B\u043A\u043B\u044E\u0447\u0435\u043D\u0438\u0435 \u0447\u0435\u0440\u0435\u0437 10 \u0441\u0435\u043A\u0443\u043D\u0434");
+    setTimeout(() => process.exit(1), 1e4);
+  });
+  blocks$.subscribe(async (block) => {
+    db.updateCurrentBlock(block.block_num);
+  });
+}
+
+class DeltaParser {
+  async process(db, delta) {
+    db.saveDeltaToDB(delta);
+  }
+}
+
+class DeltaParserFactory {
+  static create(code, scope, table) {
+    switch (`${code}::${scope}::${table}`) {
+      default:
+        return new DeltaParser();
+    }
+  }
+}
+
+async function DeltasParser(db, reader) {
+  const { rows$ } = reader;
+  rows$.subscribe(async (delta) => {
+    console.log(`
+DELTA - code: ${delta.code}, scope: ${delta.scope}, table: ${delta.table}, primary_key: ${delta.primary_key}, data: ${JSON.stringify(delta.value)}`);
+    const source = subsribedTables.find((el) => el.code === delta.code && el.table === delta.table);
+    const parser = DeltaParserFactory.create(delta.code, delta.scope, delta.table);
+    if (parser) {
+      await parser.process(db, delta);
+      if (source?.notify)
+        await publishEvent("delta", delta);
+    }
+  });
+  console.log("\u041F\u043E\u0434\u043F\u0438\u0441\u043A\u0430 \u043D\u0430 \u0434\u0435\u043B\u044C\u0442\u044B \u0430\u043A\u0442\u0438\u0432\u0438\u0440\u043E\u0432\u0430\u043D\u0430");
+}
+
+const getInfo = () => fetch(`${eosioApi}/v1/chain/get_info`).then((res) => res.json());
+function fetchAbi(account_name) {
+  return fetch(`${eosioApi}/v1/chain/get_abi`, {
+    method: "POST",
+    body: JSON.stringify({
+      account_name
+    })
+  }).then(async (res) => {
+    const response = await res.json();
+    return {
+      account_name,
+      abi: response.abi
+    };
+  });
+}
+
+const table_rows_whitelist = () => subsribedTables;
+const actions_whitelist = () => subsribedActions;
+console.log(subsribedTables);
+console.log(subsribedActions);
+async function loadReader(db) {
+  let currentBlock = await db.getCurrentBlock();
+  const info = await getInfo();
+  if (currentBlock === 0)
+    currentBlock = Number(startBlock);
+  console.log("\u0421\u0442\u0430\u0440\u0442\u0443\u0435\u043C \u0441 \u0431\u043B\u043E\u043A\u0430: ", currentBlock);
+  console.log("\u0417\u0430\u0432\u0435\u0440\u0448\u0438\u043C \u043D\u0430 \u0431\u043B\u043E\u043A\u0435: ", finishBlock);
+  console.log("\u0412\u044B\u0441\u043E\u0442\u0430 \u0446\u0435\u043F\u043E\u0447\u043A\u0438: ", info.head_block_num);
+  console.log("\u041E\u0447\u0438\u0449\u0430\u0435\u043C \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044F \u0438 \u0434\u0435\u043B\u044C\u0442\u044B \u043F\u043E\u0441\u043B\u0435 \u0431\u043B\u043E\u043A\u0430: ", currentBlock);
+  await db.purgeAfterBlock(currentBlock);
+  const unique_contract_names = [...new Set(table_rows_whitelist().map((row) => row.code)), ...new Set(actions_whitelist().map((row) => row.code))];
+  const abisArr = await Promise.all(unique_contract_names.map((account_name) => fetchAbi(account_name)));
+  const contract_abis = () => {
+    const numap = /* @__PURE__ */ new Map();
+    abisArr.forEach(({ account_name, abi }) => numap.set(account_name, abi));
+    return numap;
+  };
+  const delta_whitelist = () => [
+    "account_metadata",
+    "contract_table",
+    "contract_row",
+    "contract_index64",
+    "resource_usage",
+    "resource_limits_state"
+  ];
+  const eosioReaderConfig = {
+    ws_url: shipApi,
+    rpc_url: eosioApi,
+    ds_threads: 2,
+    ds_experimental: false,
+    delta_whitelist,
+    table_rows_whitelist,
+    actions_whitelist,
+    contract_abis,
+    request: {
+      start_block_num: currentBlock,
+      end_block_num: Number(finishBlock),
+      // info.head_block_num,
+      max_messages_in_flight: 50,
+      have_positions: [],
+      irreversible_only: true,
+      fetch_block: true,
+      fetch_traces: true,
+      fetch_deltas: true
+    },
+    auto_start: true
+  };
+  return await createEosioShipReader(eosioReaderConfig);
+}
+
+class Parser {
+  async start() {
+    const reader = await loadReader(db);
+    try {
+      BlockParser(db, reader);
+      ActionsParser(db, reader);
+      DeltasParser(db, reader);
+    } catch (e) {
+      console.error("\u041E\u0448\u0438\u0431\u043A\u0430: ", e);
+      this.start();
+    }
+  }
+}
+
+const app = express();
+app.use(express.json());
+app.use(express.urlencoded({ extended: true }));
+const port = process.env.PORT || 4e3;
+const parser = new Parser();
+init().then(() => {
+  app.listen(port, () => {
+    console.log(`API \u043E\u0431\u043E\u0437\u0440\u0435\u0432\u0430\u0442\u0435\u043B\u044F \u0437\u0430\u043F\u0443\u0449\u0435\u043D\u043E \u043D\u0430 http://localhost:${port}`);
+    if (process.env.ACTIVATE_PARSER === "1")
+      parser.start();
+  });
+});
+app.get("/get-tables", async (req, res) => {
+  const page = Number(req.query.page) || 1;
+  const limit = Number(req.query.limit) || 10;
+  const filter = req.query.filter ? JSON.parse(req.query.filter) : {};
+  const result = await db.getTables(filter, page, limit);
+  res.json(result);
+});
+app.get("/get-actions", async (req, res) => {
+  const page = Number(req.query.page) || 1;
+  const limit = Number(req.query.limit) || 10;
+  const filter = req.query.filter ? JSON.parse(req.query.filter) : {};
+  const result = await db.getActions(filter, page, limit);
+  res.json(result);
+});
+app.get("/get-current-block", async (req, res) => {
+  const result = await db.getCurrentBlock();
+  res.json(result);
+});
+app.use((err, req, res, _next) => {
+  console.error("\u0433\u043B\u043E\u0431\u0430\u043B\u044C\u043D\u0430\u044F \u043E\u0448\u0438\u0431\u043A\u0430: ", err);
+  res.status(500).send(err.message);
+});
+
+export { parser };

package/package.json

@@ -0,0 +1,89 @@
+{
+  "name": "@coopenomics/parser",
+  "type": "module",
+  "version": "2.2.0",
+  "private": false,
+  "packageManager": "pnpm@9.0.6",
+  "description": "",
+  "author": "Alex Ant <dacom.dark.sun@gmail.com>",
+  "license": "MIT",
+  "homepage": "https://github.com/copenomics/cooparser#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/copenomics/cooparser.git"
+  },
+  "bugs": "https://github.com/copenomics/cooparser/issues",
+  "keywords": [],
+  "sideEffects": false,
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.mjs",
+      "require": "./dist/index.cjs"
+    }
+  },
+  "main": "./dist/index.mjs",
+  "module": "./dist/index.mjs",
+  "types": "./dist/index.d.ts",
+  "typesVersions": {
+    "*": {
+      "*": [
+        "./dist/*",
+        "./dist/index.d.ts"
+      ]
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "deploy-testnet": "git checkout testnet && git merge main && git push origin testnet && git checkout main",
+    "deploy-production": "git checkout production && git merge testnet && git push origin production && git checkout main",
+    "build": "unbuild",
+    "dev": "concurrently -n 'PARSER' -c 'bgBlue.white' \"nodemon --watch src --ext ts,js,env --exec 'esno' src/index.ts\"",
+    "dev:test": "NODE_ENV=test concurrently -n 'PARSER' -c 'bgBlue.white' \"nodemon --watch src --ext ts,js,env --exec 'esno' src/index.ts\"",
+    "lint": "eslint .",
+    "prepublishOnly": "nr build",
+    "release": "bumpp && npm publish",
+    "start": "esno src/index.ts",
+    "test": "vitest",
+    "typecheck": "tsc --noEmit",
+    "doc": "typedoc"
+  },
+  "dependencies": {
+    "@blockmatic/eosio-ship-reader": "^1.2.0",
+    "@types/express": "^4.17.21",
+    "@types/ws": "^8.5.13",
+    "dotenv": "^16.4.5",
+    "dotenv-expand": "^11.0.6",
+    "eosjs": "^22.1.0",
+    "express": "^4.19.2",
+    "express-async-errors": "^3.1.1",
+    "ioredis": "^5.4.1",
+    "mongodb": "^6.5.0",
+    "node-fetch": "^3.3.2",
+    "typedoc": "^0.25.13",
+    "typedoc-plugin-inline-sources": "^1.0.2",
+    "ws": "^8.18.0"
+  },
+  "devDependencies": {
+    "@antfu/eslint-config": "^2.16.0",
+    "@antfu/ni": "^0.21.12",
+    "@antfu/utils": "^0.7.7",
+    "@types/node": "^20.12.7",
+    "bumpp": "^9.4.0",
+    "concurrently": "^8.2.2",
+    "eslint": "^8.57.0",
+    "esno": "^4.7.0",
+    "nodemon": "^3.1.4",
+    "pnpm": "^8.15.7",
+    "rimraf": "^5.0.5",
+    "simple-git-hooks": "^2.11.1",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.4.5",
+    "unbuild": "^2.0.0",
+    "vite": "^5.2.10",
+    "vitest": "^1.5.2"
+  },
+  "gitHead": "b05c17bee481d90c4cd82aa2a34ac428f8263a5f"
+}