bas 1.5.2 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b74f1cf242bbd193d745aa8c75166409fb495b8e080717c0db3f03419b7a6c0
-  data.tar.gz: f0eb95ba93a7cee05eb3632b38b1e9f89d3c8bb1a078001799a26a4e5af90a55
+  metadata.gz: 2cd36e19f37702304576b44e8ab38c16ad18d3cb8ca59a44ff80fd8197ec2ba6
+  data.tar.gz: 67fd4cd540aafb0460a5a8ca4c7a630e8372483045f73298f3ccf6fe9fd3b69e
 SHA512:
-  metadata.gz: 84205602165ebaf52cd11368f5622dd56761a1e6bdaa08ad8bb53d1b40bf431264248ece8c4b9da7edf06816e3d07088c74b914d350da6623354743005bdf076
-  data.tar.gz: 22f0220be75faf2e3417c186fd39bcb73b7742e468022e4022a88ec96d5a42965d2a846034f56e643ef48b579f837a5fce4cfcaac3f87b1fcf937ed942ce70bc
+  metadata.gz: 8cff215e83e700143b36682d9a5dc0a222adb877a6ee525a105706b4e5f9797f21ac624627cc1f68102eecd6cc5ff2be3e4211cb42d3da1d31393dc98cef53c8
+  data.tar.gz: 69e0e74411e93720777fa1a4c4ae453d65b4f288f826297b38f2a208cea6aa213842f6de04fe311185ab30fb4b4671dd49e2ee19f464f9be5da58677d9bf84b4

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+# 1.6.0 (12.11.2024)
+- [Add Shared Storage Abstraction](https://github.com/kommitters/bas/issues/115)
+
+# 1.5.3 (28.10.2024)
+- [Fix response length limitations](https://github.com/kommitters/bas/issues/112)
+
 # 1.5.2 (08.10.2024)
 - [Change some details in Discord utils and review media](https://github.com/kommitters/bas/issues/108)
 

data/README.md

@@ -26,84 +26,126 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
 * Ruby 2.6.0 or higher
 
-## BOT
-A bot is a tool in charge of executing a specific automation task. The pipeline of a bot consists of reading from a data source, processing a specific task, and then writing a result in storage.
+## Terms
 
-## Use case
-A use case is an automation problem greater than the one managed by a single bot. A use case comprises a set of bots that are agnostic between them and each one solves a specific task (parts of the automation problem). To connect the bots, a shared storage should be used. This shared storage could be a PostgresDB database, an S3 bucket, or any kind of storage where the bots will read and write data so another bot can use it to execute their tasks.
+### BOT
+A bot is a tool responsible for executing a specific automation task. The bot's pipeline consists of reading from a shared storage (if required), processing a particular task, and then writing the result to a shared storage.
 
-For example, a system to notify birthdays in a company (automation problem) can be solved with three bots: one to fetch the data from an external data source, one to format the birthday message, and the last one to notify somewhere.
+### Shared Storage
+Shared storage is a central location where data can be read and written by multiple bots to facilitate coordination in an automation task. It serves as a common data exchange point, allowing each bot to access information generated by others as needed to complete its specific task. Examples of shared storage include a PostgreSQL database, an S3 bucket, or other types of storage systems that support concurrent access and data persistence.
 
-## Building my own BOT
-
-The gem provides with basic interfaces, types, and methods to shape your own bot in an easy way. Since the process of reading and writing in a shared storage is separated from the main task, two base classes were defined to deal with this executions, leaving the logic of the specific task in the bot file.
+### Use case
+A use case refers to an automation problem larger than what a single bot can manage. It typically involves a set of bots, each solving a specific part of the overall problem independently. To enable these bots to interact, shared storage is used. This shared storage could be a PostgreSQL database, an S3 bucket, or another shared storage type, allowing bots to read and write data that other bots can access to complete their tasks.
 
-### 1. Read - Obtaining the data from the Shared Storage
+## Building my own BOT
 
-Specifically, a reader is an object in charged of bringing data from a shared storage. The gem already provides the base interface
-for building your own reader for your specific shared storage, or rely on already built classes if they match your purpose.
+The gem provides essential interfaces, types, and methods to help you easily create your own bot. For instance, two base classes are provided: one for handling shared storage read-write operations and another for defining the bot’s specific task logic.
 
-The base interface for a reader can be found under the `bas/read/base.rb` class.
+### 1. SharedStorage - Read and Write data.
 
-### 2. Write - Apply changes in a shared storage
-The **Write** is in charge of creating or updating information in a shared storage. This is the last step for every BOT. These changes can be a transaction in a database, adding files in a cloud storage, or simply creating logs.
+The `SharedStorage` class abstracts the process of reading from and writing to shared storage. This class specification is available in `bas/shared_storage/base.rb`, from which custom implementations can be built.
 
-The base interface for a writer can be found under the `bas/write/base.rb` class.
+Currently, the gem supports: `PostgreSQL`.
 
-### 3. Bot - Solve a specific automation task
-The bot execute the logic to solve an specific task. For this, it can use the data from the read step, and then returns a processed response to be wrote by the write component. Every bot reads from a shared storage and writes in a shared storage.
+### 2. Bot - Solve a specific automation task
+The bot executes the logic required to complete a specific task. For this, an instance of shared storage (for reading and for writing) must be provided to the bot.
 
-The base interface for a bot can be found under the `bas/bot/base.rb` class.
+The base interface for a bot is available in the `bas/bot/base.rb class`.
 
 ## Examples
 
-In this example, we demonstrate how to instantiate a birthday notification bot and execute it in a basic Ruby project. We'll also cover its deployment in a serverless configuration, specifically using a simple Lambda deployment.
-
 ### Preparing the configurations
 
-We'll need some configurations for this specific use case:
-* A **Notion database ID**, from a database with the following structure:
+The current implementation of the PostgreSQL shared storage expect a table with the following structure: 
+
+```sql
+CREATE TABLE api_data(
+    id SERIAL NOT NULL,
+    "data" jsonb,
+    tag varchar(255),
+    archived boolean,
+    stage varchar(255),
+    status varchar(255),
+    error_message jsonb,
+    version varchar(255),
+    inserted_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
+    updated_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY(id)
+);
+```
 
-| Complete Name (text) |    BD_this_year (formula)   |         BD (date)        |
-| -------------------- | --------------------------- | ------------------------ |
-|       John Doe       |       January 24, 2024      |      January 24, 2000    |
-|       Jane Doe       |       June 20, 2024         |      June 20, 2000       |
+### Define the bot
 
-With the following formula for the **BD_this_year** column: `dateAdd(prop("BD"), year(now()) - year(prop("BD")), "years")`
+In this example, we demonstrate how to configure a bot to fetch data from an API and save it in a postgres database.
 
-* A Notion secret, which can be obtained, by creating an integration here: `https://developers.notion.com/`, browsing on the **View my integrations** option, and selecting the **New Integration** or **Create new integration** buttons.
+For this, a bot could be defined as follow:
+```ruby
+require "httparty"
 
-### Instantiating the FetchBirthdayFromNotion Bot
+module Bas
+  module Bot
+    class FetchFromAPi
+      URL = 'some-url.com'
 
-The specific bot can be found in the `bas/bot/fetch_birthdays_from_notion.rb` file.
+      def process
+        request = HTTParty.get(URL, { headers: })
 
-**Normal ruby code**
+        if request.code == 200
+          { success: request.body }
+        else
+          { error: request.error_message }
+        end
+      end
+
+      private
+
+      def headers
+        {
+          "Authorization" => "Bearer #{process_options[:token]}",
+          "Content-Type" => "application/json"
+        }
+      end
+    end
+  end
+end
 ```
+
+The `Bot::Base` interface expects the process method to return a hash with a single key, which can be `success` (when the response is valid) or `error` (when an error occurs). This consistency allows SharedStorage to handle the same data types regardless of the specific bot used.
+
+Finally, to execute the bot we could define it like follows:
+
+```ruby
+connection = {
+  host: "localhost",
+  port: 5432,
+  dbname: "bas",
+  user: "postgres",
+  password: "postgres"
+}
+
+read_options = {
+  connection:,
+  db_table: "api_data",
+  tag: "AnotherBot"
+}
+
+write_options = {
+  connection:,
+  db_table: "api_data",
+  tag: "FetchFromAPi"
+}
+
 options = {
-  process_options: {
-    database_id: "notion database id",
-    secret: "notion secret"
-  },
-  write_options: {
-    connection: {
-      host: "host",
-      port: 5432,
-      dbname: "bas",
-      user: "postgres",
-      password: "postgres"
-    },
-    db_table: "use_cases",
-    bot_name: "FetchBirthdaysFromNotion"
-  }
+  token: "api_token"
 }
 
-bot = Bot::FetchBirthdaysFromNotion.new(options)
-bot.execute
+shared_storage = SharedStorage.new(read_options:, write_options:)
 
+bot = Bas::Bot::FetchFromAPi.new(options, shared_storage)
+bot.execute
 ```
 
-## Real case usage
-For implementation details, please refer to the examples folder in the repository. You can find an example of how to configure a Lambda function with a bot there.
+The `tag` parameter in the read and write options specifies which record to look at (when reading) and what tag to assign to the record (when writing). This way, each bot can work independently with its own records in the shared storage.
 
 ## Development
 

data/lib/bas/bot/base.rb

@@ -2,73 +2,56 @@
 
 require_relative "../utils/exceptions/function_not_implemented"
 require_relative "../utils/exceptions/invalid_process_response"
-require_relative "../write/postgres_update"
 
-module Bot
-  ##
-  # The Bot::Base class serves as the foundation for implementing specific bots. Operating
-  # as an interface, this class defines essential attributes and methods, providing a blueprint
-  # for creating custom bots formed by a Read, Process, and Write components.
-  #
-  class Base
-    attr_reader :read_options, :process_options, :write_options
-    attr_accessor :read_response, :process_response, :write_response
+module Bas
+  module Bot
+    ##
+    # The Bot::Base class serves as the foundation for implementing specific bots. Operating
+    # as an interface, this class defines essential attributes and methods, providing a blueprint
+    # for creating custom bots formed by a Read, Process, and Write components.
+    #
+    class Base
+      attr_reader :process_options, :shared_storage_reader, :shared_storage_writer
+      attr_accessor :read_response, :process_response, :write_response
 
-    def initialize(config)
-      @read_options = config[:read_options] || {}
-      @process_options = config[:process_options] || {}
-      @write_options = config[:write_options] || {}
-    end
-
-    def execute
-      @read_response = read
-
-      write_read_response_in_process
-
-      @process_response = process
-      raise Utils::Exceptions::InvalidProcessResponse unless process_response.is_a?(Hash)
-
-      write_read_response_processed
-
-      @write_response = write
-    end
+      def initialize(options, shared_storage_reader, shared_storage_writer = nil)
+        @process_options = options || {}
+        @shared_storage_reader = shared_storage_reader
+        @shared_storage_writer = shared_storage_writer || shared_storage_reader
+      end
 
-    protected
+      def execute
+        @read_response = read
 
-    def read
-      raise Utils::Exceptions::FunctionNotImplemented
-    end
+        @shared_storage_reader.set_in_process
 
-    def process
-      raise Utils::Exceptions::FunctionNotImplemented
-    end
+        @process_response = process
+        raise Utils::Exceptions::InvalidProcessResponse unless process_response.is_a?(Hash)
 
-    def write
-      raise Utils::Exceptions::FunctionNotImplemented
-    end
+        @shared_storage_reader.set_processed
 
-    def unprocessable_response
-      read_data = read_response.data
+        @write_response = write
+      end
 
-      read_data.nil? || read_data == {} || read_data.any? { |_key, value| [[], "", nil].include?(value) }
-    end
+      protected
 
-    private
+      def read
+        @shared_storage_reader.read
+      end
 
-    def write_read_response_in_process
-      return if read_options[:avoid_process].eql?(true) || read_response.id.nil?
-
-      options = { params: { stage: "in process" }, conditions: "id=#{read_response.id}" }
-
-      Write::PostgresUpdate.new(read_options.merge(options)).execute
-    end
+      def process
+        raise Utils::Exceptions::FunctionNotImplemented
+      end
 
-    def write_read_response_processed
-      return if read_options[:avoid_process].eql?(true) || read_response.id.nil?
+      def write
+        @shared_storage_writer.write(process_response)
+      end
 
-      options = { params: { stage: "processed" }, conditions: "id=#{read_response.id}" }
+      def unprocessable_response
+        read_data = read_response.data
 
-      Write::PostgresUpdate.new(read_options.merge(options)).execute
+        read_data.nil? || read_data == {} || read_data.any? { |_key, value| [[], "", nil].include?(value) }
+      end
     end
   end
 end

data/lib/bas/shared_storage/base.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require_relative "../utils/exceptions/function_not_implemented"
+
+module Bas
+  module SharedStorage
+    # SharedStorage base interface
+    #
+    class Base
+      attr_reader :read_options, :read_response, :write_response
+      attr_accessor :write_options
+
+      # Initializes the read with essential configuration parameters.
+      #
+      def initialize(options = {})
+        @read_options = options[:read_options] || {}
+        @write_options = options[:write_options] || {}
+      end
+
+      def set_in_process; end
+
+      def set_processed; end
+
+      protected
+
+      def read
+        raise Utils::Exceptions::FunctionNotImplemented
+      end
+
+      def write
+        raise Utils::Exceptions::FunctionNotImplemented
+      end
+    end
+  end
+end

data/lib/bas/shared_storage/default.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require_relative "base"
+require_relative "types/read"
+
+module Bas
+  module SharedStorage
+    ##
+    # The SharedStorage::Default class serves as a shared storage implementation for bots
+    # who don't read from a <b>shared storage</b>".
+    #
+    class Default < Bas::SharedStorage::Base
+      def read
+        Bas::SharedStorage::Types::Read.new
+      end
+    end
+  end
+end

data/lib/bas/shared_storage/postgres.rb

@@ -0,0 +1,95 @@
+# frozen_string_literal: true
+
+require_relative "base"
+require_relative "types/read"
+require_relative "../utils/postgres/request"
+require_relative "../version"
+
+require "json"
+
+module Bas
+  module SharedStorage
+    ##
+    # The SharedStorage::Postgres class serves as a shared storage implementation to read and write on
+    # a shared storage defined as a postgres database
+    #
+    class Postgres < Bas::SharedStorage::Base
+      TABLE_PARAMS = "data, tag, archived, stage, status, error_message, version"
+
+      def read
+        params = { connection: read_options[:connection], query: read_query }
+
+        first_result = Utils::Postgres::Request.execute(params).first || {}
+
+        @read_response = Bas::SharedStorage::Types::Read.new(first_result[:id], first_result[:data],
+                                                             first_result[:inserted_at])
+      end
+
+      def write(data)
+        params = { connection: write_options[:connection], query: write_query(data) }
+
+        @write_response = Utils::Postgres::Request.execute(params)
+      end
+
+      def set_in_process
+        return if read_options[:avoid_process].eql?(true) || read_response.id.nil?
+
+        update_stage(read_response.id, "in process")
+      end
+
+      def set_processed
+        return if read_options[:avoid_process].eql?(true) || read_response.id.nil?
+
+        update_stage(read_response.id, "processed") unless @read_response.nil?
+      end
+
+      private
+
+      def read_query
+        query = "SELECT id, data, inserted_at FROM #{read_options[:db_table]} WHERE status='success' AND #{where}"
+
+        [query, where_params]
+      end
+
+      def write_query(data)
+        query = "INSERT INTO #{write_options[:db_table]} (#{TABLE_PARAMS}) VALUES ($1, $2, $3, $4, $5, $6, $7);"
+        params = write_params(data)
+
+        [query, params]
+      end
+
+      def where
+        return read_options[:where] unless read_options[:where].nil?
+
+        "archived=$1 AND tag=$2 AND stage=$3 ORDER BY inserted_at ASC"
+      end
+
+      def where_params
+        return read_options[:params] unless read_options[:params].nil?
+
+        [false, read_options[:tag], "unprocessed"]
+      end
+
+      def write_params(data)
+        if data[:success]
+          [data[:success].to_json, write_options[:tag], false, "unprocessed", "success", nil, Bas::VERSION]
+        else
+          [nil, write_options[:tag], false, "unprocessed", "failed", data[:error].to_json, Bas::VERSION]
+        end
+      end
+
+      def update_stage(id, stage)
+        params = { connection: read_options[:connection], query: update_query(id, stage) }
+
+        Utils::Postgres::Request.execute(params)
+      end
+
+      def update_query(id, stage)
+        query = "UPDATE #{read_options[:db_table]} SET stage=$1 WHERE id=$2"
+        values = [stage, id]
+
+        [query, values]
+      end
+    end
+  end
+end

data/lib/bas/shared_storage/types/read.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Bas
+  module SharedStorage
+    module Types
+      ##
+      # Represents a response from a read component. It encapsulates the requested data
+      # from the <b>shared storage</b> to be processed by a Bot.
+      class Read
+        attr_reader :id, :data, :inserted_at
+
+        def initialize(id = nil, response = nil, inserted_at = nil)
+          @id = id
+          @data = response.nil? ? {} : JSON.parse(response)
+          @inserted_at = inserted_at
+        end
+      end
+    end
+  end
+end

data/lib/bas/utils/discord/request.rb

@@ -39,10 +39,23 @@ module Utils
         }
       end
 
-      def self.write_media_text(params)
+      def self.write_media_text(params, combined_paragraphs)
         url_message = URI.parse("#{DISCORD_BASE_URL}/channels/#{params[:channel_id]}/messages")
         headers = headers(params[:secret_token])
-        HTTParty.post(url_message, { body: params[:body].to_json, headers: })
+        body = { content: combined_paragraphs }.to_json
+
+        HTTParty.post(url_message, { body:, headers: })
+      end
+
+      def self.split_paragraphs(params)
+        paragraphs = params[:body].split("--DIVISION--").map(&:strip).reject(&:empty?)
+
+        paragraphs.each_slice(2) do |paragraph|
+          next if paragraph.empty?
+
+          combined_paragraphs = paragraph.join("\n\n")
+          write_media_text(params, combined_paragraphs)
+        end
       end
 
       def self.headers(secret_token)

data/lib/bas/utils/openai/run_assistant.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "json"
 require "httparty"
 
 module Utils

data/lib/bas/utils/postgres/request.rb

@@ -24,7 +24,9 @@ module Utils
       def self.execute(params)
         pg_connection = PG::Connection.new(params[:connection])
 
-        execute_query(pg_connection, params[:query])
+        results = execute_query(pg_connection, params[:query])
+
+        results.map { |result| result.transform_keys(&:to_sym) }
       end
 
       # Execute the Postgres query

data/lib/bas/version.rb

@@ -2,5 +2,5 @@
 
 module Bas
   # Gem version
-  VERSION = "1.5.2"
+  VERSION = "1.6.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bas
 version: !ruby/object:Gem::Version
-  version: 1.5.2
+  version: 1.6.0
 platform: ruby
 authors:
 - kommitters Open Source
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-10-08 00:00:00.000000000 Z
+date: 2024-11-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gmail_xoauth
@@ -166,7 +166,7 @@ dependencies:
         version: '1.5'
 description: A versatile business automation system offering key components for building
   various use cases.     It provides an easy-to-use tool for implementing automation
-  workflows without excessive complexity.     Formerly known as 'bas'.
+  workflows without excessive complexity.     Formerly known as 'bns'.
 email:
 - oss@kommit.co
 executables: []
@@ -187,40 +187,10 @@ files:
 - examples/serverless_example.md
 - lib/bas.rb
 - lib/bas/bot/base.rb
-- lib/bas/bot/compare_wip_limit_count.rb
-- lib/bas/bot/create_work_item.rb
-- lib/bas/bot/fetch_billing_from_digital_ocean.rb
-- lib/bas/bot/fetch_birthdays_from_notion.rb
-- lib/bas/bot/fetch_domain_services_from_notion.rb
-- lib/bas/bot/fetch_domains_wip_counts_from_notion.rb
-- lib/bas/bot/fetch_domains_wip_limit_from_notion.rb
-- lib/bas/bot/fetch_emails_from_imap.rb
-- lib/bas/bot/fetch_github_issues.rb
-- lib/bas/bot/fetch_images_from_discord.rb
-- lib/bas/bot/fetch_media_from_notion.rb
-- lib/bas/bot/fetch_next_week_birthdays_from_notion.rb
-- lib/bas/bot/fetch_next_week_ptos_from_notion.rb
-- lib/bas/bot/fetch_ptos_from_notion.rb
-- lib/bas/bot/format_birthdays.rb
-- lib/bas/bot/format_do_bill_alert.rb
-- lib/bas/bot/format_emails.rb
-- lib/bas/bot/format_wip_limit_exceeded.rb
-- lib/bas/bot/garbage_collector.rb
-- lib/bas/bot/humanize_pto.rb
-- lib/bas/bot/notify_discord.rb
-- lib/bas/bot/notify_do_bill_alert_email.rb
-- lib/bas/bot/review_domain_availability.rb
-- lib/bas/bot/review_media.rb
-- lib/bas/bot/update_review_media_state.rb
-- lib/bas/bot/update_work_item.rb
-- lib/bas/bot/verify_issue_existance_in_notion.rb
-- lib/bas/bot/write_domain_review_requests.rb
-- lib/bas/bot/write_media_review_in_discord.rb
-- lib/bas/bot/write_media_review_requests.rb
-- lib/bas/read/base.rb
-- lib/bas/read/default.rb
-- lib/bas/read/postgres.rb
-- lib/bas/read/types/response.rb
+- lib/bas/shared_storage/base.rb
+- lib/bas/shared_storage/default.rb
+- lib/bas/shared_storage/postgres.rb
+- lib/bas/shared_storage/types/read.rb
 - lib/bas/utils/digital_ocean/request.rb
 - lib/bas/utils/discord/integration.rb
 - lib/bas/utils/discord/request.rb
@@ -238,9 +208,6 @@ files:
 - lib/bas/utils/openai/run_assistant.rb
 - lib/bas/utils/postgres/request.rb
 - lib/bas/version.rb
-- lib/bas/write/base.rb
-- lib/bas/write/postgres.rb
-- lib/bas/write/postgres_update.rb
 - renovate.json
 - sig/business_automation_system.rbs
 homepage: https://github.com/kommitters/bas
@@ -264,7 +231,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.16
+rubygems_version: 3.5.22
 signing_key:
 specification_version: 4
 summary: BAS - Business automation system