prometheus-splash 0.0.3 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4f2d1cf627c0fd2371da011a6a97a24e82498b9a28b3d4d0485e185841bf7438
-  data.tar.gz: e0982026ea749cad8366f420e88cd376bcc8bd0a45e0ec292cc1b9a8237764c2
+  metadata.gz: 4ae0ce3acc9966d9dc6bcc162a6691b01a97fa0368595dacbc4e12c36d830371
+  data.tar.gz: b236034c20105a79dd770ce10334aea80c67dc4f624d059b82f425e862b601e1
 SHA512:
-  metadata.gz: b43dcf0493e86ac31da666c4a68c8f2f22f804231f1fd6c0bcc98e3235605e80a4c8e8def976a7c4bf1c54110bdbfe0e67d64040c9d857d23dc16a324cdd1826
-  data.tar.gz: 18b7dea993feea9e1a276bf02b5542d98a772754eb68b113bde17ba5f69ec9c346eaa56520d8b960aa658ba1cc4b53a17dd5f7e4af256f8d07b404131573acd4
+  metadata.gz: 21daf35dfb3e696ef172cd678d7d2a644c2ef46aceef16cd32c889356f60a0403cd8cbfd1246944b0b7a84acbf8ea627ecd718ae830d48d839abdbd0d3ac5db9
+  data.tar.gz: 8e6cea17fcdf8fbd7facf65e34ca79e166cf3f26b517f454024fd46ef0b9d18aec7e103ff4ff4efdbf94531beeb6a1f490817e23e9b18bda395955d63cc34d0a

data/README.md

@@ -1,27 +1,661 @@
 # Splash
 
-SPLASH is *Supervision with Prometheus of Logs and Asynchronous tasks for Services or Hosts*
+![Splash logo](assets/images/logo_splash_reduce.png) _Orchestration and Supervision made easy_
+
+SPLASH is **Supervision with Prometheus of Logs and Asynchronous tasks orchestration for Services or Hosts**
 
 
 Prometheus Logs and Batchs supervision over PushGateway
 
-## Installation
 
-Add this line to your application's Gemfile:
+## Preconfiguration
+
+You need a Prometheus PushGateway operational, if the service not run on localhost:9091,
+See Prometheus server Configuration chapter to precise it in the configuration
+
+You need Ruby on the server you want to run Splash
+Splash is succesfully tested with Ruby 2.7.0, but it should works correctly with all Ruby 2.X versions.
+
+On Ubuntu :
+
+    # apt install ruby
+
+In some use case, Splash also require some other components :
+
+- Redis
+- RabbitMQ
+
+It's not strictly required, Redis is a real option for backend; you could configure backend to flat file, but
+RabbitMQ is required by the Splash Daemon when using host2host sequence execution.
+
+Redis, is usefull when you need a centralized Splash management.
+
+On Ubuntu :
+
+    # apt install redis-server rabbimq-server
+
+See Backends Configuration  and Transports Configuration to specify this services configurations
 
-    gem 'splash'
 
-And then execute:
+## Installation
 
-    $ bundle
 
-Or install it yourself as:
+Install with gem command :
 
     $ gem install splash
 
+
+## Configuration
+
+As root or with rvmsudo, if you use RVM.
+
+    # splash config setup              
+    Splash -> setup :
+    * Installing Configuration file : /etc/splash.yml : [OK]
+    * Installing template file : /etc/splash_execution_report.tpl : [OK]
+    * Creating/Checking pid file path : /var/lib/splash : [OK]
+    * Creating/Checking trace file path : /var/lib/splash : [OK]
+    Splash config successfully done.
+
+*WARNING : if you have already configured Splash, running this command without --preserve flag, RESET the Splash Configuration.*
+
+
+As root, edit /etc/splash.conf and adapt Prometheus Pushgateway Configuration :
+
+    # vi /etc/splash.yml
+    [..]
+      :prometheus:
+        :pushgateway:
+          :host: <SERVER>
+          :port: <PORT>
+    [..]
+
+With :
+
+- SERVER : IP or fqdn of the Gateway.
+- PORT : the specific TCP port of the Gateway.
+
+
+### Sanitycheck
+
+As root or with rvmsudo, if you use RVM.
+
+    # splash conf san
+    Splash -> sanitycheck :
+    * Config file : /etc/splash.yml : [OK]
+    * PID Path : /tmp : [OK]
+    * trace Path : /tmp/splash : [OK]
+    * Prometheus PushGateway Service running : [OK]
+    Sanitycheck finished with no errors
+
+*WARNING* : setup or Sanitycheck could precises errors if path defined in configuration is *Symbolic links*, type :mode.
+But it's not a problem for Splash to be operational.
+
+For file/folders if problems is detected, it could be such as :
+
+- :mode : UNIX rights errors
+- :owner : UNIX file owner errors
+- :group : UNIX file group errors
+- :inexistant : file/folder is missing
+
+### getting current VERSION
+
+run :
+
+    $ splash config version
+    Splash version : 0.0.3, Author : Romain GEORGES <gems@ultragreen.net>
+    Ultragreen (c) 2020 BSD-2-Clause
+
+
 ## Usage
 
-TODO: Write usage instructions here
+### Logs monitoring
+
+#### Edit your configuration
+
+
+In the /etc/splash.yml, you need to adapt default config to monitor your logs.
+
+    # vi /etc/splash.yml
+    [..]
+    ### configuration of monitored logs
+      :logs:
+        - :log: /a/log/path.log
+          :pattern: <regexp pattern>
+        - :log: /an/other/log/path.log
+          :pattern: <regexp pattern
+        - <etc...>
+    [..]
+
+Config for log is a YAML list of Hash, with keys :
+
+- :log : a log absolut paths
+- :pattern : a regular expression splash need to detect
+
+
+#### Prerequisite
+
+To ensure you have the default configuration values run as root :
+
+    # splash conf set
+
+*INFO* : comamnds must be reduce with the Thor completion facilities
+
+To see all monitoring commands with Splash, run :
+
+    # splash logs
+
+or
+
+    # slash logs help
+    Commands:
+      splash logs analyse         # analyze logs in config
+      splash logs help [COMMAND]  # Describe subcommands or one specific subcommand
+      splash logs list            # Show configured logs monitoring
+      splash logs monitor         # monitor logs in config
+      splash logs show LOG        # show configured log monitoring for LOG
+
+*Typicallly, the way work with all Splash commands or subcommands*
+
+
+
+#### Run a first test
+
+Verify /tmp/test and /tmp/test2 not existence
+
+    # rm /tmp/test /tmp/test2
+
+Verify configured logs :
+
+    # splash logs list
+    Splash configured log monitoring :
+     *  log monitor : /tmp/test
+     *  log monitor : /tmp/test2
+
+You could run list commands with --detail option , verify it with :
+
+    # splash command subcommand help
+
+like :
+
+    # splash logs list --detail
+    Splash configured log monitoring :
+     *  log monitor : /tmp/test
+      ->   pattern : /ERROR/
+     *  log monitor : /tmp/test2
+      ->   pattern : /ERROR/
+
+You cloud view a specific logs record detail with
+
+    # splash logs show /tmp/test2  
+    Splash log monitor : /tmp/test2
+      ->   pattern : /ERROR/
+
+Run a first analyse, you would see :
+
+    # splash logs analyse
+    SPlash Configured logs status :
+      * Log : /tmp/test : [KO]
+        - Detected pattern : ERROR
+        - detailled Status : missing
+      * Log : /tmp/test2 : [KO]
+        - Detected pattern : ERROR
+        - detailled Status : missing
+    Global Status : [KO]
+
+Create empty Files, or without ERROR string in.
+
+    # echo 'foo' > /tmp/test
+    # touch /tmp/test2
+
+Re-run analyse :
+
+    # splash log an
+    SPlash Configured logs status :
+      * Log : /tmp/test : [OK]
+        - Detected pattern : ERROR
+        - detailled Status : clean
+          nb lines = 1
+      * Log : /tmp/test2 : [OK]
+        - Detected pattern : ERROR
+        - detailled Status : clean
+          nb lines = 0
+    Global Status : [OK]
+
+It's alright, log monitoring work fine.
+
+#### Send metrics to Prometheus gateway
+
+Splash is made to run a specific daemon to do this job, but you could do one time, with :
+
+    # splash logs monitor
+    Sending metrics to Prometheus Pushgateway
+      * Sending metrics for /tmp/test
+      * Sending metrics for /tmp/test2
+    Sending done.
+
+if Prometheus Gateway is not running or misconfigured, you could see :
+
+    Prometheus PushGateway Service IS NOT running
+    Exit without notification.
+
+Otherwise Prometheus PushGateway have received the metrics :
+
+
+- *logerrors*, Prometheus Gauge : with label: <the logname> and job: 'Splash'
+  => description : SPLASH metric log error'
+  => content :<nb match> the number of pattern matching for the log
+
+- *logmissing*, Prometheus Gauge : with label: <the logname> and job: 'Splash'
+  => description : SPLASH metric log missing'
+  => content :0 if log exist, 1 if log missing
+
+- *loglines*, Prometheus Gauge : with label: <the logname> and job: 'Splash'
+  => description : SPLASH metric log line numbers'
+  => content :0 if log missing, <nb lines in the log> the number of lines in the logs
+
+#### See it in Prometheus PushGateway
+
+visit http://<prometheus_pushgateway_host>:<prometheus_pushgateway_port>/
+
+![prom PG logs](assets/images/prom_pg_logs.png)
+
+![prom PG details logs](assets/images/detail_prom_splash.png)
+
+
+### Commands Orchestration, running and monitoring
+
+#### List of commands
+
+To see all the commands in the 'commands' submenu :
+
+    $ splash commands help
+
+    $ splash commands                           
+    Commands:
+      splash commands help [COMMAND]   # Describe subcommands or one specific subcommand
+      splash commands lastrun COMMAND  # Show last running result for specific configured command COMMAND
+      splash commands list             # Show configured commands
+      splash commands run NAME         # run for command/sequence or ack result
+      splash commands show COMMAND     # Show specific configured command COMMAND
+      splash commands treeview         # Show commands sequence tree
+
+#### Prepare test with default configuration
+
+Commands or Commands Sequences must be defined in the main configuration file '/etc/splash.yml'
+
+*Exemple* in default configuration :
+
+    ### configuration of commands and scheduling
+      :commands:
+        :id_root:
+          :desc: run id command on root
+          :command: id root
+
+        :true_test:
+          :desc: "test command returning true : 0"
+          :command: "true"
+          :schedule:
+            :every: "1h"
+          :on_failure: :ls_slash_tmp
+          :on_success: :pwd
+
+        :false_test:
+          :desc: "test command returning false > 0"
+          :command: "false"
+          :schedule:
+            :every: "1h"
+          :on_failure: :ls_slash_tmp
+          :on_success: :pwd
+
+        :ls_slash_tmp:
+          :desc: list file in /tmp
+          :command: ls -al /tmp
+          :user: daemon
+          :on_success: :echo1
+
+        :pwd:
+          :desc: run pwd
+          :command: pwd
+          :on_success: :echo1
+          :on_failure: :echo2
+
+        :echo1:
+        :desc: echo 'foo'
+        :command: echo foo
+        :on_failure: :echo3
+
+      :echo2:
+        :desc: echo 'bar'
+        :command: echo bar
+
+      :echo3:
+        :desc: echo 'been'
+        :command: echo been
+
+A configuration block for commands must include :
+
+* *key* : a name as Symbol (:xxxxxx)
+* *values* : (hash)
+  * :desc : a brief Description
+  * :command : the full command line
+
+may include :
+
+* :user: the userneme to use to run the command
+* :on_failure: the name of an other defined command, to, execute if exit_code > 0
+* :on_success: the name of an other defined command, to, execute if exit_code = 0
+* :schedule:  (hash) a scheduling for daemon, after in this documentation, it support :
+  * :every: "<timing>" ex: "1s", "3m", "2h"
+  * :at: "<date/time>" ex: "2030/12/12 23:30:00"
+  * :cron: * * * * * a cron format
+
+[Rufus Scheduler Doc](https://github.com/jmettraux/rufus-scheduler)
+
+if you want to inject default configuration, again as root :
+
+  # splash conf set
+
+
+#### listing the defined Commands
+
+You could list the defined commands, in your case :
+
+    $ splash commands list
+    Splash configured commands :
+    * id_root
+    * true_test
+    * false_test
+    * ls_slash_tmp
+    * pwd
+    * echo1
+    * echo2
+    * echo3
+
+#### Show specific commands
+
+You could show a specific command :
+
+    $ splash com show pwd
+    Splash command : pwd
+      - command line : 'pwd'
+      - command description : 'run pwd'
+      - command failure callback : 'echo2'
+      - command success callback : 'echo1'
+
+#### View Sequence execution for commands
+
+You could trace execution sequence for a commands as a tree, with :
+
+    # splash com treeview
+    Command : true_test
+      * on failure => ls_slash_tmp
+        * on success => echo1
+          * on failure => echo3
+      * on success => pwd
+        * on failure => echo2
+        * on success => echo1
+          * on failure => echo3
+
+In your sample, in all case :
+- :true_test return 0
+- :pwd return 0
+- :echo1 return 0
+
+commands execution sequence will be :
+
+:true_test => :pwd => :echo1
+
+:ls_slash_tmp, :echo2 and :echo3 will be never executed.
+
+#### Executing a standalone command :
+
+Running a standalone command with ONLY as root
+
+    # splash com execute echo1
+    Executing command : 'echo1'
+      * Tracefull execution
+       => exitcode 0
+      * Prometheus Gateway notified.
+
+This command :
+
+1. Execute command line defined in command 'echo1' defined in  configurations
+2. Trace information in a execution report :
+  - :start_date  the complete date time of execution start.
+  - :end_date  the complete date time of execution end.
+  - :cmd_name the name of the command
+  - :cmd_line the complete command line executed
+  - :stdout STDOUT of the command
+  - :stderr STDERR of the command
+  - :desc the description of the command
+  - :status : PID and exit_code of the command
+  - :exec_time : the timing of the command
+3. Notify Prometheus
+
+There is some usefull modifiers for this command :
+
+    --no-trace : prevent Splash to write report for this execution in configured backend
+    --no-notify : prevent Splash to nofify Prometheus PushGateway metric (see later in this documentation)
+    --no-callback : never execute callback (see it after)
+
+
+
+#### Executing a sequence of callback Commands
+
+Splash allow execution of callback (:on_failure, :on_success), you have already see it in config sample.
+In our example, we have see :true_test have a execution sequence, we're going to test this, as root :
+
+    # splash com exe true_test
+    Executing command : 'true_test'
+      * Tracefull execution
+       => exitcode 0
+      * Prometheus Gateway notified.
+      * On success callback : pwd
+    Executing command : 'pwd'
+      * Tracefull execution
+       => exitcode 0
+      * Prometheus Gateway notified.
+      * On success callback : echo1
+    Executing command : 'echo1'
+    * Tracefull execution
+     => exitcode 0
+    * Prometheus Gateway notified.
+
+We could verify the sequence determined with lastrun command.
+
+If you want to prevent callback execution, as root :
+
+      # splash com exe true_test --no-callback
+      Executing command : 'true_test'
+       * Tracefull execution
+        => exitcode 0
+       * Prometheus Gateway notified.
+       * Without callbacks sequences
+
+#### Display the last execution trace for a command
+
+If you want to view the last execution trace for  commande, (run with --trace : default)
+
+    # splash com lastrun
+    Splash command pwd previous execution report:
+
+    Command Execution report
+    ========================
+
+    Date START: 2020-04-07T18:25:22+02:00
+    Date END: 2020-04-07T18:25:22+02:00
+    Command : pwd
+    full command line : pwd
+    Description : run pwd
+    errorcode : pid 86782 exit 0
+    Execution time (sec) : 0.006568
+
+    STDOUT:
+    -------
+
+    /Users/ruydiaz/labo/prometheus-splash
+
+
+
+    STDERR:
+    -------
+### Advanced  Configuration
+
+#### Backend configuration
+
+For the moment Splash come with two types of backend :
+- :file if you would a standalone splash Usage
+- :redis if you want a distributed Splash usage
+
+backend are usable for :
+
+- execution trace
+
+##### File backend
+
+The file backend is very simple to use :
+
+Edit /etc/splash.yml, as root :
+
+    # vi /etc/splash.yml
+    [...]
+    :backends:
+      :stores:
+        :execution_trace:
+            :type: :file
+            :path: /var/run/splash
+    [...]
+
+- :type must be :file
+- :path should be set to the dedicated executions traces files path (default : /var/run/splash )
+
+##### Redis backend
+
+A little bit more complicated for Redis :
+
+Edit /etc/splash.yml, as root :
+
+    # vi /etc/splash.yml
+    [...]
+    :backends:
+      :stores:
+        :execution_trace:
+          :type: :redis
+          :host: localhost
+          :port: 6379
+          #:auth: "mykey"
+          :base: 1
+    [...]
+
+- :type must be :redis
+- :host must be set as the Redis server hostname (default: localhost)
+- :port must be set as the Redis server port (default: 6379)
+- :base must be set as the Redis base number (default: 1)
+- :auth should be set if Redis need an simple authentification key <mykey>
+
+##### Prometheus PushGateway configuration
+
+Prometheus PushGateway could be configured in /etc/splash.yaml
+
+    # vi /etc/splash.yml
+    [...]
+      :prometheus:
+        :pushgateway:
+          :host: "localhost"
+          :port: 9091
+    [...]
+
+-  :host should be set as the Prometheus PushGateway hostname (default: localhost)
+-  :port should be set as the Prometheus PushGateway port (default: 9091)
+
+### The Splash daemon
+
+#### Introduction
+
+We're going to discover the Big part of Splash the Daemon, usefull to :
+
+- orchestration
+- scheduling
+- Log monitoring (without CRON scheduling)
+- host2host sequences execution (optionnal )
+
+#### Controlling the daemon
+
+TODO
+
+#### Configuring the daemon
+
+TODO
+
+
+### Ecosystem
+
+#### Execution report Template adaptions
+
+TODO
+
+#### Ubuntu Ansible playbook
+
+TODO
+
+#### Systemd integration fo daemon
+
+TODO
+
+#### CRON usage with or without rvmsudo
+
+TODO
+
+#### Default values for configuration
+
+ defined in the lib/splash/constants.rb
+
+```ruby
+
+# the only one non-overridable by the configuration
+CONFIG_FILE = "/etc/splash.yml"
+
+
+TRACE_PATH="/var/run/splash"
+
+DAEMON_LOGMON_SCHEDULING={ :every => '20s'}
+DAEMON_PROCESS_NAME="Splash : daemon."
+DAEMON_PID_PATH="/var/run"
+DAEMON_PID_FILE="splash.pid"
+DAEMON_STDOUT_TRACE="stdout.txt"
+DAEMON_STDERR_TRACE="stderr.txt"
+
+AUTHOR="Romain GEORGES"
+EMAIL = "gems@ultragreen.net"
+COPYRIGHT="Ultragreen (c) 2020"
+LICENSE="BSD-2-Clause"
+
+PROMETHEUS_PUSHGATEWAY_HOST = "localhost"
+PROMETHEUS_PUSHGATEWAY_PORT = "9091"
+
+EXECUTION_TEMPLATE="/etc/splash_execution_report.tpl"
+EXECUTION_TEMPLATE_TOKENS_LIST = [:end_date,:start_date,:cmd_name,:cmd_line,:stdout,:stderr,:desc,:status,:exec_time]
+
+BACKENDS_STRUCT = { :list => [:file,:redis],
+                    :stores => { :execution_trace => { :type => :file, :path => "/var/run/splash" }}}
+TRANSPORTS_STRUCT = { :list => [:rabbitmq],
+                    :active => :rabbitmq,
+                    :rabbitmq => { :url => 'amqp://localhost/'} }
+
+```
+
+
+#### Splash CLI return code significations
+
+TODO
+
+### In the Futur ?
+
+- IHM
+- Webservice
+
+
 
 ## Contributing