flnews_post_proc 1.7

data/doc/rst/flnews_post_proc.rst

@@ -0,0 +1,336 @@
+=======================
+ flnews_post_proc
+=======================
+------------------------------------------
+ Post-Processor for the flnews newsreader
+------------------------------------------
+
+SYNOPSIS
+=======================
+
+An article is piped-in to the post-processor. This is normally done
+automatically as soon as the variable “post_proc“ in the flnews configuration
+file is set to the flnews_post_proc.
+
+Locally stored articles can be piped in to the post-processor, as needed
+for testing purposes, with a command-line like the following:
+
+**flnews_post_proc < article**
+
+DESCRIPTION
+=======================
+The flnews newsreader is sufficient for Usenet access, i.e. to receive and read
+articles from -, as well as to write and post articles to newsgroups.
+
+When you compare news-clients, you will always notice the differences and
+choose the software that you prefer. Flnews however, has the charm that you can
+influence how the program itself works but also modify posts that flnews
+produces, just before the program will transmit them to the chosen nntp-server.
+
+The flnews_post_proc can add and change details of a post, in ways that are
+currently not possible with flnews alone.  As the program is configurable, it
+can probably respond to the needs of some Usenet users. You should, however,
+rather take it as an example for what can be done and an inspiration for your
+own creations.
+
+The limits of a basic newsreader — what the program can do 
+----------------------------------------------------------
+While the articles that flnews creates, are complete and ready to be posted,
+some users may not always agree with the result and for arbitrary reasons:
+
+* There may be inconveniences when you post to different newsgroups in
+  different languages, as an introductory line which refers to a previous
+  post can only be set once in the flnews-configuration. The consequence
+  can be that your post to a french newsgroup begins with an introduction
+  in English.
+  My post-processor program can set an introductory line specifically
+  chosen for one or several newsgroups.
+
+* The same conflict arises, when you have set a standard signature-text and
+  would like to replace it against another, based on the newsgroup you are
+  about to post to.
+  The post-processor program sets specific signatures as configured for one
+  or several newsgroups. You can even specify that a signature should be
+  picked randomly from a file, containing several signatures.
+
+* Some custom headers may serve to convey additional information to
+  interested readers of your post, like GnuPG key IDs, your language skills
+  or the like. The signature may be a better choice than custom headers.
+  You are free. I just mention face and x-face but prefer that you do not
+  remember I did.
+  Custom-headers may be defined in the configuration file for the program
+  and will then be added to each outgoing post.
+
+* The Archive- and the X-No-Archive header are sometimes set to avoid that an
+  article be saved and stays available to search-engines (Google, notably).
+  Test- postings, for example, do probably not justify at all that they would
+  be referenced in search-results.
+  The post-processor program can impose both headers for all posts to certain
+  newsgroups.
+
+  **ATTN** As of 2024, the header “X-No-Archive“ has lost most of its utility  
+  and it is the decision of server operators to honor it or not.  
+
+* If a news post contains many references to either other posts or Web
+  pages, the text can be cluttered with URLs.
+  The post-processor can identify marked text fragments and transform them
+  into footnotes, which will be attached as a list at the bottom of the
+  post.
+  This works almost like the <ref/> link in Wikipedia, but the delimiter
+  can be determined in the configuration file.
+  Example (with %=):
+  This is an object %=and this becomes the footnote, describing the object
+  further=%
+
+Dialog to override settings
+---------------------------
+You can have a dialog displayed just before the post-processor is invoked, to
+**disable** some configuration options. Provided that either YAD, Zenity,
+Whiptail or only xterm are available on your computer, you can choose from 
+the following options. You **cannot** use the dialog to enable options which
+have not yet been set in the configuration.
+
+* Signatures, as set in the configuration **can be ignored**. Either
+  a default signature will appear as set in flnews or none.
+* Custom Headers, if configured, can be **omitted**.
+* The Archive- and X-No-Archive headers, if set for the current newsgroup,
+  **can be ignored**.
+* Logging can be **switched off**, if set.
+
+Pushing Esc or the cancle-button of the dialog interrupts the process, flnews
+will not post the article.
+
+You can disable the dialog, which ensures that all configured options will be
+applied, without the need for further interaction (see below, option
+OVERRIDE_CONFIG). 
+
+CONFIGURATION
+===============
+On first execution of the program, a copy of the original configuration file is
+created in */home/[user]/.flnews_post_proc.conf*
+It is this file which is used from then on. If you delete it, it will be
+recreated, on the next occasion, but your own changes will be lost.
+
+The configuration file is in YAML syntax and full of explanations. The
+variables defined in this file can be classified as belonging to one of two
+categories:
+
+* Variables describing values originally set by flnews, which should be used or
+  replaced. The important elements are usually matched in a capture group.
+
+* Variables defining the new or altered content.
+
+**FUP_NAME**
+ A Regular Expression, describing the string which contains the name of
+ previous poster who is the author of a quoted post.  This string is
+ recognized in the original article and may be used with the fitting element
+ from *GROUP_INTRO*, below. The Regexp-format is that of the Regexp class in
+ Ruby, noted as a String. Beware to mask a backslash '\\' by another one,
+ like in the example. A capture-group '()' serves to extract the name from the
+ match result.
+
+ Leave this field empty to keep the default from the FLNews configuration
+ intact.
+
+ CONTENT: A String equivalent of a regular expression.
+
+ DEFAULT: EMPTY
+
+ EXAMPLE1: "On \\\\d+.\\\\d+.\\\\d{2,4} at \\\\d+:\\\\d+ **(.*)** wrote:"
+
+ EXAMPLE2: "**(.*)** wrote:"
+
+
+**FUP_GROUP**
+ A Regular Expression, describing the string which contains the newsgroup
+ where the previous post, that you are referring to in the followup, had been
+ published.
+
+ Leave this field empty to ignore the precise group.
+
+ CONTENT: A String equivalent of a regular expression.
+
+ DEFAULT: EMPTY
+
+ EXAMPLE: "wrote in **(.*)**:"
+
+**GROUP_INTROS**
+ Introductory strings, referring to the previous poster who is the author of a
+ quoted post. If you match the newsgroup of the post (see FUP_GROUP), you can
+ use these variables in the result.
+ Currently only %fup_name% and %fup_group% are reproduced in the resulting
+ introductory string.
+
+ CONTENT: A newsgroup or regexp per line, followed by a colon, a space and a String
+
+ DEFAULT: As configured in FLNews
+ 
+ EXAMPLE: alt.test: "Thus spoke %fup_name% on that baleful %fup_date%:"
+
+**GROUP_SIGS**
+ A signature line per Newsgroup or a file path. The file should contain signatures,
+ already formatted and separated by 1 empty line. The program will randomly pick 
+ one signature from the list.
+
+ ATTN! In multi line signatures, you have to use \\r\\n for line breaks.
+
+ CONTENT: A newsgroup or regexp per line, followed by a colon, a space and a String.
+
+ DEFAULT: As configured in flnews
+
+ EXAMPLE: alt.test: "Signature for alt.test\\r\\nsecond line"
+ 
+ EXAMPLE: comp.*: /home/[user]/.my_sigs
+
+**CUSTOM_HEADERS**
+ Additional headers for the outgoing article
+
+ CONTENT: 1 line per header : a dash and space, then a String, comprising the
+ name of the header, ending in a colon and the value of the header. 
+
+ DEFAULT: undefined
+
+ | EXAMPLE (2 headers): 
+ | - 'X-My-Header: nothing fancy'
+ | - 'X-Another-Header: care not!' 
+
+**NO_ARCHIVE_GROUPS** 
+ The newsgroups, where the headers “Archive: no” and “X-No-Archive: YES” shall
+ be set.
+
+ CONTENT: a dash and space, then a String, containing the name of the group or a regexp.
+
+ DEFAULT: empty
+
+ | EXAMPLE (1 group, 1 hierarchy): 
+ | - "alt.test"
+ | - "^news.*"
+
+
+**DEBUG_LOG**
+ The name of a file, where debug messages are written. Setting this
+ variable will enable the log. Leave empty to disable logging.
+
+ CONTENT: The name of a writable file, which will be created if inexistent 
+ and overwritten if need be.
+
+ DEFAULT: empty
+
+ EXAMPLE: '/tmp/a_log-file.txt'
+
+**LOG LEVEL**
+ One of debug, fatal, error, info, warn
+
+**REFERENCES_SEPARATOR**
+ A symbol or sequence of symbols marking the end of the message-body and the
+ beginning of a list of “references” or “footnotes”. It will only appear, if
+ the original message-body contains text marked for use as such a footnote.
+ See *REFERENCES_DELIMITER*.
+
+ If the option is not defined or empty, the list of footnotes will appear
+ below the last line of the message body and no separator will be inserted.
+
+ CONTENT: A quoted symbol or sequence of symbols.
+
+ DEFAULT: empty
+
+ EXAMPLE: '---------'
+
+**REFERENCES_DELIMITER**
+ A sequence of at least two symbols marking the beginning of a text which will
+ serve as footnote (or reference). The **reversed sequence** musst be used to
+ mark the end of the text. The presence of this sequence or symbol in the
+ original message body will cause the enclosed text to be moved below the
+ message body.  The *REFERENCES_SEPARATOR*, if defined, will separate the
+ message from the list of footnotes.
+ If this option is not defined or empty, footnotes are not created.
+
+ CONTENT a quoted symbol or sequence of symbols.
+
+ DEFAULT: none/empty
+
+ EXAMPLE: '%?'
+
+**REFERENCE_FORMAT**
+ A format-string, using %s for a number, replacing the reference-
+ text in the message body.
+
+ DEFAULT: " %s)"  -> becomes 1) ... 2)  ... 3)
+
+ EXAMPLE: "(%s)"  -> becomes (1) ... (2)  ... (3)
+
+**VFY_URLS**  
+ A Boolean constant. It determines if the program shall verify and possibly
+ try to correct URLs. Even if URLs are identified as such, only a few
+ manipulations are attempted :
+ * Angular brackets '<' and '>' are added, if missing
+ * Slashes are added, if they are found missing after "http(s):" 
+
+ If the variable is not set, a value 'yes' is assumed.
+
+ CONTENT: One of YES, yes, NO, no, and other variations of case.
+
+ DEFAULT: yes
+
+ Example: ... I let you guess.
+
+**OVERRIDE_CONFIG**
+ A Boolean constant. You can choose to override the following configuration
+ options before an article is posted: GROUP_SIGS, XNAY_GROUPS, CUSTOM_HEADERS,
+ DEBUG_LOG and VFY_URLS.  A dialog may be displayed which allows you to disable
+ any of these five options, so that the defaults from flnews prevail.
+
+ ATTN! Canceling the dialog or pushing the Esc-key does interrupt the process.
+ Flnews will not post the article.
+
+ Set this option to no, NO or similar to disable the dialog.
+
+ DEFAULT: yes
+
+ EXAMPLE: No
+
+Other Information
+=================
+
+Testing
+-------
+The effects that the execution of the program will have on a posting can be
+verified in two ways:
+
+1. By piping-in a post that had previously been saved to a file:
+
+   **:~$ /usr/local/bin/[post-processor] < [test-article]**
+
+   This will show the resulting new version of the article on screen, but you
+   can also pipe the output into another file. This is a great way to test a
+   program during development or to test your own configuration of the program.
+2. By posting directly into a test-newsgroup (like alt.test or similar).  This
+   is mandatory before you really post to thematic newsgroups and when the
+   settings of the post-processor will affect the article.
+
+
+Source-Code
+-----------
+The gem-file that you get with the gem-utility or from rubygems.org contains
+all the code of the program and some documentation (this page notably). To read
+its content, you must
+
+1. untar the gem-file with tar -xf flnews_post_proc-0.1.gem
+2. uncompress the data.gz archive: gunzip data.gz
+3. untar the resultig data.tar archive: tar -xf data.tar
+
+This creates the directories bin, doc and lib.
+
+License
+-------
+flnews_post_proc is distributed under the conditions of the WTFPL-2.0 or later
+License (see http://www.wtfpl.net/txt/copying/ or license-text in the doc
+directory of the gem-file).
+
+Author
+------
+| flnews_post_proc has been developed by 
+| Michael Uplawski <michael.uplawski@uplawski.eu>
+
+Ω
+==

data/lib/_quoting_style_regexp

@@ -0,0 +1,2 @@
+# UNUSED
+" > >> > testerei ei ei".strip.gsub( /^>+(\s+>+)*/) {|m| m.to_s.delete(" ")}

data/lib/basic_logging.rb

@@ -0,0 +1,202 @@
+#!/bin/env ruby
+#encoding: UTF-8
+=begin
+/***************************************************************************
+ *   2023-2024, Michael Uplawski <michael.uplawski@uplawski.eu>            *
+ *   This program is free software; you can redistribute it and/or modify  *
+ *   it under the terms of the WTFPL 2.0 or later, see                     * 
+ *   http://www.wtfpl.net/about/                                           *
+ *                                                                         *
+ *   This program is distributed in the hope that it will be useful,       *
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of        *
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.                  *
+ *                                                                         *
+ ***************************************************************************/
+=end
+
+#
+# Simplified logging. 
+# See example code at the bottom of this file.
+# Execute this file to see the output.
+module BasicLogging
+
+  DEBUG = 0
+  INFO =  1
+  WARN =  2
+  ERROR = 3
+  FATAL = 4
+  UNKNOWN = nil
+  
+  # this is mainly for the translation of method calls into log levels
+  Levels = {:debug => DEBUG, :info => INFO, :warn => WARN, :error => ERROR,
+            :fatal => FATAL, :unknown => UNKNOWN}
+
+  @@log_level = UNKNOWN
+  @@target = STDOUT
+  @@muted = []
+
+  # do not log, if caller is obj (class or instance)
+  def self.mute(obj)
+    name = obj.class == Class ? obj.name.dup : obj.class.name
+    @@muted << name
+  end
+
+  def self.is_muted?(obj)
+    name = obj.class == Class ? obj.name.dup : obj.class.name
+    @@muted.include?(name)  
+  end
+
+  # set the log level 
+  def set_level(lv)
+    if lv.respond_to?(:to_str) && Levels.keys.include?(lv.strip.to_sym)
+      lv = Levels[lv.to_sym]
+    elsif lv.respond_to?(:to_sym) && Levels.keys.include?(lv)
+      lv = Levels[lv]
+    end
+
+    if(!lv || (lv.respond_to?(:to_int) && lv >= DEBUG && lv <= FATAL) )
+      @@log_level = lv
+    else
+      msg = __FILE__.dup << ": ERROR : invalid log level \"" << lv.to_s << "\""
+      msg << "\n" << "Keepinng old log level " << Levels.keys.detect {| k| Levels[k] == @@log_level}.to_s
+      STDERR.puts msg
+      puts msg
+    end
+  end
+
+  # set the log target 
+  def set_target(tg)
+    if tg.respond_to?(:to_io) 
+      @@target = tg
+    elsif(!File::exist?(tg) || ( File.file?(tg) && File.writable?(tg) ) )
+      @@target = File.open(tg, 'w+')  
+    elsif !tg || tg.respond_to?(:to_str) && tg.strip.empty?
+      @@target = nil
+    else
+      STDERR.puts __FILE__.dup << ': ERROR : target ' << tg << ' cannot be set'
+      STDERR.puts "Keeping old target " << @@target.inspect
+      return
+    end
+  end
+
+  # Output of log messages, depending on the log level set for the calling class
+  # and the name of the alias method which is actually called.
+  def log(message)
+    if !BasicLogging.is_muted?(self)
+      # how has this method been called?
+      mlevel = __callee__
+      if Levels.has_key?(mlevel) && Levels[mlevel] <=  FATAL
+        # output only for levels equal or above the value that corresponds to
+        # the calling alias.
+        format_log( message, mlevel) if @@log_level && Levels[mlevel] >= @@log_level 
+      else
+        STDERR.puts __FILE__.dup << ": ERROR : invalid log level \"" << mlevel.to_s << "\""
+      end
+    end
+  end
+
+  def target
+    @@target.path if @@target
+  end
+
+  def level
+    @@level.to_s if @@level
+  end
+
+  # Clear the log (-file)
+  def clear_log
+    if @@target && @@target.respond_to?(:truncate)
+      lock_target{ @@target.truncate(0) }
+    end
+  end
+
+  alias :debug :log
+  alias :info :log
+  alias :warn :log
+  alias :error :log
+  alias :fatal :log
+
+
+  private 
+
+  def lock_target(&block)
+    begin
+      if @@target.respond_to?(:flock) 
+        @@target.flock(File::LOCK_EX)
+        block.call
+        @@target.flock(File::LOCK_UN)
+      elsif @@target.respond_to?(:to_io)
+        block.call
+      end
+    rescue => ex
+      STDERR.puts __FILE__.dup  << ": ERROR : cannot lock target (" << ex.message << ")"
+    end
+  end
+
+  # 1 format_log for all loggers.
+  def format_log(message, mlevel)
+    if @@target
+      # indicate if a registered class or the registered object of a class is calling.
+      name = self.class == Class ? self.name.dup << ' [class]' : self.class.name
+      lock_target{@@target.puts '' << name << ' ' << mlevel.to_s << ' ' << Time.now.strftime("%H:%M:%S:%6N")  << ': ' << message.gsub("\n", "\n |")}
+    end
+  end
+end
+#---------test: execute file----------
+if $0 == __FILE__
+  Array.extend(BasicLogging)
+  Array.set_level(BasicLogging::INFO)
+  Array.info('TEST')
+  ar = Array.new
+  ar.extend(BasicLogging)
+  # --- no output : 
+  l = __LINE__
+  ar.debug(l.next.to_s << ': debug-test 0')
+  # output
+  ar.set_level(BasicLogging::DEBUG)
+  l = __LINE__
+  ar.debug(l.next.to_s << ': debug-test 1')
+
+  obj = Object.new
+  obj.extend(BasicLogging)
+  obj.set_level(BasicLogging::DEBUG)
+  puts "--------debug-----------"
+  obj.debug('debug')
+  obj.info('info')
+  obj.warn('warn')
+  obj.error('error')
+  obj.fatal('fatal')
+  puts "--------info-----------"
+  obj.set_level("info")
+  obj.debug('debug')
+  obj.info('info')
+  obj.warn('warn')
+  obj.error('error')
+  obj.fatal('fatal')
+  puts "--------fatal-----------"
+  obj.set_level("fatal")
+  obj.debug('debug')
+  obj.info('info')
+  obj.warn('warn')
+  obj.error('error')
+  obj.fatal('fatal')
+  puts "--------UNKNOWN-----------"
+  obj.set_level(nil)
+  obj.debug('debug')
+  obj.info('info')
+  obj.warn('warn')
+  obj.error('error')
+  obj.fatal('fatal')
+  puts " ------ Output into file ----"
+  obj.set_target "/tmp/test_log.log"
+  puts " ------ INFO -----------"
+  obj.set_level BasicLogging::INFO
+  obj.info('info output')
+
+  obj.info('info output 2') 
+  puts "---------- invalid -------"
+  obj.set_target "/dev/sr0"
+  obj.set_level "power"
+end
+
+# EOF