universal_pipe_handler 0.0.10

data/doc/README.gen

@@ -0,0 +1,417 @@
+ADD_DEFAULT_HEADER
+ADD_GOBOLINUX_LINK
+
+# The Universal Pipe Handler Project
+
+Handling Pipes - via Ruby
+
+<i>„Because one pipe can rule them all ...“</i>
+
+<img src="https://i.imgur.com/c7lFded.jpg" style="margin: 1em">
+
+## Description
+
+This project aims to implement pipes, in ruby. The idea here is to be able
+to use a fully OOP-centric implementation of pipes, rather than "merely"
+a simulation of UNIX/Linux pipes as such. Thus, the project attempts to
+include ideas from other projects, such as the powershell on Windows.
+It also attempts to include ideas gained from projects such
+as Avisynth/VirtualDub, which was a really useful project when it
+came to <b>applying filters</b> to image, video and audio files.
+
+The actual implementation of the project may be split up into separate
+projects, but will be unified within the Universal Pipe Handler (UPH).
+
+The general input-output follows this simple schematic:
+
+<img src="https://i.imgur.com/Mp9y6ZY.png" style="margin: 1em">
+
+## How to require the project
+
+Use this to require the project:
+
+    require 'universal_pipe_handler'
+
+## Usage examples
+
+This subsection will try to show various usage examples that should be
+guaranteed to work, provided that you have installed various other
+projects (gems). For instance, for multimedia-related actions you
+will require the gem called <b>multimedia_paradise</b>.
+
+Extract audio from a .mp4 file:
+
+    foobar.mp4 | extract_audio
+    foobar.mp4 | extract audio # both variants must work
+
+Extract audio from a .mp4 file and convert it into a .wav file:
+
+    foobar.mp4 | extract audio | convert into .wav file
+    foobar.mp4 | extract audio | convert_into_wav_file
+    foobar.mp4 | extract audio | to_wav
+    foobar.mp4 | extract audio | towav
+    foobar.mp4 | extract audio | 2wav
+    foobar.mp4 | extract_audio | 2wav
+
+Camelcase all files in the current working directory:
+
+    ls | camel_case
+    ls | camel case
+    ls | camelCase
+    ls | camel
+
+## The unimportant history of this project
+
+I initially started this project back in <b>2007</b> or so, under another name -
+the Hermes pipe. The idea here was that Hermes, from the Greek mythology,
+would deliver messages to others. In German the name is "Götterbote".
+
+After the Hermes pipe I created the Solar pipe, then the Master Pipe,
+then PipeHandler and eventually renamed the project to
+**PipeParadise**. Then it was idle for about 5 years or so, until
+<b>June 2022</b>. Quite a lot of history here, eh?
+
+Most of my more successful projects end up with the string
+*paradise*, such as **ftp_paradise** and so forth.
+
+I will most likely stick to the name <b>universal pipe handler</b>,
+but who knows really. I thought I'd stick with **PipeParadise** as
+a name as well, largely because many of my other ruby-based
+projects have the suffix *paradise*, but then I changed my
+mind again. So who knows.
+
+Why so many rewrites, aside from the various name changes, though?
+
+The Hermes pipe ended up being way too hard to maintain, so I
+thought about *simplifying** it and using a better, simpler concept,
+including a full specification of the behaviour. This part is
+very important: I wanted to **document what is available**, so
+that if I need to rewrite anything in the future again, one day, I
+can just refer to that specification and continue from there.
+
+Thus, this is one goal of the universal pipe handler: to really
+specify what it can do, at all times.
+
+## Current status of the Universal Pipe Handler project
+
+The current status of the Universal Pipe Handler project is not
+complete at all, but it is still better than the prior status
+quo, which had no specification at all whatsoever.
+
+## Definition and Specification for the Universal Pipe Handler
+
+This subsection will contain some definitions and specifications
+for the universal pipe handler project.
+
+The definition of a **cmdlet**, short for <b>Commandlet</b>. An
+alternative name for a <b>cmdlet</b> would be a <b>command
+snippet</b>.
+
+A **cmdlet**, as far as the universal pipe handler project is
+concerned, is everything before and after a pipe token
+(the <b>|</b> token). The pipe must be able to interprete
+these cmdlets.
+
+If you look at the above example again:
+
+    foobar.mp4 | extract_audio | to_wav
+
+Then you can see two | pipe tokens, and a total of three
+individual **cmdlets**.
+
+Let's look at another example:
+
+    ls | nl | remove comments from mp3 files
+
+This would list all entries from the correct working directory,
+number them (from 1 to n), then remove all comments from mp3
+files found, and finally display the result to the user.
+
+Thus, in this case you have three individual <b>cmdlets</b>:
+
+    ls
+    nl
+    remove comments from mp3 files
+
+So everything before and after any individual '|' token
+is an individual "cmdlet". The primary objective for
+the universal pipe handler is to interprete and evaluate
+the cmdlet correctly.
+
+We also have to store data that has to be exchanged between
+the individual command-snippets. This data will be
+stored in a toplevel variable called @dataset.
+
+If there is an unknown instruction given, that is, an unknown
+cmdlet, then this must be reported to the user. Then the pipe
+will either exit (and thus terminate) or continue nonetheless,
+depending on the behaviour used for this pipe instruction.
+
+## Individual cmdlets described
+
+This subsection will describe individual cmdlets. Ideally
+**every** cmdlet should be described, so that we can
+evaluate the behaviour and check it against the main
+specification.
+
+Note that each individual cmdlet **must** be registered
+in the Array that is stored in the file called
+<b>allowed_cmdlets.yml</b>.
+
+Keep the following list sorted alphabetically.
+
+<b>add_audio</b>: This cmdlet attempts to add an audio to a
+video file. It also supports shortcuts via **BeautifulUrl**.
+
+<b>all</b>: This is a general wrapper towards doing an action on "all"
+files or directories. It allows you to pattern-match towards a
+specific pattern. For instance, if you do "all rb files" then
+we will fetch all .rb files of a given directory.
+
+<b>all_images_from</b>: this cmdlet will obtain all images from
+a subdirectory. It also responds to :symbols, such as in the
+following example: <b>all_images_from :njoy_dir</b>. The symbols
+specify from where something should be read.
+
+<b>any</b>: This is a commandlet for convenience. For instance, if you
+do "any avi" then we will fetch any avi file (randomly). This can be
+used in pipes like: "any mp3 | stat" to report the size of that
+file. Not too terribly useful, unless you want to quickly test
+something.
+
+<b>ascii_video</b>: This commandlet allows the user to play a video,
+in mplayer, as ASCII Art, using the <b>aalib library</b>.
+
+<b>assign</b>: this cmdlet allows the user to assign to
+a specific file. Is the default for the first pipe
+processing task, to allow the user to operate with
+existing local files.
+
+<b>copy_directories</b>: will copy all directories found within
+a directory into another target.
+
+<b>colourize</b>: This little cmdlet allows the user to colourize
+string input in any way they see fit. Right now it is a bit
+limited though, you have to extend it as you use that.
+
+<b>crop</b>: Crop can be used to crop (make it smaller) any
+given image or a video, such as from 1024px to 800px.
+
+<b>cut</b>: cut audio ... like cut 30%. This would take from the beginning at
+0% up to 30% of the audio file. We start from the left side by default, which
+is where audio and video files are normally started. If you provide a 
+negative number here, like -30%, then we instead count from 100% to the 
+left, in other words, -30% would mean 70% up to 100%. To recapitulate, if 
+you use -30%, we start from the right side instead. (This allows you to 
+clip away only the last part of an audio file.)
+
+Here are two examples to highlight this:
+   assign foo.mp3 | cut -30%
+     ^^^ cut away the last 30% of the audio file.
+   assign foo.mp3 | cut -30
+     ^^^ cut away the last 30 seconds of the audio file.
+
+Note though that the general action "cut" could also mean to "cut this
+video file", so since December 2011 we assume that cutting a video file
+means that we want to just cut away some parts of it.
+
+You can also use percentages, such as "30%-80%". This will start 
+cutting at 30% and proceed up to 80% of the file length.
+
+<b>decolourize</b>: this cmdlet get rid of colours from a
+<b>video</b> file specifically.
+
+<b>download</b>: Like wget, this allows the user to download
+something. This uses a pure ruby solution, so the user
+is independent of <b>wget</b> actually.
+
+<b>extract</b>: this is a meta instruction. Applied on video
+data, it will try to extract the audio and the video
+separately. Applied to an archive, such as a .zip file,
+it will extract said archive.
+
+<b>extract_all</b>: this cmdlet allows the user to
+extract all packages in a given location.
+
+<b>find_all</b>: this cmdlet allows the user to find a
+substring. <b>Grep</b> is an alias to this, so this is
+mostly a simulation of <b>grep</b>.
+
+<b>generate_string</b>: this cmdlet allows us to generate a string,
+in particular a random string.
+
+<b>get_all_files</b>: This commandlet will obtain all files
+of a given directory. It should equal the UNIX command
+"ls", more or less.
+
+<b>get_last_characters</b>: How many characters we can get from a
+string or a file.
+
+<b>help?</b>: lists all available commands that are supported
+by the <b>universal pipe handler</b>.
+
+<b>identify</b>: This cmdlet can be used to identify a
+multimedia component, be it audio or video related data.
+
+<b>install</b>: We try to install something with this cmdlet.
+This can be used like in "install | htop" to install htop.
+The latter will make use of the <b>rbt</b> project.
+
+<b>match_regex</b>: This cmdlet allows the user to match to
+regexes found in a given file.
+
+<b>n_words</b>: report how many words are in a given file, or
+somewhere else.
+
+<b>play</b>: use this cmdlet to play an audio or a video
+file. Some aliases exist to this, i.e. <b>play_video_file</b>.
+
+<b>random line</b>: fetch a random line from a file. This
+is usually done to show (display) that line.
+
+<b>read_file</b>: This cmdlet will read the content of a file, similar
+to the unix "cat" command. If you use a <b>:</b> then we will use
+BeautifyUrl here. This allows us to access registered files more
+readily using symbols as shortcuts. Specific example for this
+functionality: <b>read_file :pc</b>
+
+<b>read_line</b>: Read a specific line in a file. Like "read_line 9"
+allows you to read the ninth line. (Show line number x). Note that
+the keyword "last" will fetch the last line of a file.
+
+<b>remove_audio</b>: This action attempts to remove the audio found
+in a video container or somewhere else. It will then create a new
+file, for instance called output.avi
+
+<b>remove_comments</b>: this commandlet will remove comments from a
+file. The default comment specifier is the '#' character. Anything
+after that token will be removed, including that token itself.
+
+<b>remove_directories</b>: removes all directories. Use with 
+care. Note that <b>/</b> will never be assumed to be a 
+valid directory for removal via this functionality.
+
+<b>remove_html</b>: This cmdlet will remove all <html> tags found
+in a given String. It can be used to "sanitize" a downloaded
+.html page, for instance, if the user needs the raw text.
+
+<b>remove_newlines</b>: This cmdlet will simply remove newlines.
+In pure ruby code, this would be equal to .delete("\n").
+<b>strip_newlines</b> is an alias to this cmdlet.
+
+<b>repackage_to</b>: allows us to fill in the @result into a
+method and repackage that. A typical example for this may be
+<b>repackage_to .tar.xz</b>.
+
+<b>resize_image</b>: This cmdlet allows the user to resize a given
+image. Right now, resize is an alias to resize_image but in the
+future we may wish to use resize on videos as well.
+
+<b>search</b>: the search-cmdlet is primarily used to search
+for torrents that can be downloaded, since as of June 2011.
+However had, it may be extended eventually towards a more
+comprehensible search-functionality.
+
+<b>shuffle_csv</b>: This commandlet can be used to shuffle CSV 
+values. shuffle_csv 1,5,6 means that the entries at 5 and 6 will
+become right next to 1. It thus reoganizes the entries in a 
+csv file.
+
+Specific Example:
+
+    assign /Depot/j/test.csv | shuffle_csv 1,5,6 | save_as /Depot/j/output.csv
+
+<b>starts_with?</b>: This queries whether a line starts with
+a specific substring or not. If yes, then we add it.
+
+<b>this_dir</b>: This is not a commandlet-action as such,
+but a special alias. It is expanded into "assign Dir.pwd"
+for the most part (or, more accurately, into the method
+return_pwd, which is more or less equal to Dir.pwd, but
+with a trailing <b>/</b>).
+
+<b>to_camel_case</b>: This will camelCase the given input.
+If a list of files is passed then this will be applied
+onto each file. Thus, use with caution.
+
+<b>to_pdf</b>: This cmdlet can be used to create a .pdf
+file.
+
+<b>word_count</b>: This will simply count the word frequency of a
+file or all files in a given directory. In case a directory is
+provided here, the current behaviour is to find all files in
+that given directory and count the words found therein, in all
+of these files.
+
+<b>word_wrap</b>: wraps words after <n characters>. word_wrap
+76 would wrap after 76 characters. Aliases like wrap at
+exist, with wrap at 30 wrapping at every 30th character.
+
+## How to use several cmdlets
+
+You can combine these cmdlets like in the following manner:
+
+    handler = UniversalPipeHandler.new 'assign foo.avi | filter huffyman | save to /Depot/bla.avi'
+
+This will run the passed cmdlets - first assign to foo.avi, then run
+through a <b>huffyman filter</b>, and then save the result of
+that into a new file. The whole line is called a <b>pipe</b>.
+
+Rather than UniversalPipeHandler.new() you could also use the slightly
+shorter UniversalPipeHandler[] method call.
+
+## class UniversalPipeHandler::CmdletsHandler
+
+This class will handle individual cmdlets. Individual cmdlets
+will, in turn, be handled by <b>class UniversalPipeHandler::Cmdlet</b>,
+so <b>class UniversalPipeHandler::CmdletsHandler</b> is the kind of
+cmdlet-"controller".
+
+## Readline support
+
+The universal_pipe_handler should be able to make use of Readline,
+in particular upon completion of all registered cmdlets. This
+should also work with aliases towards cmdlets.
+
+Readline can be used for <b>tab completion</b>. The current
+behaviour is to tab-complete on every available cmdlet
+snippet, including the aliases to it. This should allow the
+user to complete individual instructions towards a cmdlet.
+
+## The variable @result
+
+Data that is processed between two different cmdlets is always
+stored in the toplevel instance variable @result. Access to
+this toplevel instance variable should occur via the method
+<b>.result?</b>.
+
+## Compatibility to Avisynth
+
+Avisynth was a really interesting program that would allow the
+user to do various video-related manipulations in a pipe-oriented
+manner. You could do numerous things with it, including overlaying
+commercial ads on a TV show with some ad-hoc bar that would
+censor it away, programmatically. This focus on using avisynth
+like a **script** was what really was nifty. I am trying to
+add support for this into the **universal pipe handler**
+from the get go, via ffmpeg, but who knows how far I'll get
+with this.
+
+The following subsection shall show some examples how avisynth
+could be used:
+
+    # Add a subtitle
+    AviSource("foobar.avi")
+    Subtitle("Hello World!", size=56, align=5)
+
+    # Flip the video vertically
+    DirectShowSource('foobar.avi')
+    FlipVertical
+
+    # resize the dimensions of the video frame to 320x240
+    LanczosResize(320, 240)
+
+    # fade-in the first 15 frames from black
+    FadeIn(15)
+
+ADD_CONTACT_INFORMATION