j1_template_mde 2018.4.10

data/lib/starter_web/collections/posts/series/_posts/000_includes/documents/100-docker-using-shared-folders.asciidoc

@@ -0,0 +1,415 @@
+[[readmore]]
+== Docker smooth on Windows
+
+What the Docker team recomments: Update to the newer package, if possible.
+This should be discussed, especially for Windows, in more detail ...
+
+[WARNING]
+====
+*The Docker Team mentions the following about Docker Toolbox:*
+
+Docker Toolbox is for *older Windows* systems (or *Apple MacOS* versions)
+that *does not meet* the requirements of the *native* Docker version.
+====
+
+Docker won't never run *natively* on Windows. The *Container Technology*
+used by Docker depends on *Linux* (at least on a Unix flavour of the host
+operating system). For all non-Linux (or Unix) platforms, a *Hypervisor* is
+needed to make it possible to run the needed *Linux OS* for Docker.
+
+Yes, Windows support a *native* Hypervisor layer: `Hyper-V`. Using the Windows
+component `Hyper-V`, any operating system can be run as *virtual machines* on
+Windows. But it should be kept in mind: the Hypervisor layer is *always* needed;
+as well as for the so-called the *native* Docker version on Windows. But:
+real *native* Docker versions does *not* need a Hypervisor layer; they run
+containers direcly on the host OS! This should be kept on mind.
+
+They are some pittfals using *Docker for Windows* if you go for:
+
+* Stability - found quite often issues running containers on *Docker for Windows*.
+If you're  not a Windows specialist, it is hard to findout issues for `Hyper-V`
+, the VM and|or both.
+* Flexibility - it is *NOT* possible to use the Hypervisor `Hyper-V` in
+parallel to *VirtualBox* as an alternative
+* Incompatibility - on some Windows installations, and for sure they met the
+requirements of the *native* Docker version, the installaton was *NOT*
+successful and Docker fails to start
+
+Again: the software included in the *Docker Toolbox* are *current* versions.
+The latest Toolbox contains:
+
+* the Docker Engine Docker, version 18.03.0-ce, build 0520e24302
+* the Docker Machine, version 0.14.0, build 89b8332
+* the Docker Composer, version 1.20.1, build 5d8c71b2
+* a current MINGW64-based GNU bash, version 4.4.19(2)-release (x86_64-pc-msys)
+
+For all the people having their (IT) roots on Unix respectively Linux, the
+*Docker Toolbox* is the better solution on Windows (for now). No need to go
+for the Windows Hypervisor `Hyper-V` - Oracle *VirtualBox* can be used
+instead; as quite often used for testing and development.
+
+Let's have a look how to *Manage Linux on Windows*, to use Docker on the
+Windows platform.
+
+== Installing Docker Toolbox
+
+As discussed ealier, *Docker Toolbox* is using *VirtualBox* for the
+Hypervisor to create the Linux VMs needed. To get the latest version,
+install VirtualBox first (Docker Toolbox includes a installer package
+for VBox but not nessesarily at their latets version).
+
+Go for the link:{virtualbox-install-on-windows}[VirtualBox Download Page]
+and select under *VirtualBox 5.x.y platform packages* the latest version
+for Windows (Windows hosts). Install the package - the installation process
+is quite simple supported by an build in installer wizard.
+
+If done, go for the installation of the Toolbox. A current package can be
+retrieved from link:{docker-install-toolbox-on-windows}[Docker Documents].
+The page includes all the info needed to install the software.
+
+NOTE: For the installation of the *Docker Toolbox*, please deselect the
+installation of *VirtualBox* included as the software has been already
+installed.
+
+NOTE: The commandline interface used by *Docker Toolbox* is based on
+*Git MSYS-git UNIX tools*. If you have *Git* installed already for your
+Windows host, please deselect the installation of *Git* included.
+
+*Docker Toolbox* comes with *NO* graphical user interface (GUI). To manage
+Docker, the included GNU bash of the *Git MSYS-git UNIX tools* shoud be used.
+A very first version of a (native) Docker UI on Windows is *Kitematic*,
+currently in a alpha version included.
+
+ifdef::backend-html5[]
+.Docker UI Kinematic on Windows
+lightbox::docker-windows-kinematic-alpha[ 800, {docker-windows-kinematic-alpha-data} ]
+endif::[]
+
+TIP: You should check *Kitematic*. If started for *Docker Toolbox* on Windows,
+change (the Hypervisor) to Virtualbox.
+
+For Windows users a commandline interface to manage a complex system like
+Docker may sound a bit stone age. It's not for good reasons:
+
+* the knowlegde of the base Docker commands is essential to understand the
+Container Technology better
+
+* Blalal blaaa
+
+
+
+== Manage Linux on Windows
+
+VirtualBox is a greate piece of software for desktop and workstation
+virtualization. A huge number of IT specialist and developers are using VB
+for: testing and development environments. Sometimes for production
+environments – if it's applicable.
+
+One good idea for VirtualBox is, if you are Web|App developer for Linux you
+can keep *Windows* as your primary OS - yeah, yeah those people are living on
+our planet! One can install a Linux distribution of choice in VirtualBox VM
+and reap both benefits of Windows as primary OS and Linux as a server
+platform for good reasons.
+
+=== Create a Linux VM
+
+lorem:sentences[5]
+
+lorem:sentences[3]
+
+
+=== Sharing Data
+
+For (real) native Docker installations, it is easy to share data between then
+host OS and containers managed by Docker. It is simple this:
+
+[source, sh]
+----
+docker run --rm \
+  --volume=$PWD:/srv/jekyll \
+  -p 35729:35729 -p 40000:40000 \
+  -it jekyll-one/j1:latest \
+  j1 serve --incremental --livereload
+----
+
+The container, instructed by `--volume=$PWD:/srv/jekyll` shares the
+filesystem of the host - the *native* way to bring in data into a
+container for processing. In the example a static Jekyll Web is created
+by J1 Template (j1), served by the ports 35729 for livereload and 40000
+serving the web.
+
+For the Windows platform, using a virtual Linux Box for Docker containers,
+this is *NOT* possible. The virtual system is managed by Virtualbox and no
+direct relationship is in between the VBox and the host operating system
+Windows! No (direct) access to the host filesystem is possible. Bad luck so
+far.
+
+Generally spoken, there are 3 ways of sharing files between a Windows host
+and Linux guest boxes; keep files:
+
+* in the guest (Linux) and share them with host through Samba
+* on host (Windows) and use VirtualBox *Shared Folders*
+* on host (Windows) and mount Windows *Public Folders* via CIFS
+from within guest (Linux)
+
+For all people experienced on the Linux platform, the first option is
+probably the easiest. But it has it’s drawbacks. In most cases for testing,
+it's *NOT* wanted to keep files in a VM. In case of corruption of the VM,
+reinstallation needs, or whatever yor data is *lost* in most cases. Truly,
+not an option.
+
+Typically developers want them kept on native Windows *disks* as they are
+expected as more safely stored on physical devices and easier to backup.
+So first option is clearly out.
+
+*Shared Folders* are really a great idea. After installing the *VirtualBox
+Guest Additions* (which brings the mount.vboxsf binary and vboxsf kernel
+module) the guest can mount automatically a Windows OS folder inside the VM.
+Some klicks further the guest start's, mount the *shared folder* in guest and
+voila: you're done! Sounds really great.
+
+But there's a catch – performance. Shared Folders are really, really horrible
+slow. Which in turn kills the idea of programming and testing code without
+loosing hair because of the slowness. You can find tons of articles around
+slow shared folders on the Internet; both for the Big Dogs: VMware and
+VirtualBox. But no solution. Since decades.
+
+Analyzing the performance, it's obvious that kernel spends really large
+amount of time in IO mode – so the vboxsf CIFS emulation of VirtualBoxis
+behaving something like a slug. Birds love slugs but IT professionals don’t
+like that, do they?
+
+It seems that the only viable solution is sharing folders though Windows and
+mounting them via CIFS on the guest, the virtual Linux box. This way, a
+similiar flexible solution for sharing folders can be achived but at a much
+better performance.
+
+
+=== Performance tests
+
+Anyway, *shared folders* will be always slow as the are of type *NAS* - network
+attached storage. For a local network nowadays, typically at Gigabit level and
+quite low in latency, an acceptable performance cab be achived.
+
+I did some simple tests to show the performance differences. There are three
+directories I did for my tests:
+
+* /mnt/media/repo1, data on a physical (local) disk on the host
+* /mnt/media/repo2, data on a physical (local) disk in the guest
+* /mnt/media/repo3, data on a Windows share mounted via cifs in the guest
+
+I've been using simple (bash) script based on *dd* commands for testing.
+The *dd* commands used as a base are shown below. For sure not a quite
+sophisticated test scenario but it's giving a general impression what can
+achived in real world.
+
+NOTE: For multi-platform testing, on Windows the `dd` command is used as well.
+For *Docker Toolbox*, beside the GNU bash, a full set of *GNU CoreUtils* are
+available as well (accessible out of the *Docker Quickstart Terminal*). The
+used (bash) test script needs to be run out of *Docker Quickstart Terminal*.
+Download the tester script from here.
+
+The test script *docker_perf_test.sh* supports a test scenario for
+*General System Performance*. This is focussing on pure CPU|Memory power.
+On both systems (host and guest) these tests are using pure *kernel services*
+for creating input and output data. One is creating random input data as
+fast as possible, the other discards everything it gets as fast as it can.
+
+.General System Performance
+[source, sh]
+----
+dd if=/dev/urandom of=/dev/null bs=1M count=10000
+----
+
+Please see those tests for *General System Performance* for the background
+*What could be achived in therory*. These tests may helpful to compare your
+current setup for the host and the guest same way.
+
+As mentioned, all tests for *docker_perf_test.sh* are based on the `dd`
+command. For *real world* measures, standard *dd tests* are used. You'll
+find a lot of those on the Internet. All tests for *docker_perf_test.sh* are
+changed (from `/dev/zero`) to `/dev/urandom` for the input to put some
+*pressure* on the CPU while processing data for reading or writing from the
+filesystem to be checked. All the tests are *rules of thumb*, no technical
+measures.
+
+NOTE: Zum Messen der Schreibperformance liest man die zu Schreibenden
+Daten am besten aus /dev/zero und schreibt eine normale Datei im Filesystem
+(z.B. mit of=/root/testfile). Die folgenden Beispiele verwenden ein Testdatei,
+um irrtümlichen Datenverlust zu vermeiden. Die dabei erzielbare Performance
+Werte sind etwas geringer (da dadurch auch Metadaten im Filesystem geschrieben
+werden).
+
+Find the base `dd` commands for read an write performance tests below. The
+exported shared folder on Windows is *C:\Users\Public* mapped to the Unix
+(POSIX) folder scheme of */c/Users/Public* on the guest OS.
+
+.Write Performance - Windows host (local disk), Linux guest (shared disk)
+[source, sh]
+----
+dd if=/dev/urandom bs=1M count=1024 | split -b 1M - name.
+----
+.Write Performance (local disk), Linux guest
+[source, sh]
+----
+dd if=/dev/urandom of=/var/tmp/test/samplefile bs=1M count=1024
+----
+
+.Read Performance - Windows host (local disk), Linux guest (shared disk)
+[source, sh]
+----
+dd if=/c/Users/Public/test/samplefile of=/dev/null bs=1M count=1024 iflag=direct
+----
+.Read Performance (local disk), Linux guest
+[source, sh]
+----
+dd if=/var/tmp/test/samplefile of=/dev/null bs=1M count=1024 iflag=direct
+----
+
+Have a look at the table below for the measures found on a *Windows 10*
+desktop host (Intel Core Quad Q9550) using a *SATA-300* disk for the local
+physical disk and a Gigabit network interface (*Realtek PCIe GBE* familiy)
+running a *CentOS 7 Linux* (version 3.10.0-862) as a guest:
+
+.Data transfer rates for 1 GB of data
+[cols="2,2,4,3, options="header", role="table-responsive, full-width"]
+|===============================================================================
+|direction |mountpoint |source |througput [MB/s]
+
+|`read`
+|/mnt/media/repo1
+|physical (local) disk on the *host*
+|85.20
+
+|`read`
+|/mnt/media/repo2
+|physical (local) disk on the *guest*
+|147.00
+
+|`read`
+|/mnt/media/repo3
+|shared disk on the *guest*
+|30.3
+
+|`write`
+|/mnt/media/repo1
+|physical (local) disk on the *host*
+|234.00
+
+|`write`
+|/mnt/media/repo2
+|physical (local) disk on the *guest*
+|34.00
+
+|`write`
+|/mnt/media/repo3
+|shared disk on the *guest*
+|39.9
+
+|===============================================================================
+
+WARNING: Bei Verwendung von if=/dev/zero und bs=1G benötigt das Linuxsystem
+1GB freien Platz im RAM. Falls Ihr Testsystem nicht ausreichend RAM ...
+
+NOTE: Um praxisnahe Ergebnisse zu erhalten empfehlen wir die beschrieben Tests
+mehrmals (z.B. 3 bis 10x) durchzuführen. Damit können Sie Ausreißer rechtzeitig
+erkennen.
+
+
+lorem:sentences[5]
+
+== Publish a Shared Folder
+
+lorem:sentences[5]
+
+lorem:sentences[3]
+
+
+== Mount Shared Folders on Linux
+
+lorem:sentences[5]
+
+=== Installing CIFS
+
+For the Linux VM, a current Ubunt Server of version 18.06 is used. To mount
+CIFS shares, the package `cifs-utils` needs to be installed:
+
+[source, bash]
+----
+sudo apt-get -y install cifs-utils
+----
+
+Mounting filesystems in general, therefor the same to the CIFS share, are
+done as root. This ends up in the situation this folder cannot (accessed??)
+written as normal user.
+
+. Create the `cifs` group
++
+[source, bash]
+----
+sudo groupadd cifs
+----
+
+[start=2]
+.  Add your user to the `cifs` group
++
+[source, bash]
+----
+sudo usermod -aG cifs $USER
+----
+
+[start=3]
+.  Create the mountpoint (to be used by Docker) as root
++
+[source, bash]
+----
+mkdir -p /c/Users/Public
+----
+
+[start=4]
+.  Change the ownership of the mountpoint
++
+[source, bash]
+----
+chown -R root:cifs  /c/Users/Public
+----
+
+[start=5]
+.  Create a credentials file to place user|password data safe.
++
+[source, bash]
+----
+touch ~/.smbcredentials
+----
+
+.smbcredentials
+----
+username=<your_windows_user>
+password=<windows_user_password>
+----
+
+=== Mount the share manually
+
+To enable non-authorized users to read and write the shared files, specify
+user and group ID that the mounted shares folders should use. This would
+allow particular users and group members to read|write to the share.
+
+To do so, the following options are used for a public mount:
+
+* username
+* password
+* uid
+* gid
+
+Add the Windows host IP address to the hosts file
+
+./etc/hosts
+<ip_v4_address>  <name_of_the windows_box>  <alias_name>
+
+
+=== Mount the share at boot-time
+
+lorem:sentences[3]
+
+./etc/fstab
+&#47;&#47;win10pc/Public /c/Users/Public cifs uid=1000,gid=5555,credentials=/home/jadams/.smbcredentials,iocharset=utf8,rw  0 0

data/lib/starter_web/collections/posts/series/_posts/000_includes/documents/loop.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+FOLDER=/Users/jadams/test_files/
+TEST_FILES_PATH=${FOLDER}/files/
+FILE=name
+COUNT=3
+SIZE=1M
+SLICE=1M
+TYPE=random
+
+# Do write test of $COUNT files
+$(dd if=/dev/urandom bs=${SIZE} count=${COUNT} 2> ${FILE}.measures | split -b ${SLICE} - ${TEST_FILES_PATH}${FILE}.)
+
+dd_write=$(cat ${FILE}.measures | grep -v records | sed 's/.* s,//')
+echo ${dd_write}
+
+
+FILES=$(find ${TEST_FILES_PATH} -type f -print)
+
+for f in $FILES; do
+  # echo "Processing $f file..."
+  dd_read+=$(dd if=${f} of=/dev/null 2>&1 | grep -v records | sed 's/.* s,//')
+done
+
+echo ${dd_read}
+
+cd ..
+rm -f ${TEST_FILES_PATH}${FILE}\.*

data/lib/starter_web/collections/posts/series/_posts/000_includes/tables/debug_variables.asciidoc

@@ -0,0 +1,47 @@
+ifeval::["{debug}" == "true"] 
+.Liquid variables
+[cols="2,4,6a, options="header", role="table-responsive, full-width"]
+|===============================================================================
+|name |value |description 
+
+|`page.path`
+|{{ page.path }}
+|The path to the raw post or page. Example usage: Linking back to the page 
+or post's source on GitHub. This can be overridden in the `YAML Front Matter`.
+
+|`site.collections_dir`
+|{{ site.collections_dir }}
+|lorem:sentences[1]
+
+|`page.images.dir`
+|{{ page.images.dir }}
+|Image path set pagewise. See FRONTMATTER defaults in `_config.yml` for the 
+images path specified by: `images.dir` for specific page *type*. See the 
+example below for a page of type *pages*.
+[source, yaml]
+----
+defaults:
+  - scope:
+      path:          pages
+      type:          pages
+
+    values:
+      images:
+        dir:        /assets/images/pages
+----
+
+|`page-imagesdir`
+|{{ page-imagesdir }}
+| Images path set in the page by `:page-imagesdir:`. 
+
+|`template`
+|{{ site.template.name }}
+|Name of the template used for rendering a page. Settings are taken from 
+sitewide configuration defined by `_config.yml`.
+
+|`includedir`
+|{{ includedir }}
+|lorem:sentences[1]
+
+|===============================================================================
+endif::[]