red_amber 0.5.0 → 0.5.2

data/doc/Dev_Containers.md

@@ -0,0 +1,292 @@
+# How to use Development Containers in RedAmber
+
+We support [Development Container](https://containers.dev/) in this repository.
+You can prepare a container as a full-featured development environment for RedAmber. Dev Containers allow you to encapsulate Ruby, Apache Arrow, RedAmber with source tree, GitHub CLI, sample datasets and Jupyter Lab with IRuby kernel. You don't need to worry about the change of your local environment.
+
+`.devcontainer` directory in this repository includes settings of Dev Container for RedAmber. We don't use Dockerfile here, based on Ubuntu image for Dev Container, Python and GitHub CLI tools using Dev Container Features. I think this style has simplicity, maintainability, and reusability. Ruby is added after the container is created by script.
+
+It has some benefits below compared to make dev environment by Dockerfile;
+
+1) It automatically makes user setting with same UID/GID as local user.
+2) Additional tools can be introduced by `Dev Container Features`.
+3) Ruby Feature includes `rbenv` and it is easy to add another version afterwards.
+4) Python Feature includes Jupyter Lab as an option.
+5) Quarto is introduced. It converts Jupyter notebook from/to qmd file and it is useful to manage notebook in the source tree.
+
+We will show 2 examples here.
+
+## 1. GitHub Codespace in cloud from browser
+
+### Prerequisite
+
+You need to sign in GitHub Account.
+
+### Notice
+
+You will consume your Codespace quota of your account in the step below. For GitHub Free, you can use 120 hours per month per core (it means 60 hours per month for 2 cores VM) and have a storage with 15 GB per month for free. You can check your usage from `Codespaces` section in [Billing and plans](https://github.com/settings/billing).
+
+### Procedures
+- Open the [repository of RedAmber](https://github.com/red-data-tools/red_amber) in GitHub.
+  - If you are going to develop RedAmber, you should fork it and open your forked repository to push PR later.
+
+- Push `<>Code` button, select `Codespaces` tab, push `Create codespace on main` button to create new Codespace.
+  * You can re-connect existing Codespace if you already have there.
+- Creating Codespace takes time. You can click `View log` to browse log running or have a coffee to wait.
+  * I am planning to improve building process to save cache in GitHub Container Registory.
+
+- VS Code for browser will open the repository in the remote container.
+
+  Please refer [Operations](#operations) to use the environment.
+
+### Details
+Please see [(GitHub Docs)Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository) for detail.
+
+## 2. Start Dev Container from the local repository, and use it from VS Code
+
+### Prerequisites
+- Visual Studio Code (October 2020 Release 1.51+)
+
+  You need to install GitHub Codespaces extention, and sign into GitHub Codespaces  with your GitHub credentials. Please see [GitHub Docs - GitHub Codespaces - Prerequisites](https://docs.github.com/en/codespaces/developing-in-codespaces/using-github-codespaces-in-visual-studio-code#prerequisites) and prepare settings.
+
+- Docker
+  - Windows
+
+    In Windows 10 Pro/Enterprise, Docker Desktop 2.0+
+    In Windows 10 Home (2004+), Docker Desktop 2.3+ and WSL 2 backend.
+
+  - Mac
+
+    Docker Desktop 2.0+
+
+  - Linux
+
+    Docker CE/EE 18.06+ and Docker Compose 1.21+
+
+- Git
+
+### Procedures
+
+- Create a local clone of RedAmber repository.
+
+  - If you are going to develop RedAmber, make a fork and clone it.
+
+  ```
+  $ git clone https://github.com/(red-data-tools or your account name)/red_amber.git
+  ```
+
+  Alternatively using GitHub CLI,
+
+  ```
+  $ gh repo clone (red-data-tools or your account name)/red_amber
+  ```
+
+- Open local repo folder by VS Code.
+
+  ```
+  $ code red_amber
+  ```
+
+- Re-open by container
+
+  Re-open current folder by container.
+
+  - Click remote host indicator in the left bottom corner, then options to open remote windows will open. Choose 'reopen by container'.
+
+- Building of container will start.
+
+  It takes time for first building. If it is finished, container name will be displayed on the remote host indicator.
+
+## Operations
+
+### Check installed tools by terminal
+
+  If you don't have the terminal open, open it with ``CTRL + ` ``.
+
+  Run these command to check these tools are installed.
+
+  ```shell
+  $ ruby -v --jit
+  $ rbenv versions
+  $ gem -v
+  $ gem list
+  $ bundler -v
+  $ iruby -v
+
+  $ python --version
+  $ pip --version
+  $ pip list
+  $ pipenv --version
+  $ jupyter --version
+  $ jupyter kenelspec list
+
+  $ git -v
+  $ git config user.name
+  $ gh --version
+  ```
+
+  The user name is `vscode` in this environment. `uid` and `gid` are the same as local user.
+
+  ```shell
+  $ id
+  ```
+
+### Run tests of RedAmber
+
+  ```shell
+  $ bundle exec rake
+  ```
+
+### Try RedAmber in REPL
+
+  You can try RedAmber in `irb` using pre loaded datasets. It takes time in the first run to load the datasets from Red Datasets.
+
+  ```ruby
+  $ rake example
+
+  (snip)
+
+      81: # Welcome to RedAmber example!
+      82: # This environment will offer these pre-loaded datasets:
+      83: #   penguins, diamonds, iris, starwars, simpsons_paradox_covid,
+      84: #   mtcars, band_members, band_instruments, band_instruments2
+      85: #   import_cars, comecome, rubykaigi, dataframe, subframes
+   => 86: binding.irb
+
+  irb(main):001:0>
+  ```
+
+  This code stops in the code by `binding.irb`, you have some datasets in local variables.
+
+  ```ruby
+  irb(main):001:0> import_cars
+  =>
+  #<RedAmber::DataFrame : 5 x 6 Vectors, 0x0000000000010914>
+       Year    Audi     BMW BMW_MINI Mercedes-Benz      VW
+    <int64> <int64> <int64>  <int64>       <int64> <int64>
+  0    2017   28336   52527    25427         68221   49040
+  1    2018   26473   50982    25984         67554   51961
+  2    2019   24222   46814    23813         66553   46794
+  3    2020   22304   35712    20196         57041   36576
+  4    2021   22535   35905    18211         51722   35215
+  ```
+
+  The namespace `RedAmber` is included.
+
+  ```ruby
+  irb(main):002:0> VERSION
+  => "0.5.0"
+  irb(main):003:0> Arrow::VERSION
+  => "12.0.1"
+  ```
+
+  You can return to the first breakinng point by hitting `@`.
+
+  You can exit irb by `exit`.
+
+### Try RedAmber in Jupyter Lab
+
+  You can try Jupyter Lab with Python and IRuby kernels in your browser.
+
+  ```shell
+  $ rake jupyter
+  ```
+
+  - `doc/notebook` is allocated as notebook folder. There are 2 files in it.
+    - `red_amber.ipynb` : Examples in `README.md`.
+    - `examples_of_red_amber.ipynb` : Hundreds examples of RedAmber.
+  - `require 'red_amber'` will load from source directory `lib`.
+
+## Document authoring by Quarto
+
+[Quarto](https://quarto.org/) is an open-source scientific and technical publishing system.
+We use Quarto CLI to show usage examples of RedAmber.
+
+```mermaid
+---
+title: Document management with Quarto
+---
+flowchart LR
+    id1["Source management
+        (.qmd)"]
+    id2["Analyze and edit by JupyterLab
+        (.ipynb)"]
+    id3["Publish document
+        (.pdf)"]
+
+    id1 -- convert --> id2 -- convert -->  id1
+    id2 -- render --> id3
+    id1 -- render --> id3
+```
+
+* We can manage the source of the Jupyter notebook by Quarto's markdown format `qmd`.
+* We can convert a `.qmd` file to a Jupyter notebook file (`.ipynb`) and will be able to edit it or make analysis on Jupyter Lab.
+* We can render `.qmd` or `.ipynb` files to `.pdf`.
+
+### To show the information of Quarto
+
+Try below to show version and verify correct functioning of Quarto installation.
+
+```shell
+$ quarto -v
+$ quarto check
+```
+
+To show help,
+
+```shell
+$ quarto --help
+$ quarto render --help
+```
+
+### Convert qmd file to Jupyter Notebook
+
+To convert `.qmd` source file to `.ipynb`,
+
+```shell
+$ bundle exec rake quarto:convert
+```
+
+This command will create `ipynb` notebooks from `doc/qmd` and save them to `doc/notebook`.
+
+In more general,
+
+```shell
+$ quarto convert source_file.qmd
+$ quarto convert source_file.qmd --output Notebook.ipynb
+```
+
+The first one will output `source_file.ipynb` file in the same directory.
+
+Command below will convert qmd_document files in `doc/qmd` to `.ipynb` files and save them to `doc/notebook` . Then open `doc/notebook` with Jupyter Lab.
+
+```shell
+$ bin/jupyter
+```
+
+### Convert Jupyter Notebook to `qmd`
+
+You can convert Notebook file to qmd file.
+
+```shell
+$ quarto convert notebook.ipynb
+$ quarto convert notebook.ipynb --output output_source_file.qmd
+```
+
+### Others
+
+To render Notebook files in `doc/qmd` to pdf,
+
+```shell
+$ bundle exec rake quarto:test
+```
+To clear `doc/notebook` (and all the generated artifacts by rake),
+
+```shell
+$ rake clean
+```
+
+To know more about Quarto, see command line help by `quarto --help`, or visit [Quarto](https://quarto.org/) web.
+
+### Thanks
+
+ As for the use of Quarto, I started to try after the Kozo Nishida's work with Ruby Association Grant 2022 "Introducing Quarto into the RubyData ecosystem and promoting the combination to the Ruby community". I would like to take this opportunity to thank him.