j1-template 2022.5.0.rc1 → 2022.5.0.rc4

data/lib/starter_web/pages/public/_jupyter/notebooks/martin_renou_bqplot/Tutorials/.ipynb_checkpoints/Feature_Vector_Distribution-Iris-Digits-checkpoint.ipynb

@@ -0,0 +1,433 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a50e45d2",
+   "metadata": {},
+   "source": [
+    "## Tutorial for building a feature vector distribution plot"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f453461",
+   "metadata": {},
+   "source": [
+    "In this tutorial we will build an interactive widget using bqplot and ipywidgets. bqplot is a powerful interactive plotting library for jupyter. Its main power comes from how well integrated it is into the ipywidgets library. There are a few things you should understand before diving into this tutorial.\n",
+    "\n",
+    "ipywidgets:\n",
+    "Widgets: Widgets python objects which link directly to their html counterpart allowing easy interaction between js,css,html and python.\n",
+    "\n",
+    "Boxes: Boxes allow you to group widgets together, this can be either vertical or horizontally.\n",
+    "\n",
+    "bqplot:\n",
+    "Figures: Figures are a canvas that you will mark on. Its best to think of the figure as another widget (which it is)\n",
+    "\n",
+    "Marks: marks are things that you draw onto the figure, these are composed of a variety of chart types such as bars, lines, histograms etc. You can put a bunch of marks on a single figure.\n",
+    "\n",
+    "If you are used to matplotlib, the paradigm of how axis and scales are used in bqplot can be somewhat counterintuitive at first, so take some time to read the documentation and play around until you understand them. Once you do, they are very powerful when you want to link multiple plots together.\n",
+    "\n",
+    "Axis: Axis describe what the lines around a figure will look like. Only figures have axis, marks don't.\n",
+    "\n",
+    "Scales: The scale describes how ranges should be displayed ie. linear or logarithmic. Scales are used by both axis and marks. Their max and min can auto aujust to the data, or be set. Be careful, you can add an axis to a figure that has a different scale than the one a mark you are adding to the figure has.\n",
+    "\n",
+    "Tooltips: These allow you to add information on hover. They only accept three fields 'name', 'x' and 'y'. So in this tutorial we put all of the infomration we want to show into the name column as a string.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "51c19a0d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipywidgets import HBox, VBox, Dropdown\n",
+    "from bqplot.marks import Scatter, Bars\n",
+    "from bqplot.scales import LinearScale, OrdinalScale\n",
+    "from bqplot.figure import Figure\n",
+    "from bqplot import Tooltip\n",
+    "from bqplot.axes import Axis\n",
+    "import numpy as np\n",
+    "\n",
+    "# simple function to return the bins for the plot\n",
+    "def get_h_bins(df, bins, f_lim):\n",
+    "    if f_lim:\n",
+    "        return np.arange(f_lim['min'], f_lim['max'], (f_lim['max']-f_lim['min'])/float(bins))\n",
+    "    scale_max = int(df.describe().loc['max'].max()+1)\n",
+    "    scale_min = int(df.describe().loc['min'].min()-1)   \n",
+    "    return np.arange(scale_min, scale_max, (scale_max-scale_min)/float(bins))\n",
+    "\n",
+    "def feature_vector_distribution(features, label_column,\n",
+    "                                bins=25,\n",
+    "                                group_columns=None, \n",
+    "                                f_lim=None,\n",
+    "                                colors=None):\n",
+    "    \"\"\"\n",
+    "    features (dataframe): a data frame of feature vectors along with a label column and other metadata\n",
+    "    label_column (str): the name of the column in the features dataframe that refers to the label infomration\n",
+    "    bins (int): the number of bins in the histograms\n",
+    "    group_columns (list): if you want other metadata in the tooltip, these columns will be added\n",
+    "    f_lim (dict): this sets the limits for max and min of the plots to a constant \n",
+    "        {'max':10, 'min':10}. otherwise defaults to the values of the current features \n",
+    "        which can be missleading. \n",
+    "    colors (list): list of colors to use. Internally has a list of 10. If the labels\n",
+    "        are longer you will need to pass your own\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    dist = '640px'\n",
+    "    third_dist = '213px'\n",
+    "\n",
+    "    if f_lim:\n",
+    "        sc_x = LinearScale(min=f_lim['min'], max=f_lim['max'])\n",
+    "        sc_y = LinearScale(min=f_lim['min'], max=f_lim['max'])\n",
+    "    else:\n",
+    "        sc_x = LinearScale()\n",
+    "        sc_y = LinearScale()\n",
+    " \n",
+    "\n",
+    "    scale_y = LinearScale(min=0)\n",
+    "    \n",
+    "    x_ord_legend = OrdinalScale()\n",
+    "    y_lin_legend = LinearScale()\n",
+    "    \n",
+    "    if group_columns is None:\n",
+    "        count_column = features.columns[1]\n",
+    "        group_columns = []\n",
+    "    else:\n",
+    "        count_column = group_columns[0]\n",
+    "\n",
+    "    if colors is None:\n",
+    "        colors = [\"#E6B0AA\", \"#C39BD3\", \"#73C6B6\", \"#F7DC6F\", \"#F0B27A\",\n",
+    "                  \"#D0D3D4\", \"#85929E\", \"#6E2C00\", \"#1A5276\", \"#17202A\"]\n",
+    "    box_color = 'black'\n",
+    "    \n",
+    "    feature_x = Dropdown(description='Feature 1')\n",
+    "    feature_y = Dropdown(description='Feature 2')\n",
+    "\n",
+    "    feature_x.options = [x for x in features.columns if x not in [label_column]+group_columns]\n",
+    "    feature_y.options = [x for x in features.columns if x not in [label_column]+group_columns]\n",
+    "\n",
+    "    feature1 = feature_x.options[0]\n",
+    "    feature2 = feature_y.options[1]\n",
+    "    \n",
+    "    feature_y.value = feature2\n",
+    "\n",
+    "    tt = Tooltip(fields=['name'], labels=[', '.join(['index', label_column]+group_columns)])\n",
+    "    \n",
+    "    scatters = []\n",
+    "    hists_y = []\n",
+    "    hists_x = []\n",
+    "\n",
+    "\n",
+    "    h_bins_x = get_h_bins(features[[feature1]], bins, f_lim)\n",
+    "    h_bins_y = get_h_bins(features[[feature2]], bins, f_lim)\n",
+    "\n",
+    "    for index, group in enumerate(features.groupby([label_column])):\n",
+    "        \n",
+    "        # put the label column and any group column data in the tooltip\n",
+    "        names = []\n",
+    "        for row in range(group[1].shape[0]):\n",
+    "            names.append('{},'.format(row)+','.join([str(x) for x in group[1][[label_column]+group_columns].iloc[row].values]))\n",
+    "            \n",
+    "        # create a scatter plot for each group\n",
+    "        scatters.append(Scatter(x=group[1][feature1].values,\n",
+    "                                y=group[1][feature2].values,\n",
+    "                                names=names,\n",
+    "                                display_names=False,\n",
+    "                                default_opacities=[0.5],\n",
+    "                                default_size=30,\n",
+    "                                scales={'x': sc_x, 'y': sc_y}, \n",
+    "                                colors=[colors[index]],\n",
+    "                                tooltip=tt,\n",
+    "                                ))\n",
+    "        \n",
+    "        # create a histograms using a bar chart for each group\n",
+    "        # histogram plot for bqplot does not have enough options (no setting range, no setting orientation)\n",
+    "        h_y, h_x = np.histogram(group[1][feature1].values, bins=h_bins_x)\n",
+    "        hists_x.append(Bars(x=h_x,\n",
+    "                            y=h_y,\n",
+    "                            opacities=[0.3]*bins,\n",
+    "                            scales={'x': sc_x, 'y': scale_y},\n",
+    "                            colors=[colors[index]],\n",
+    "                            orientation='vertical'))\n",
+    "        \n",
+    "        \n",
+    "        h_y, h_x = np.histogram(group[1][feature2].values, bins=h_bins_y)\n",
+    "        hists_y.append(Bars(x=h_x,\n",
+    "                            y=h_y,\n",
+    "                            opacities=[0.3]*bins,\n",
+    "                            scales={'x': sc_x, 'y': scale_y},\n",
+    "                            colors=[colors[index]],\n",
+    "                            orientation='horizontal'))\n",
+    "        \n",
+    "\n",
+    "    # legend will show the names of the labels as well as a total count of each\n",
+    "    legend_bar = Bars(x=features.groupby(label_column).count()[count_column].index,\n",
+    "               y=features.groupby(label_column).count()[count_column].values,\n",
+    "               colors=colors,\n",
+    "               opacities=[0.3]*6,\n",
+    "               scales={'x': x_ord_legend, 'y':y_lin_legend},\n",
+    "               orientation='horizontal')\n",
+    "\n",
+    "    ax_x_legend = Axis(scale=x_ord_legend, \n",
+    "                tick_style={'font-size':24}, \n",
+    "                label='', \n",
+    "                orientation='vertical',\n",
+    "                tick_values=features.groupby(label_column).count()[count_column].index)\n",
+    "\n",
+    "    ax_y_legend = Axis(scale=y_lin_legend,\n",
+    "                       orientation='horizontal',\n",
+    "                       label='Total', \n",
+    "                       color=box_color,\n",
+    "                       num_ticks=4)\n",
+    "\n",
+    "    #these are blank axes that are used to fill in the border for the top and right of the figures\n",
+    "    ax_top = Axis(scale=sc_x, color=box_color, side='top', tick_style={'font-size':0})\n",
+    "    ax_right = Axis(scale=sc_x, color=box_color, side='right', tick_style={'font-size':0})\n",
+    "    ax_left = Axis(scale=sc_x, color=box_color,  side='left', tick_style={'font-size':0})\n",
+    "    ax_bottom = Axis(scale=sc_x, color=box_color, side='bottom', tick_style={'font-size':0})\n",
+    "    ax_top = Axis(scale=sc_x, color=box_color, side='top', num_ticks=0)\n",
+    "    ax_right = Axis(scale=sc_x, color=box_color, side='right', num_ticks=0)\n",
+    "    ax_left = Axis(scale=sc_x, color=box_color,  side='left', num_ticks=0)\n",
+    "    ax_bottom = Axis(scale=sc_x, color=box_color, side='bottom', num_ticks=0)\n",
+    "\n",
+    "    #scatter plot axis\n",
+    "    ax_x = Axis(label=feature1, scale=sc_x, color=box_color)\n",
+    "    ax_y = Axis(label=feature2, scale=sc_y, orientation='vertical', color=box_color)\n",
+    "\n",
+    "    #count column of histogram\n",
+    "    ax_count_vert  = Axis(label='', scale=scale_y, orientation='vertical', color=box_color, num_ticks=5)\n",
+    "    ax_count_horiz = Axis(label='', scale=scale_y, orientation='horizontal', color=box_color, num_ticks=5)\n",
+    "\n",
+    "    #histogram bin axis\n",
+    "    ax_hist_x = Axis(label='', scale=sc_x, orientation='vertical', color=box_color)\n",
+    "    ax_hist_y = Axis(label='', scale=sc_x, orientation='horizontal', color=box_color)\n",
+    "\n",
+    "    #create figures for each plot\n",
+    "    f_scatter = Figure(axes=[ax_x, ax_y, ax_top, ax_right],\n",
+    "                       background_style={'fill':'white'}, #css is inserted directly\n",
+    "                       marks=scatters, \n",
+    "                       min_aspect_ratio=1,\n",
+    "                       max_aspect_ratio=1,\n",
+    "                       fig_margin =  {\"top\":0, \"bottom\":60, \"left\":60, \"right\":0},\n",
+    "                       )\n",
+    "\n",
+    "    f_hists_y = Figure(axes=[ax_left, ax_count_horiz, ax_top, ax_right], \n",
+    "                       background_style={'fill':'white'},\n",
+    "                       marks=hists_y, \n",
+    "                       min_aspect_ratio=.33,\n",
+    "                       max_aspect_ratio=.33,\n",
+    "                       fig_margin =  {\"top\":0, \"bottom\":60, \"left\":10, \"right\":0},\n",
+    "                      )\n",
+    "\n",
+    "    f_hists_x = Figure(axes=[ax_count_vert, ax_bottom,  ax_top, ax_right],\n",
+    "                       background_style={'fill':'white'}, \n",
+    "                       marks=hists_x,\n",
+    "                       min_aspect_ratio=3, \n",
+    "                       max_aspect_ratio=3,\n",
+    "                       fig_margin =  {\"top\":20, \"bottom\":10, \"left\":60, \"right\":0},\n",
+    "                      )\n",
+    "\n",
+    "    f_legend = Figure(marks=[legend_bar], axes=[ax_x_legend, ax_y_legend], title='',\n",
+    "                                 legend_location ='bottom-right',\n",
+    "                                 background_style = {'fill':'white'},\n",
+    "                                 min_aspect_ratio=1,\n",
+    "                                 max_aspect_ratio=1,\n",
+    "                                 fig_margin =  {\"top\":10, \"bottom\":30, \"left\":20, \"right\":20})\n",
+    "\n",
+    "\n",
+    "    # we already set the ratios, but it is necessary to set the size explicitly anyway\n",
+    "    # this is kind of cool, inserts this into the style in html\n",
+    "    f_legend.layout.height = third_dist\n",
+    "    f_legend.layout.width = third_dist\n",
+    "    f_hists_x.layout.height = third_dist\n",
+    "    f_hists_x.layout.width = dist\n",
+    "    f_hists_y.layout.height = dist\n",
+    "    f_hists_y.layout.width = third_dist\n",
+    "    f_scatter.layout.height = dist\n",
+    "    f_scatter.layout.width = dist\n",
+    "\n",
+    "    # we create some functions that allow changes when the widgets notice an event\n",
+    "    def change_x_feature(b):\n",
+    "        h_bins_x = get_h_bins(features[[feature_x.value]], bins, f_lim)\n",
+    "        for index, group in enumerate(features.groupby([label_column])):\n",
+    "            scatters[index].x = group[1][feature_x.value]\n",
+    "            h_y, h_x = np.histogram(group[1][feature_x.value].values, bins=h_bins_x)\n",
+    "            hists_x[index].y = h_y\n",
+    "\n",
+    "        ax_x.label = feature_x.value\n",
+    "\n",
+    "    def change_y_feature(b):\n",
+    "        h_bins_y = get_h_bins(features[[feature_y.value]], bins, f_lim)\n",
+    "        for index, group in enumerate(features.groupby([label_column])):\n",
+    "            scatters[index].y = group[1][feature_y.value]\n",
+    "            h_y, h_x = np.histogram(group[1][feature_y.value].values, bins=h_bins_y)\n",
+    "            hists_y[index].y = h_y\n",
+    "\n",
+    "        ax_y.label = feature_y.value\n",
+    "\n",
+    "    # when the user selects a different feature, switch the data plotted\n",
+    "    feature_x.observe(change_x_feature, 'value')\n",
+    "    feature_y.observe(change_y_feature, 'value')\n",
+    "\n",
+    "    #return the stacked figures to be plotted\n",
+    "    return VBox([ HBox([feature_x, feature_y]),\n",
+    "           HBox([f_hists_x, f_legend]), \n",
+    "           HBox([f_scatter, f_hists_y])])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d06ec662",
+   "metadata": {},
+   "source": [
+    "## Iris Data Set\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2b525a44",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from sklearn.datasets import load_iris\n",
+    "from sklearn.decomposition import PCA\n",
+    "from sklearn.preprocessing import scale\n",
+    "\n",
+    "np.random.seed(42)\n",
+    "\n",
+    "digits = load_iris()\n",
+    "data = scale(digits.data)\n",
+    "n_features=4\n",
+    "#n_pca=3\n",
+    "#pca = PCA(n_components=n_pca).fit(data)\n",
+    "df = pd.DataFrame(data, columns=['feature_{}'.format(x) for x in range(n_features)])\n",
+    "df['leaf'] = digits.target\n",
+    "df['extra_info'] = [np.random.randint(100) for x in range(digits.target.shape[0])]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "5624f9b0",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "4517af95422f447eaafe0665f0d086c6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(HBox(children=(Dropdown(description='Feature 1', options=('feature_0', 'feature_1', 'feature_2'…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "feature_vector_distribution(df, 'leaf',\n",
+    "                                group_columns=['extra_info'],\n",
+    "                                bins=25,\n",
+    "                                f_lim = {'min':-3, 'max':3}\n",
+    "                                )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b37ea500",
+   "metadata": {},
+   "source": [
+    "## Digits data set, with PCA applied to reduce to 10 features\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "57a07ebf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from sklearn.datasets import load_digits\n",
+    "from sklearn.decomposition import PCA\n",
+    "from sklearn.preprocessing import scale\n",
+    "\n",
+    "np.random.seed(42)\n",
+    "\n",
+    "digits = load_digits()\n",
+    "data = scale(digits.data)\n",
+    "\n",
+    "n_pca=10\n",
+    "pca = PCA(n_components=n_pca).fit(data)\n",
+    "df = pd.DataFrame(pca.transform(data), columns=['pca_{}'.format(x) for x in range(n_pca)])\n",
+    "df['digit'] = digits.target\n",
+    "df['test'] = [np.random.randint(100) for x in range(digits.target.shape[0])]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "b5f20470",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a289c2d46b044381b42998cefdf32a2e",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(HBox(children=(Dropdown(description='Feature 1', options=('pca_0', 'pca_1', 'pca_2', 'pca_3', '…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "feature_vector_distribution(df, 'digit',\n",
+    "                                group_columns=['test'],\n",
+    "                                bins=20,\n",
+    "                                f_lim = {'min':-7, 'max':7}\n",
+    "                                )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d55546c2",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}