data-manipulation-utilities 0.2.6__py3-none-any.whl → 0.2.8.dev714__py3-none-any.whl

{data_manipulation_utilities-0.2.6.dist-info → data_manipulation_utilities-0.2.8.dev714.dist-info}/METADATA

@@ -1,22 +1,10 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: data_manipulation_utilities
-Version: 0.2.6
+Version: 0.2.8.dev714
+Summary: Project storing utilities needed to reduce boilerplate code when analyzing data
 Description-Content-Type: text/markdown
-Requires-Dist: logzero
-Requires-Dist: PyYAML
-Requires-Dist: scipy
-Requires-Dist: awkward
-Requires-Dist: tqdm
-Requires-Dist: joblib
-Requires-Dist: scikit-learn
-Requires-Dist: toml
-Requires-Dist: numpy
-Requires-Dist: matplotlib
-Requires-Dist: mplhep
-Requires-Dist: hist[plot]
-Requires-Dist: pandas
-Provides-Extra: dev
-Requires-Dist: pytest; extra == "dev"
+
+[TOC]
 
 # D(ata) M(anipulation) U(tilities)
 
@@ -51,6 +39,190 @@ Then, for each remote it pushes the tags and the commits.
 
 This section describes generic tools that could not be put in a specific category, but tend to be useful.
 
+## Caching data
+
+In order to reuse data that is hard to calculate one would need:
+
+- Serializable data, i.e. strings, floats, lists, etc
+- A way to get a unique identifier of that data, e.g. a hashable object
+
+If both are avalable, one can:
+
+```python
+import dmu.generic.utilities as gut
+
+def _get_something() -> float:
+    # This loads the data, if found 
+    hashable = arg1, arg2
+
+    ret = gut.load_cached(hash_obj=hashable, on_fail=-999)
+    if ret != -999:
+        return ret
+    
+    obj = very_expensive_function(arg1, arg2)
+    
+    # This saves the data
+    ret = gut.cache_data(obj, hash_obj=hashable)
+
+    return ret
+```
+
+the cached data will go to JSON files in `/tmp/dmu/cache`.
+
+## Caching with a base class
+
+Caching functionalities can be added to a class through a base class as in:
+
+```python
+from dmu.workflow.cache    import Cache     as Wcache
+
+class Tester(Wcache):
+    '''
+    Testing class, produces outputs from simple inputs
+    '''
+    # -----------------------------------
+    def __init__(
+            self,
+            nval : int):
+        '''
+        nval, some integer used to produce output data
+        '''
+        super().__init__(
+                out_path='Tester',
+                nval    =nval)
+
+        self._nval    = nval
+    # -----------------------------------
+    def run(self) -> None:
+        '''
+        Returns a list of 1's
+        '''
+        # _out_path belongs to the base class
+        obj_path = f'{self._out_path}/values.json'
+
+        if self._copy_from_cache():
+            log.warning('Output cached, not running')
+            return gut.load_json(obj_path)
+
+        log.info('Data not cached, running')
+        res = [1] * self._nval
+
+        gut.dump_json(res, obj_path)
+        self._cache()
+
+        return res
+
+# This will set the root directory where cached data goes
+# The data will go to `/some/directory/Tester`
+# This has to be done ONCE and only ONCE.
+Wcache.set_cache_root(root='/some/directory')
+
+obj = Tester(nval=3)
+...
+```
+
+where the tester class has access to extra functionalities to:
+
+- Cache outputs to a hashed directory
+- For the next run, check if the directory exists, if so pick 
+the outputs and put them in the output directory
+- If not rerun the process
+
+Several hashed directories might exist, like in the diagram:
+
+![](doc/images/cache_hash.png)
+
+**Important**: This class will also use the hash of the module where the `Test`
+class is defined. Thus, changes in the code or in the input data, will invalidate the hash.
+
+### Turning caching off
+
+This can be done temporarily with:
+
+```python
+with Wcache.turn_off_cache(val=['Tester']):
+    obj = Tester(nval=4)
+    out = obj.run()
+```
+
+for any list of classes that inherit from `Cache` by passing the list of class names.
+If `val=None` is passed, ALL the classes caching is turned off.
+
+## Silencing import messages
+
+To silence messages given by modules not in the user's control do:
+
+```python
+import dmu.generic.utilities as gut
+
+with gut.silent_import():
+    import tensorflow
+```
+
+## Silencing messages going to __stderr__ originating deep from C++ code
+
+This is an issue with frameworks like `Tensorflow`. Some messages are impossible
+to kill, which interferes with the debugging process. In order hide selectively
+those messages, do:
+
+```python
+from dmu.logging import messages as mes 
+
+l_msg = ['ONE', 'TWO']
+with mes.filter_stderr(banned_substrings=l_msg):
+        os.write(2, b'MSG ONE\n')
+        os.write(2, b'MSG TWO\n')
+        os.write(2, b'MSG THREE\n')
+```
+
+The context manager above will only allow `THREE` into the error stream.
+
+## YAML
+
+When dumping data to yaml files do it like:
+
+```python
+import dmu.generic.utilities as gut
+
+yaml.dump(data, Dumper=gut.BlockStyleDumper)
+```
+
+to make sure the indentation is correct.
+
+## Hashing
+
+### Hashing python objects
+
+The snippet below:
+
+```python
+from dmu.generic  import hashing
+
+obj = [1, 'name', [1, 'sub', 'list'], {'x' : 1}]
+val = hashing.hash_object(obj)
+```
+
+will:
+
+- Make the input object into a JSON string
+- Encode it to utf-8
+- Make a 64 characters hash out of it
+
+in two lines, thus keeping the user's code clean.
+
+### Hashing files
+
+The following snippet:
+
+```python
+from dmu.generic  import hashing
+
+path = '/some/file/path.txt'
+val  = hashing.hash_file(path=obj)
+```
+
+should provide a hash to a file, given its path.
+
 ## Timer
 
 In order to benchmark functions do:
@@ -67,9 +239,9 @@ def fun():
 fun()
 ```
 
-## JSON dumper
+## JSON/YAML dumper and loader
 
-The following lines will dump data (dictionaries, lists, etc) to a JSON file:
+The following lines will dump data (dictionaries, lists, etc) to a JSON/YAML file and load it back:
 
 ```python
 import dmu.generic.utilities as gut
@@ -77,8 +249,48 @@ import dmu.generic.utilities as gut
 data = [1,2,3,4]
 
 gut.dump_json(data, '/tmp/list.json')
+data = gut.load_json('/tmp/list.json')
+```
+
+this will dump to either JSON or YAML files, depending on the extension, extensions allowed are:
+
+```
+.json
+.yaml
+.yml
+```
+
+and it's meant to allow the user to bypass all the boilerplate and keep their code brief.
+
+## PKL dumper and loader
+
+In the same way one can do:
+
+```python
+import dmu.generic.utilities as gut
+
+data = [1,2,3,4]
+
+gut.dump_pickle(data, '/tmp/list.pkl')
+data = gut.load_pickle('/tmp/list.pkl')
+```
+
+## Loader of files and configurations from data packages
+
+YAML and JSON files can be loaded from data packages with:
+
+```python
+import dmu.generic.utilities as gut
+
+data = gut.load_data(package='dmu_data', fpath=f'tests/data.json')
+conf = gut.load_conf(package='dmu_data', fpath=f'tests/config.json')
 ```
 
+the former will return a python dictionary, list, etc. 
+The later will return a `DataConf` object from the `omegaconf` project.
+Check [this](https://omegaconf.readthedocs.io/en/2.3_branch/index.html) 
+for more information.
+
 # Physics
 
 ## Truth matching
@@ -119,8 +331,72 @@ samples:
 
 # Math
 
+## Weighted data
+
+`Wdata` is a small class symbolizing weighted data that contains extra functionality. It can
+be used as:
+
+```python
+from dmu.stats.wdata        import Wdata
+
+arr_mass = numpy.random.normal(loc=0, scale=1.0, size=Data.nentries)
+arr_wgt  = numpy.random.normal(loc=1, scale=0.1, size=Data.nentries)
+
+# Make an instance
+wdata    = Wdata(data=arr_mass, weights=arr_wgt)
+
+# create a zfit dataset, if needed
+obs      = zfit.Space('obs', limits=(-3, +3))
+zdata    = wdata.to_zfit(obs=obs)
+
+# Add datasets
+wdata_1  = Wdata(data=arr_mass, weights=arr_wgt)
+wdata_2  = Wdata(data=arr_mass, weights=arr_wgt)
+wdata_3  = wdata_1 + wdata_2
+
+# Extract information from dataset
+
+wdata.sumw() # sum of weights
+wdata.size() # Number of entries
+
+# Update weights creating a new Wdata instance
+arr_wgt_new  = numpy.random.normal(loc=1, scale=0.2, size=Data.nentries)
+
+# New weights
+wdata_2 = wdata.update_weights(weights=arr_wgt_new, replace=True)
+
+# Multiply old weights by new ones and update
+wdata_3 = wdata.update_weights(weights=arr_wgt_new, replace=False)
+```
+
 ## PDFs
 
+### Suppressing tensorflow messages from zfit import
+
+If you work with zfit, you will see messages from tensorflow, by importing zfit through:
+
+```python
+from dmu.stats.zfit import zfit
+```
+
+these messages should be hidden. If `ROOT` is installed, the wrapper will import it before
+importing tensorflow. That will prevent crashes which usually happen when `tensorflow` 
+is imported before `ROOT`.
+
+### Toy models
+
+For quick tests, one can retrieve simple models with :
+
+```python
+from dmu.stats  import utilities as sut
+
+# For a Gaussian plus Exponential, extended
+pdf = sut.get_model(kind='s+b')
+
+# For a Gaussian signal, non extended
+pdf = sut.get_model(kind='signal')
+```
+
 ### Model building
 
 In order to do complex fits, one often needs PDFs with many parameters, which need to be added.
@@ -132,7 +408,27 @@ from dmu.stats.model_factory import ModelFactory
 
 l_pdf = ['cbr'] + 2 * ['cbl']
 l_shr = ['mu', 'sg']
-mod   = ModelFactory(obs = Data.obs, l_pdf = l_pdf, l_shared=l_shr)
+l_flt = ['mu', 'sg']                    # Will mark these parameters as floating for the fit done afterwards
+d_rep = {'mu' : 'scale', 'sg' : 'reso'} # Optional, will reparametrize for scale and resolution
+d_fix = {'al_cbl' : 3, 'nr_cbr' : 1}    # Optional, will fix two parameters whose names start with the keys
+
+# If mu and sg are meant to be shared among all the models
+# The parameters can be passed here.
+# In this case, they are also meant to be floating
+mu = zfit.param.Parameter('mu_flt', 5280, 5000, 5500)
+sg = zfit.param.Parameter('sg_flt',   80,   20,  100)
+l_reuse = [mu, sg]
+
+mod   = ModelFactory(
+    preffix = 'pref',   # Preffix for parameter naming
+    obs     = Data.obs, 
+    l_pdf   = l_pdf, 
+    l_shared= l_shr, 
+    l_float = l_float,
+    l_reuse = l_reuse,  # Optional
+    d_rep   = d_rep,    # Optional
+    d_fix   = d_fix)    # Optional
+
 pdf   = mod.get_pdf()
 ```
 
@@ -145,10 +441,63 @@ pol1: Polynomial of degree 1
 pol2: Polynomial of degree 2
 cbr : CrystallBall with right tail
 cbl : CrystallBall with left tail
-gauss : Gaussian 
+gauss : Gaussian
 dscb : Double sided CrystallBall
 ```
 
+### Model building with reparametrizations
+
+In order to introduce reparametrizations for the means and the resolutions, such that:
+
+$\mu\to\mu+\Delta\mu$
+$\sigma\to\sigma\cdot s_{\sigma}$
+
+where the reparametrized $\mu$ and $\sigma$ are constant, while the scale and resolution is floating, do:
+
+```python
+import zfit
+from dmu.stats.model_factory import ModelFactory
+
+l_shr = ['mu', 'sg']
+l_flt = []
+d_rep = {'mu' : 'scale', 'sg' : 'reso'}
+obs   = zfit.Space('mass', limits=(5080, 5680))
+
+mod   = ModelFactory(
+        preffix = name,
+        obs     = obs,
+        l_pdf   = l_name,
+        d_rep   = d_rep,
+        l_shared= l_shr,
+        l_float = l_flt)
+pdf   = mod.get_pdf()
+```
+
+Here, the floating parameters **should not** be the same as the reparametrized ones.
+
+### Overriding parameters
+
+The models above have their parameter ranges chosen for fits to B meson distributions
+e.g. the mean of the distributions is around 5GeV. To make these models extensible for other
+resonances do:
+
+```python
+from dmu.stats.parameters  import ParameterLibrary as PL
+
+# This will override the ranges and starting value
+PL.set_values(kind='cbr', parameter='mu', val=3000, low=2500, high=3500)
+
+# This will fix a parameter, the three arguments need to be equal
+PL.set_values(kind='cbr', parameter='sg', val=  30, low=  30, high=  30)
+```
+
+before using the `ModelFactory` class.
+For a summary of all the parameters and values available do:
+
+```python
+PL.print_parameters(kind='cbr')
+```
+
 ### Printing PDFs
 
 One can print a zfit PDF by doing:
@@ -210,6 +559,25 @@ print_pdf(pdf,
           txt_path = 'tests/stats/utilities/print_pdf/pdf_const.txt')
 ```
 
+
+### Storing PDF as latex
+
+The file above can be transformed into a `tex` file by running:
+
+```python
+from dmu.stats.utilities   import pdf_to_tex
+
+d_par = {
+    'ar_dscb_Signal_002_1_reso_flt' : r'$\alpha_{DSCB}^{1}$',
+    'ar_dscb_Signal_002_2_reso_flt' : r'$\alpha_{DSCB}^{2}$',
+    }
+
+# It will skip fixed parameters by default
+pdf_to_tex(path='/path/to/pdf.txt', d_par=d_par, skip_fixed=True)
+```
+
+where `d_par` will rename the `Parameters` column, such that it's in latex.
+
 ## Fits
 
 The `Fitter` class is a wrapper to zfit, use to make fitting easier.
@@ -273,8 +641,8 @@ strategy      :
 # The lines below will split the range of the data [0-10] into two subranges, such that the NLL is built
 # only in those ranges. The ranges need to be tuples
 ranges        :
-      - !!python/tuple [0, 3]
-      - !!python/tuple [6, 9]
+      - [0, 3]
+      - [6, 9]
 #The lines below will allow using contraints for each parameter, where the first element is the mean and the second
 #the width of a Gaussian constraint. No correlations are implemented, yet.
 constraints   :
@@ -356,6 +724,10 @@ obj   = ZFitPlotter(data=sam, model=pdf)
 d_leg = {'gauss': 'New Gauss'}
 obj.plot(nbins=50, d_leg=d_leg, stacked=True, plot_range=(0, 10), ext_text='Extra text here')
 
+#Alternatively one can do:
+obj.plot(nbins=50, d_leg=d_leg, stacked=True, ranges=[[0,3], [3,10]])
+# For plotting only sidebands, useful if one has a blinded fit
+
 # add a line to pull hist
 obj.axs[1].plot([0, 10], [0, 0], linestyle='--', color='black')
 ```
@@ -367,6 +739,71 @@ this class supports:
 - Stacking and overlaying of PDFs.
 - Blinding.
 
+## Fit saving
+
+To save in one go everything regarding your fit do:
+
+```python
+from dmu.stats              import utilities as sut
+from dmu.stats.zfit_plotter import ZFitPlotter
+
+ptr = ZFitPlotter(data=dat, model=pdf)
+ptr.plot()
+
+sut.save_fit(data=data, model=pdf, res=fit_result, fit_dir='/some/directory', d_const=constraints)
+```
+
+and the function will save everything that you would normally need from a fit.
+If the lines with `ZFitPlotter` were called before `save_fit` the fit plot will also be saved.
+
+### Transforming fit results to DictConfig
+
+The `OmegaConf` library offers `DictConfig` objects, which are easier to handle
+when reading nested data. To transform a zfit result object into one of these 
+objects do:
+
+```python
+from dmu.stats import utilities as sut
+
+cres = sut.zres_to_cres(res=res)
+```
+
+and then one would access the information like:
+
+```python
+error = cres.mu.error
+value = cres.mu.value
+```
+
+and these objects can be saved to JSON with:
+
+```python
+OmegaConf.save(config=cres, f='results.yaml')
+```
+
+## Placeholdef fits
+
+In order to create a _fake_ fit on top of which one could develop other tools, do:
+
+```python
+from dmu.stats import utilities
+
+utilities.placeholder_fit(kind='s+b', fit_dir='/some/directory')
+```
+
+## Retrieving information on fits
+
+Once the fit has be done and the results are saved to a given directory one can do:
+
+```python
+from dmu.stats.fit_stats    import FitStats
+
+obj =FitStats(fit_dir='/directory/with/fit')
+val = obj.get_value(name='var_name', kind='value or error')
+```
+
+and the tool will retrieve the value. This is useful when the values are needed elsewhere
+in the code, i.e. it would connect the fitting part with other parts.
 ## Arrays
 
 ### Scaling by non-integer
@@ -413,6 +850,24 @@ xval = numpy.lispace(0, 5, num=100)
 yval = fun(xval)
 ```
 
+## Other utilities
+
+These are here to decrease boilerplate code
+
+```python
+from dmu.stats import utilities as sut
+
+# Retrieves name of observable from observable
+name = sut.name_from_obs(obs=obs)
+
+# Retrieves range of observable from observable
+minx, maxx = sut.range_from_obs(obs=obs)
+
+# This is needed because when building a KDE with too little data, that KDE cannot be evaluated
+# and when trying it, tensorflow emits an exception.
+is_pdf_usable(pdf)
+```
+
 # Machine learning
 
 ## Classification
@@ -427,16 +882,31 @@ rdf_bkg = _get_rdf(kind='bkg')
 cfg     = _get_config()
 
 obj= TrainMva(sig=rdf_sig, bkg=rdf_bkg, cfg=cfg)
-obj.run(skip_fit=False) # by default it will be false, if true, it will only make plots of features
+obj.run(
+skip_fit=False, # by default it will be false, if true, it will only make plots of features
+opt_ntrial=20,  # By default this is zero, if a larger number is chosen, a hyperparameter optimization with optuna will run with this number of trials
+load_trained=False, # If true, it will not train the models but will just load them, only makes sense if models already exist. Useful to add postprocessing code, like the diagnostics section.
+)
 ```
 
 where the settings for the training go in a config dictionary, which when written to YAML looks like:
 
 ```yaml
 dataset:
+    # This section is optional. It can be used to redefine
+    # columns in different ways for different samples
+    #
+    # When evaluating the model, the same definitions will be used
+    # but they will be taken from the `sig` section.
+    samples:
+        sig:
+            definitions:
+                x : v + w
+        bkg:
+            definitions:
+                x : v - w
     # Before training, new features can be defined as below
     define :
-        x : v + w
         y : v - w
     # If the key is found to be NaN, replace its value with the number provided
     # This will be used in the training.
@@ -455,8 +925,8 @@ training :
       learning_rate     : 0.1
       min_samples_split : 2
 saving:
-    # The actual model names are model_001.pkl, model_002.pkl, etc, one for each fold
-    path : 'tests/ml/train_mva/model.pkl'
+    # The model names are model_001.pkl, model_002.pkl, etc, one for each fold
+    path : 'tests/ml/train_mva'
 plotting:
     roc :
         min : [0.0, 0.0] # Optional, controls where the ROC curve starts and ends
@@ -474,10 +944,7 @@ plotting:
       title      : 'Correlation matrix'
       size       : [10, 10]
       mask_value : 0                # Where correlation is zero, the bin will appear white
-    val_dir : 'tests/ml/train_mva'
     features:
-        saving:
-            plt_dir : 'tests/ml/train_mva/features'
         plots:
           w :
             binning : [-4, 4, 100]
@@ -499,6 +966,20 @@ plotting:
 
 the `TrainMva` is just a wrapper to `scikit-learn` that enables cross-validation (and therefore that explains the `nfolds` setting).
 
+#### Outputs
+
+The trainer will produce in the output:
+
+- Models in form of `pkl` files
+- Plots of the features
+- For each fold:
+1. Covariance plot
+1. ROC curve plot
+1. Feature importance table in latex
+1. JSON file with data to build the ROC curve
+- For the full dataset it will provide the ROC curve, scores distribution and JSON file with `x`, `y` coordinates for ROC curve.
+- Latex table with hyperparameters and NaN replacements.
+
 ### Caveats
 
 When training on real data, several things might go wrong and the code will try to deal with them in the following ways:
@@ -538,6 +1019,18 @@ If a sample exists, that was used in the training of _every_ model, no model can
 During training, the configuration will be stored in the model. Therefore, variable definitions can be picked up for evaluation
 from that configuration and the user does not need to define extra columns.
 
+### Further optimization
+
+If not all the entries of the ROOT dataframe are needed for the prediction (e.g. some entries won't be used anyway) define
+a column as:
+
+```python
+rdf = rdf.Define('skip_mva_prediction', 'mass < 3000')
+```
+
+and the predictor will assign scores of `-1` to all the entries with `mass < 3000`.
+This should speed up the prediction and reduce resource consumption.
+
 ### Caveats
 
 When evaluating the model with real data, problems might occur, we deal with them as follows:
@@ -552,12 +1045,158 @@ When evaluating the model with real data, problems might occur, we deal with the
     - For whatever features that are still NaN, they will be _patched_  with zeros when evaluated. However, the returned probabilities will be
 saved as -1. I.e. entries with NaNs will have probabilities of -1.
 
+## Diagnostics
+
+To run diagnostics on the trained model do:
+
+```python
+from dmu.ml.cv_diagnostics import CVDiagnostics
+
+# Where l_model is the list of models and cfg is a dictionary with the config
+cvd = CVDiagnostics(models=l_model, rdf=rdf, cfg=cfg)
+cvd.run()
+```
+
+the configuration can be loaded from a YAML file and would look like:
+
+```yaml
+# Directory where plots will go
+output         : /tmp/tests/dmu/ml/cv_diagnostics/overlay
+  # Optional, will assume that the target is already in the input dataframe
+  # and will use it, instead of evaluating models
+score_from_rdf : mva
+correlations:
+  # Variables with respect to which the correlations with the features will be measured
+  target :
+    name : mass
+    overlay :
+      # These are the working points at which the "mass" variable will be plotted
+      # If there is a correlation the shape should change
+      wp :
+        - 0.2
+        - 0.5
+        - 0.7
+        - 0.9
+      general:
+        size : [20, 10]
+      saving:
+        plt_dir : /tmp/tests/dmu/ml/cv_diagnostics/from_rdf
+      plots:
+        z :
+          binning    : [1000, 4000, 30]
+          yscale     : 'linear'
+          labels     : ['mass', 'Entries']
+          normalized : true
+  methods:
+    - Pearson
+    - Kendall-$\tau$
+  figure:
+    title: Scores from file
+    size : [10, 8]
+    xlabelsize: 18 # Constrols size of x axis labels. By default 30
+    rotate    : 60 # Will rotate xlabels by 60 degrees
+```
+
+## Comparing classifiers
+
+### Simple approach
+To do that run:
+
+```bash
+compare_classifiers -c /path/to/config.yaml
+```
+
+where the config looks like:
+
+```yaml
+out_dir : /path/to/plots
+classifiers:
+  label for model 1 : /path/to/directory/with/model1
+  label for model 2 : /path/to/directory/with/model2
+```
+
+However this will only compare the classifiers ROC curves with respect to the
+samples that were used to train them.
+
+### With custom samples
+
+However the models' peformances can also be compared by _plugging_ any
+signal and backgroud proxy for any model, like:
+
+```python
+import matplotlib.pyplot as plt
+from dmu.ml.cv_performance import CVPerformance
+
+cvp = CVPerformance()
+cvp.plot_roc(
+        sig  =rdf_sig_1, bkg=rdf_bkg_1,
+        model=l_model_1, name='def', color='red')
+cvp.plot_roc(
+        sig  =rdf_sig_1, bkg=rdf_bkg_2,
+        model=l_model_2, name='alt', color='blue')
+
+plt.legend()
+plt.grid()
+plt.show()
+```
+
+This should show an overlay of different ROC curves made for a specific combination
+of signal and background proxies with a given model.
+
+# Dask dataframes
+
+In order to process large ammounts of data a `Dask` dataframe is more suitable.
+A set of `ROOT` files can be loaded into one of these with:
+
+
+```python
+from dmu.rfile.ddfgetter   import DDFGetter
+
+# Can also pass directly the configuration dictionary with the `cfg` argument
+# If no columns argument is passed, will take all the columns
+
+ddfg = DDFGetter(config_path='config.yaml', columns=['a', 'b'])
+ddf  = ddfg.get_dataframe()
+
+# This will provide the pandas dataframe
+df   = ddf.compute()
+...
+```
+where `config.yaml` would look like:
+
+```yaml
+tree   : tree_name
+primary_keys:
+  - index
+files :
+  - file_001.root
+  - file_002.root
+  - file_003.root
+samples:
+  - /tmp/tests/dmu/rfile/main
+  - /tmp/tests/dmu/rfile/frnd
+```
+
 # Pandas dataframes
 
 ## Utilities
 
 These are thin layers of code that take pandas dataframes and carry out specific tasks
 
+### NaN filter
+
+The following snippet will remove NaNs from the dataframe
+if up to 2% of the rows have NaNs. Beyond that, an exception will be risen.
+
+```python
+import dmu.pdataframe.utilities as put
+
+# Default is 0.02
+df = put.dropna(df, nan_frac=0.02)
+```
+
+The usecase is cleaning up automatically, data that is not expected to be perfect.
+
 ### Dataframe to latex
 
 One can save a dataframe to latex with:
@@ -582,10 +1221,41 @@ put.df_to_tex(df,
         caption  = 'some caption')
 ```
 
+### Dataframe to and from YAML
+
+This extends the existing JSON functionality
+
+```python
+import dmu.pdataframe.utilities as put
+
+df_1 = _get_df()
+put.to_yaml(df_1, yml_path)
+df_2 = put.from_yaml(yml_path)
+```
+
+and is meant to be less verbose than doing it through the YAML module.
 # Rdataframes
 
 These are utility functions meant to be used with ROOT dataframes.
 
+## Cutflows from RDataFrames
+
+When using the `Filter` method on a ROOT dataframe, one can:
+
+```python
+rep = rdf.Report()
+rep.Print()
+```
+
+however this `rep` object is not python friendly, despite it is basically a table that can be
+put in pandas dataframe. Precisely this can be done with:
+
+```python
+from dmu.rdataframe import utilities as ut
+
+df = ut.rdf_report_to_df(rep)
+```
+
 ## Adding a column from a numpy array
 
 ### With numba
@@ -649,6 +1319,18 @@ obj = AtrMgr(rdf)
 obj.to_json('/path/to/file.json')
 ```
 
+## Filtering for a random number of entries
+
+The built in method `Range` only can be used to select ranges. Use
+
+```python
+import dmu.rdataframe.utilities as ut
+
+rdf = ut.random_filter(rdf, entries=val)
+```
+
+to select **approximately** a random number `entries` of entries from the dataframe.
+
 # Logging
 
 The `LogStore` class is an interface to the `logging` module. It is aimed at making it easier to include
@@ -668,6 +1350,25 @@ log.error('error')
 log.critical('critical')
 ```
 
+In order to get a specific logger do:
+
+```python
+logger = LogStore.get_logger(name='my_logger_name')
+```
+
+In order to get the logging level fromt the logger do:
+
+```python
+level = log.getEffectiveLevel()
+```
+
+And a context manager is available, which can be used with:
+
+```python
+    with LogStore.level('logger_name', 10):
+        log.debug('Debug message')
+```
+
 # Plotting from ROOT dataframes
 
 ## 1D plots
@@ -703,10 +1404,34 @@ definitions:
 plots:
     x :
         binning    : [0.98, 0.98, 40] # Here bounds agree => tool will calculate bounds making sure that they are the 2% and 98% quantile
-        yscale     : 'linear' # Optional, if not passed, will do linear, can be log
+        yscale     : linear # Optional, if not passed, will do linear, can be log
         labels     : ['x', 'Entries'] # Labels are optional, will use varname and Entries as labels if not present
-        title      : 'some title can be added for different variable plots'
-        name       : 'plot_of_x' # This will ensure that one gets plot_of_x.png as a result, if missing x.png would be saved
+        title      : some title can be added for different variable plots
+        name       : plot_of_x # This will ensure that one gets plot_of_x.png as a result, if missing x.png would be saved
+        weights    : my_weights # Optional, this is the column in the dataframe with the weights
+        # Can add styling to specific plots, this should be the argument of
+        # hist.plot(...)
+        styling :
+        # This section will update the styling of each category
+        # The categories (class A, etc) are the keys of the dictionary of dataframes
+        class A:
+            # These are the arguments of plt.hist(...)
+            histtype : fill 
+            color    : gray
+            alpha    : 0.3
+        class B:
+            color    : red 
+            histtype : step
+            linestyle: '-'  # Linestyle is by default 'none', 
+                            # needs to be overriden to see _steps_
+        # This will add vertical lines to plots, the arguments are the same
+        # as the ones passed to axvline
+        vline   :
+          x     : 0
+          label : label
+          ls    : --
+          c     : blue
+          lw    : 1
     y :
         binning    : [-5.0, 8.0, 40]
         yscale     : 'linear'
@@ -725,11 +1450,52 @@ style:
     # The line below would place the legend outside the figure to avoid ovelaps with the histogram
     bbox_to_anchor : [1.2, 1]
 stats:
-  nentries : '{:.2e}' # This will add number of entries in legend box
+  sumw : '{:.2f}' # This will add sum of weights to label. If no weights, then it will be the nentries value
 ```
 
 it's up to the user to build this dictionary and load it.
 
+### Pluggins
+
+Extra functionality can be `plugged` into the code by using the pluggins section like:
+
+#### FWHM
+```yaml
+plugin:
+  fwhm:
+    # Can control each variable fit separately
+    x :
+      plot   : true
+      obs    : [-2, 4]
+      plot   : true
+      format : FWHM={:.3f}
+      add_std: True
+    y :
+      plot   : true
+      obs    : [-4, 8]
+      plot   : true
+      format : FWHM={:.3f}
+      add_std: True
+```
+
+where the section will
+
+- Use a KDE to fit the distribution and plot it on top of the histogram
+- Add the value of the FullWidth at Half Maximum in the title, for each distribution with a specific formatting.
+
+#### stats
+
+```yaml
+plugin:
+  stats:
+    x :
+      mean : $\mu$={:.2f}
+      rms  : $\sigma$={:.2f}
+      sum  : $\Sigma$={:.0f}
+```
+
+Can be used to print statistics, mean, rms and weighted sum of entries for each distribution.
+
 ## 2D plots
 
 For the 2D case it would look like: