modelflowib 2.73__py3-none-any.whl

modelflowib-2.73.dist-info/METADATA

@@ -0,0 +1,156 @@
+Metadata-Version: 2.4
+Name: modelflowib
+Version: 2.73
+Summary: A tool to solve and manage dynamic economic and financial models
+Author-email: Ib Hansen <Ib.Hansen.Iv@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/IbHansen/ModelFlow2
+Project-URL: Repository, https://github.com/IbHansen/ModelFlow2
+Keywords: economics,dynamic models,BLL,numba,modelling
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: license.md
+Requires-Dist: pandas
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Requires-Dist: sympy
+Requires-Dist: numpy
+Requires-Dist: networkx
+Requires-Dist: tqdm
+Requires-Dist: scipy
+Requires-Dist: numexpr
+Requires-Dist: bottleneck
+Requires-Dist: XlsxWriter
+Requires-Dist: pyarrow
+Requires-Dist: tabulate
+Requires-Dist: openpyxl
+Requires-Dist: xarray
+Requires-Dist: cvxopt
+Requires-Dist: numba
+Requires-Dist: dash
+Requires-Dist: dash-bootstrap-components
+Requires-Dist: dash-interactive-graphviz
+Requires-Dist: lmfit
+Requires-Dist: jupyter
+Requires-Dist: ipywidgets
+Requires-Dist: ipydatagrid
+Provides-Extra: shiny
+Requires-Dist: shiny; extra == "shiny"
+Requires-Dist: shinywidgets; extra == "shiny"
+Dynamic: license-file
+
+# ModelFlow
+A Python toolkit to manage models
+
+The easy way to start is here [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/IbHansen/modelflow/master). 
+
+This will start **ModelFlow** as an online Jupyter notebook. Select one of the notebooks (files with the extension .ipynb). Some of the 
+notebooks are ModelFlow related. Other implement different model 
+
+ - FRBUS  will start the FRB/US model from the Federal reserve board. 
+ - Q-JEM will start the Quarterly Japanese Economic Model from Bank of Japan
+ - ADAM will start the Anual Danish Aggregate Model for the Danish Statistical agency. 
+ - Solow will implement a simple textybook Solow model. 
+
+You can also look at the python files of the system. Located at the **modelflow/** folder
+
+Alternative look at the *getting started* section later in this file. It explains how to run ModelFlow localy. 
+
+The **Pandas** library is a great library to handle all kinds of datamanipulation and transformations. 
+
+However when it comes to models which contains lags or models which requires solving simultanous equations, Pandas is not quite helpful. 
+
+ModelFlow extends Pandas to handle a range of such models. And they can be large.  
+
+It requires you to specify the *model* specified as equations (the **business logic**) and place the *data* in a Pandas  **DataFrame**.  ModelFlow allows the model to meet the data and return the result as a new DataFrame. 
+
+A number of **analytical tools for model and result analytic** helps to understand the model and its results.
+
+The user can **extend and modify the tools** to her or his needs.
+
+**Onboarding models and combining models from different sources**. Creating a Macro prudential model often entails recycling several models specified in different ways: Excel, Latex, Dynare, Python or other languages. Python's ecosystem makes it possible to transform many different models into ModelFlow models or to wrap them into functions which can be called from ModelFlow models.
+
+**Models can be specified in a high level Business logic language (a Domain Specific language)**. This allows the formulation of a model in a concise and expressive language which is close to the economic of the model. The user can concentrate on the economic or financial content - not the coding of the solution. The code for solving the model is generated by the tool. Then you can *solve the
+simultaneous* (or *non-simultaneous* model) in an efficient way. 
+
+## Introduction 
+
+**ModelFlow is written in Python**. Python comes "batteries included" and is
+the basis of a very rich ecosystem, which consists of a wide array of
+libraries. ModelFlow is just another library. It supplements the existing
+libraries regarding modeling language and solving and allows the use of
+Python as a model management framework.
+
+**Data handling and wrangling is done in the Pandas library**. This
+library is the Swiss army knife of data science in Python. It can import and export data to most systems and it is very powerful in manipulating and transforming data.
+The core
+element of Pandas is the *Dataframe*. A Dataframe is a two-dimensional
+tabular data structure. Each *column* consists of cells of the same type
+-- it can be a number, a string, a matrix or another Python data object.This includes matrices and other dataframes. Each *row is indexed.* The index can basically be any type of variable
+including dates, which is especially relevant for economic and financial models.
+
+**ModelFlow gives the user tools for more than solving models**. This
+includes:
+
+-   *Visualization* and comparison of results
+
+-   *Integration* of models from different sources
+
+-   *Analyze the logical structure of a model*. By applying graph theory, 
+    ModelFlow can find data lineage, find a suitable calculating sequence and trace 
+    causes of changes through the calculations.
+
+-   *Inverting* the model to calculating the necessary instruments to
+    achieve a desired target.
+
+-   Calculating the *attributions* from input to the results of a model.
+
+-   Calculating the *attribution* from input to the result of each
+    formula.
+
+-   Finding and calculating partial *derivatives* of formulas
+
+-   *Integrating user defined python functions* in the Business logic
+    language (like optimization, calculating risk weights or to make a matrices consistent with the RAS algorithm  )
+
+-   *Wrap matlab* models so they can be used in the Business logic
+    language.
+
+-   *Speed up* solving using "Just in time compilation"
+
+-   Analyze the model structure through tools from graph theory
+
+-   Handle *large models.* 1,000,000 formulas is not a problem.
+
+-   Integrate model management in Jupyter notebooks for *agile and user
+    friendly model use*
+
+
+**The core code of ModelFlow is small and
+documented.** Thus it can easily be modified and expanded to the specific need of the user. *ModelFlow is a toolset*. It can handle models, which conform to the tools.
+
+If you need a feature or have a model which can't be handled in ModelFlow,
+you are encouraged to improve ModelFlow. Please share the
+improvement, other users may have the same need, or can be inspired by
+your work.
+
+Also bear in mind that ModelFlow is experimental. It is provided ”as is”, without any representation or warranty of any kind either express or implied.   
+
+## Getting started
+
+You need Python 3.7+ with asssociated libraries. The easy way is to install Anaconda Python is to download the distribution from [https://www.anaconda.com/distribution](https://www.anaconda.com/distribution)
+
+Now you can install ModelFlow by running a command window from the Anaconda prompt and execute theese commands 
+```
+conda install modelflow -c ibh -c defaults -c conda-forge
+```
+You will find the anaconda prompt by searching "anaconda" in the start menu search field
+
+Try out one of the workbooks. to do this you:
+```
+click on the Adaconda prompth in the Anaconda folder
+cd <The location you have downloaded the workbooks>
+jupyter notebook
+```

modelflowib-2.73.dist-info/RECORD

@@ -0,0 +1,44 @@
+modelBLfunk.py,sha256=2behWirmYQ7XMcp2E8plH5wjQ3M2nnqL-mX0SgaoHp4,4769
+model_Excel.py,sha256=RiqfAWPrzmVT3eRLTNAy0ASHdx5pQsRABe5XlW_Kye4,11840
+model_cvx.py,sha256=PKBJ2gv6Iryl4N5YrWi6R5XTenxIeOnlR57HQ2EhPYk,6102
+model_dynare.py,sha256=WLW4m1SrfVlv0c6gdYhLCn9S8W8nLYkZst0etPe9JnQ,8037
+model_financial_stability.py,sha256=3Wl20LV0T7EDcbf7THu2XXun-ParERaFatEGWG2EyKM,3084
+model_latex.py,sha256=PKC0EWY37e4odM0he_sFLM9Ywbc9ydAPTFp6bZfbruI,18287
+model_latex_class.py,sha256=rgVrG5rEozrlWndrsSfRkporn2Fwy0L9ktNov26ZxE0,28471
+model_parquet_mixin.py,sha256=y75nrLIBovJYg5Ecq1cUqo3VRRCkM4Gg8incvkolMW0,16699
+modelclass.py,sha256=Sa8LsLW8VTEUxkjPsOaPVmTA011Afil8Ho1e6FTY3So,444613
+modelconstruct.py,sha256=Iv0C36egeinys5ppgXnl1_6W2VUCpt7w-sYSH26a4SE,51054
+modelconstruct_estimation.py,sha256=0Yzm6ZqEKWmD-kIElT66Om8Hcu6Sl7OM6DZsWkMXlJg,106664
+modeldash.py,sha256=Nns5JqAP5xjrdkD5752w2m9hwdm0F1mSi9RUNR0ZZRY,10641
+modeldashboot.py,sha256=rSyg5aYgRjVeJNHTGbE2X_1bvUmpW_sK-oYcDCrb_gU,7516
+modeldashsidebar.py,sha256=mUPCPFeYb6YPF1SFm4NALhbTZuYTBFnPq8RA1eeNhdA,18547
+modeldekom.py,sha256=rnc8wBEZ65KRq44ICvbX2aviC7XPM7bATKeCrXthoK8,30360
+modeldiff.py,sha256=erNGFxi2jy-59yxfkMv5D9lBSPauuljXlPNU2twrQyM,22500
+modeldisplay.py,sha256=Ock9hJuOYh5onqyzT21VTIKR8_cqyoBEbMgQN5bCaLc,21301
+modelestimation.py,sha256=bhEV2FsIINIad2oo-dAphdigmKaxZSyg4sHjZbcTb7I,70482
+modelestimator_new.py,sha256=JEAYhwtjoPAm6tleR2xa0AqIiMm0jjyOwccMGgD5610,107136
+modelgrab.py,sha256=3bJJRzfGChNExTOGfxzCfkq8aPiXRnBNRfLBeRB-GMc,14769
+modelgrabgdx.py,sha256=pdi3HOM2NP7BvpC0Stg6PxUsOMlQMjrRh7oT9ZOheFk,21644
+modelgrabwf2.py,sha256=6N62awkP8qx0yS4Tp-4Gz54QTKP_d5TYgMEECScXdpI,45239
+modelhelp.py,sha256=47Lq9N-orlhry2ldfWzI1jICRRzT6jZ_TfOBcsl25Vs,21135
+modelhtml.py,sha256=hOpczHInDqcwHLDo_WynBXfb4vvlMabk7yTsITlaYRg,22083
+modelinvert.py,sha256=sbPPk29RlpMMnzB15g5FyE4SZvI9hxtGDkUbvK4_hIc,11217
+modeljupyter.py,sha256=onKYvNUI4YfX8PjO8pJPe5vcQBwVa1Uua6mxLueyFoM,34218
+modeljupytermagic.py,sha256=il7CXYQ6RiaOISu3ZJMBsZrzqGvLCKewIXCCkHfJVQA,27271
+modelmacrograb.py,sha256=tBoISMYPkuckiop0HaGSKb3OwMGbJejXKmyJNXAQGAo,3771
+modelmanipulation.py,sha256=olHIxKueVuspSGI47gmgQLoha_aaQq5jeaA7ot9yNX0,60771
+modelmf.py,sha256=fNHB3KcoFYOhiIpX2X8xevnBkK8xD1Z5I8ehdwT5xgU,12409
+modelnet.py,sha256=dMvJ0mdY9OCVD-VvmVdis-9TgQbediHfsKLHGCKO4b4,4960
+modelnewton.py,sha256=3YbE7yGhOSuzL5Y-YtLSERkEKGFNXA40yDHtJT1vcto,104652
+modelnormalize.py,sha256=0JwQ4XLVMXgZvdSiHiDutnyl6p_AIXde146mqOddg_E,18053
+modelpattern.py,sha256=yw5ivHReI_uuEw3R6Grc7ehwrwDVCSBRQ5g75jByBkg,16809
+modelreport.py,sha256=WRolsT4qInCLULRXSa4SYQNB40xUAmga02OaJAckxFI,88391
+modeluserfunk.py,sha256=ro3C4bRFR0c-jxNwbmGJF5Fs9021ol1a5HrPqZfe1gE,3349
+modelvis.py,sha256=hPPlPwSlhZ7tY4B_58QsH2olvM1zA0-zDdcFH-RaIYE,43430
+modelwidget.py,sha256=tUy4aJgggSJdVH4kz5SSNnGNINjx2eg7Qxlr9L3I8Vg,27515
+modelwidget_input.py,sha256=Tn_uBPGsNlMrU3Ch0y4assLqegyKdfRyfFInJ7EfbO0,69446
+modelflowib-2.73.dist-info/licenses/license.md,sha256=WHoD6zNrj781B-IpjmCnu4gPDhkzexjJkVtnYfWFy3Y,1353
+modelflowib-2.73.dist-info/METADATA,sha256=FAE2UIa2GxeHec5hljk4sTk_8dqpKdwxF3bL2X-UgH4,7440
+modelflowib-2.73.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+modelflowib-2.73.dist-info/top_level.txt,sha256=GIAiCmVrEXo48UyogZXtVa4JeKSTgAd9dP_s8weM4QA,539
+modelflowib-2.73.dist-info/RECORD,,

modelflowib-2.73.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

modelflowib-2.73.dist-info/licenses/license.md

@@ -0,0 +1,10 @@
+Copyright © 2019 Danmarks Nationalbank and ECB
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Soft-ware"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+Except as contained in this notice, the name of Danmarks Nationalbank and ECB shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Software without prior written authorization from Danmarks Nationalbank and ECB.
+
+Based on MIT License. 

modelflowib-2.73.dist-info/top_level.txt

@@ -0,0 +1,39 @@
+modelBLfunk
+model_Excel
+model_cvx
+model_dynare
+model_financial_stability
+model_latex
+model_latex_class
+model_parquet_mixin
+modelclass
+modelconstruct
+modelconstruct_estimation
+modeldash
+modeldashboot
+modeldashsidebar
+modeldekom
+modeldiff
+modeldisplay
+modelestimation
+modelestimator_new
+modelgrab
+modelgrabgdx
+modelgrabwf2
+modelhelp
+modelhtml
+modelinvert
+modeljupyter
+modeljupytermagic
+modelmacrograb
+modelmanipulation
+modelmf
+modelnet
+modelnewton
+modelnormalize
+modelpattern
+modelreport
+modeluserfunk
+modelvis
+modelwidget
+modelwidget_input

modelgrab.py

@@ -0,0 +1,318 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Mon Jun 10 21:11:08 2019
+
+@author: hanseni
+
+    modules to grab models with different specifications and make them ModelFlow conforme 
+
+   **GrabWbModel** will take a eviews model  and transform it to Business logic
+  - Create a normalized model, add dampning for the stocastic equations 
+  - Add add-factors to the stocastic equations 
+  - Generate BL for a model which calculates add-factors so a solution will match teh existing values 
+  - Generate BL for the model   
+
+  -grap data from excel sheet 
+
+  - Make model instance for model and add-factor model 
+
+  - Run the model, check that the results match. 
+
+For debuggging valuesthe last part checs value in the order, in which they are calculated,
+and then displays the input to off mark equations  
+"""
+
+import pandas as pd
+import re
+from dataclasses import dataclass
+import functools
+from tqdm import tqdm 
+import json 
+from pathlib import Path
+
+
+from modelclass import model 
+import modelmf 
+import modelmanipulation as mp 
+
+import modelnormalize as nz
+
+assert 1==1
+
+
+@dataclass
+class GrapWbModel():
+    '''This class takes a world bank model specification, variable data and variable description
+    and transform it to ModelFlow business language'''
+    
+    
+    frml      : str = ''            # path to model 
+    data      : str = ''          # path to data 
+    des       : any = ''            # path to descriptions
+    scalars   : str = ''           # path to scalars 
+    modelname : str = 'No Name'           # modelname
+    start     : int = 2017
+    end       : int = 2040 
+    country_trans   : any = lambda x:x[:]    # function which transform model specification
+    country_df_trans : any = lambda x:x     # function which transforms initial dataframe 
+    from_wf2  : bool = False
+    make_fitted  : bool = False # if True, a clean equation for fittet variables is created
+    fit_start   : int =2000   # start of fittet model unless overruled by mfmsa
+    fit_end     : int = 2100  # end of fittet model unless overruled by mfmsa
+    do_add_factor_calc     : bool = True  # calculate the add factors 
+    mfmsa       : str = '' # path to mfsa options 
+    
+    
+    def __post_init__(self):
+        # breakpoint()
+        
+        print(f'\nProcessing the model:{self.modelname}',flush=True)
+        self.rawmodel_org = open(self.frml).read()
+        eviewsline  = [l for l in self.rawmodel_org.split('\n') if len(l.strip()) >=2 
+                                                               and not ('*******' in l or '----------' in l)] 
+        # breakpoint()
+        self.rawmodel = self.country_trans(self.rawmodel_org)
+        rawmodel6 = self.trans_eviews(self.rawmodel)
+        # breakpoint()
+        bars = '{desc}: {percentage:3.0f}%|{bar}|{n_fmt}/{total_fmt}'
+        if self.from_wf2:
+            
+            orgline = [l for l in rawmodel6.split('\n')]
+            line_type = ['ident' if l.startswith('@IDENTITY') else 'stoc' for l in orgline]
+            line = [l.replace('@IDENTITY ','').replace(' ','')  for l in orgline]
+        else:    
+            line_type = []
+            line =[] 
+            with tqdm(total=len(rawmodel6.split('\n')),desc='Reading original model',bar_format=bars) as pbar:
+                for l in rawmodel6.split('\n'):
+                    if ('*******' in l or '----------' in l)  and 'IDEN' in l.upper():
+                        sec='iden'
+                        #print(l)
+                    elif '*******' in l and 'STOC' in l:
+                        sec='stoc'
+                        #print(l)
+                    else: 
+                        line_type.append(sec)
+                        line.append(l)
+                        # print(f' {sec} {l[:30]} ....')
+                    pbar.update(1)   
+                    
+                    
+        errline1 = [(l,o,e) for l,o,e in zip(line,orgline,eviewsline) if '@' in l or '.coef' in l ]
+        if len(errline1):
+            print('Probably errors as @ in lines:')
+            for l,o,e in errline1:
+                print(f'\nEviews line      :{e}')
+                print(f'Original line     :{o}')
+                print(f'New modelflow line:{l}')
+            raise Exception('@ in lines ')
+            
+        self.all_frml = [nz.normal(l,add_add_factor=(typ=='stoc'),make_fitted=(typ=='stoc'),make_fixable =(typ=='stoc')) for l,typ in tqdm(zip(line,line_type),desc='Normalizing model',total=len(line),bar_format=bars)]
+        self.all_frml_dict = {f.endo_var: f for f in self.all_frml}
+        lfname = ["<Z,EXO> " if typ == 'stoc' else '' for typ in line_type ]
+        self.rorg = [fname + f.normalized for f,fname in zip(self.all_frml,lfname) ]
+        
+        if self.make_fitted:
+            self.rfitmodel = ['<FIT> ' + f.fitted for f in self.all_frml if len(f.fitted)]
+            self.mfitmodel = model('\n'.join(self.rfitmodel))
+            self.mfitmodel.modelname = self.modelname + ' calc fittet values'
+        else: 
+            self.rfitmodel = []
+                
+        self.rres = [f'{f.calc_add_factor}' for f in self.all_frml if len(f.calc_add_factor)]
+        self.rres_tomodel ='\n'.join([f'FRML <CALC_ADD_FACTOR> {f.calc_add_factor}$' for f in self.all_frml if len(f.calc_add_factor)])
+        # self.fmodel = mp.exounroll(mp.tofrml ('\n'.join(self.rorg+self.rfitmodel)))+self.rres_tomodel
+        self.fmodel = mp.tofrml ('\n'.join(self.rorg+self.rfitmodel))+self.rres_tomodel
+        # breakpoint()
+        self.fres =   ('\n'.join(self.rres))
+        self.mmodel = model(self.fmodel,modelname = self.modelname)
+        self.mmodel.set_var_description(self.var_description)
+        self.mmodel.wb_MFMSAOPTIONS = self.mfmsa_options
+        self.mres = model(self.fres,modelname = f'Calculation of add factors for {self.modelname}')
+        # breakpoint()
+        self.start,self.end = self.mfmsa_start_end
+        if self.do_add_factor_calc:
+            self.base_input = self.mres.res(self.dfmodel,self.start,self.end)
+        else: 
+            self.base_input = self.dfmodel
+        
+    @staticmethod
+    def trans_eviews(rawmodel):
+        rawmodel0 = '\n'.join(l for l in rawmodel.upper().split('\n') if len(l.strip()) >=2)
+        # trailing and leading "
+        rawmodel1 = '\n'.join(l[1:-1] if l.startswith('"') else l for l in rawmodel0.split('\n'))
+        # powers
+        rawmodel2 = rawmodel1.replace('^','**').replace('""',' ').replace('"',' ').\
+            replace('@EXP','exp').replace('@RECODE','recode').replace('@MOVAV','movavg').replace('@LOGIT','logit_inverse') \
+            .replace('@MEAN(@PC(','@AVERAGE_GROWTH((').replace('@PCY','PCT_GROWTH').replace('@PC','PCT_GROWTH')\
+            .replace('@PMAX','MAX').replace('@TREND','EVIEWS_TREND')      
+        # @ELEM and @DURING 
+        # @ELEM and @DURING 
+        rawmodel3 = nz.elem_trans(rawmodel2) 
+        rawmodel4 = re.sub(r'@DURING\( *([0-9]+) *\)', r'during_\1',rawmodel3) 
+        rawmodel5 = re.sub(r'@DURING\( *([0-9]+) *([0-9]+) *\)', r'during_\1_\2',rawmodel4) 
+        
+        # during check 
+        ldur = '\n'.join(l for l in rawmodel5.split('\n') if '@DURING' in l)
+        ldur2 = '\n'.join(l for l in rawmodel5.split('\n') if 'during' in l)
+        
+        # check D( 
+        ld  = '\n'.join(l for l in rawmodel5.split('\n') if re.search(r'([^A-Z]|^)D\(',l) )
+        ld1  = '\n'.join(l for l in rawmodel5.split('\n') if re.search(r'([^A-Z0-9_]|^)D\(',l) )
+        # breakpoint()
+        rawmodel6 = nz.funk_replace('D','DIFF',rawmodel5) 
+        # did we get all the lines 
+        ldif = '\n'.join(l for l in rawmodel6.split('\n') if 'DIFF(' in l )
+        return rawmodel6
+    
+    @property 
+    def var_description(self):
+        '''
+        Adds var descriptions for add factors, exogenizing dummies and exoggenizing values
+        '''
+        
+        if isinstance(self.des,dict):
+            return self.des
+        
+        try:
+            # breakpoint()
+            des_file = Path(self.des)
+            if des_file.suffix == '.xlsx':
+                trans0 = pd.read_excel(self.des).loc[:,['mnem','Excel']].set_index('mnem').to_dict(orient = 'dict')['Excel']
+                var_description = {str(k) : str(v) for k,v in trans0.items() if 'nan' != str(v)}
+            else:
+                with open(des_file,'rt') as f: 
+                    var_description = json.load(f)
+            add_d =   { newname : 'Add factor:'+ var_description.get(v,v)      for v in self.mmodel.endogene if  (newname := v+'_A') in self.mmodel.exogene }
+            dummy_d = { newname : 'Exo dummy:'+ var_description.get(v,v)  for v in self.mmodel.endogene if  (newname := v+'_D')  in self.mmodel.exogene }
+            exo_d =   { newname : 'Exo value:'+ var_description.get(v,v)      for v in self.mmodel.endogene if  (newname := v+'_X')  in self.mmodel.exogene }
+            fitted_d =   { newname : 'Fitted  value:'+ var_description.get(v,v)      for v in self.mmodel.endogene if  (newname := v+'_FITTED')  in self.mmodel.endogene }
+            var_description =  {**var_description,**add_d,**dummy_d,**exo_d,**fitted_d}
+            self.mmodel.set_var_description(var_description)
+        except:
+            print('*** No variable description',flush=True)
+            var_description = {}
+        return var_description    
+    
+    @functools.cached_property
+    def mfmsa_options(self):
+        '''Grab the mfmsa options, a world bank speciality''' 
+        if self.mfmsa:
+            with open(self.mfmsa,'rt') as f:
+                options = f.read()
+        else:
+            options = ''
+        return options     
+    
+    @functools.cached_property
+    def mfmsa_start_end(self):
+        import xml
+        root = xml.etree.ElementTree.fromstring(self.mfmsa_options)
+        start = int(root.find('iFace').find('SolveStart').text)   
+        end = int(root.find('iFace').find('SolveEnd').text)  
+        return start,end 
+            
+    
+    @functools.cached_property
+    def dfmodel(self):
+        '''The original input data enriched with during variablees, variables containing 
+        values for specific historic years and model specific transformation '''
+        # Now the data 
+        if self.from_wf2:
+            df = pd.read_excel(self.data,index_col=0)
+        else:       
+            df = (pd.read_excel(self.data).
+                  pipe( lambda df : df.rename(columns={c:c.upper() for c in df.columns})).
+                  pipe( lambda df : df.rename(columns={'_DATE_':'DATEID'})).
+                  pipe( lambda df : df.set_index('DATEID'))
+                  )
+            df.index = [int(i.year) for i in df.index]
+        
+        try:
+            sca = pd.read_excel(self.scalars ,index_col=0,header=None).T.pipe(
+                lambda _df : _df.loc[_df.index.repeat(len(df.index)),:]).\
+                set_index(df.index)
+            df= pd.concat([df,sca],axis=1)    
+        except: 
+            print(f'{self.modelname} no Scalars prowided ')
+        # breakpoint()
+        if self.make_fitted:
+            df = self.mfitmodel.res(df,self.fit_start,self.fit_end)
+
+            
+        #% Now set the vars with fixedvalues 
+        value_vars = self.mmodel.vlist('*_value_*')
+        for var,val,year in (v.rsplit('_',2) for v in value_vars) : 
+            df.loc[:,f'{var}_{val}_{year}'] = df.loc[int(year),var]
+        self.showvaluevars = df[value_vars] 
+        
+        #% now set the values of the dummies   
+        # breakpoint() 
+        during_vars = self.mmodel.vlist('*during_*')
+        for varname,(dur,per) in ((v,v.split('_',1)) for v in during_vars):
+            df.loc[:,varname]=0
+            # print(varname,dur,per)
+            pers = per.split('_')
+            if len(pers) == 1:
+                df.loc[int(pers[0]),varname] = 1
+            else:
+                df.loc[int(pers[0]):int(pers[1]),varname]=1.
+        self.showduringvars = df[during_vars] 
+        # breakpoint()
+        df_out = self.mmodel.insertModelVar(df).pipe(self.country_df_trans).fillna(0.0)
+        return df_out
+    
+    def __call__(self):
+
+        return self.mmodel,self.base_input
+    
+    def test_model(self,start=None,end=None,maxvar=1_000_000, maxerr=100,tol=0.0001,showall=False):
+        '''
+        Compares a straight calculation with the input dataframe. 
+        
+        shows which variables dont have the same value 
+
+        Args:
+            df (TYPE): dataframe to run.
+            start (TYPE, optional): start period. Defaults to None.
+            end (TYPE, optional): end period. Defaults to None.
+            maxvar (TYPE, optional): how many variables are to be chekked. Defaults to 1_000_000.
+            maxerr (TYPE, optional): how many errors to check Defaults to 100.
+            tol (TYPE, optional): check for absolute value of difference. Defaults to 0.0001.
+            showall (TYPE, optional): show more . Defaults to False.
+
+        Returns:
+            None.
+
+        '''
+        _start = start if start else self.start
+        _end    = end if end else self.end
+        # breakpoint()
+    
+        resresult = self.mmodel(self.base_input,_start,_end,reset_options=True,silent=0,solver='base_res')
+        self.mmodel.basedf = self.dfmodel
+        pd.options.display.float_format = '{:.10f}'.format
+        err=0
+        print(f'\nChekking residuals for {self.mmodel.name} {_start} to {_end}')
+        for i,v in enumerate(self.mmodel.solveorder):
+            # if v.endswith('_FITTED'): continue
+            if i > maxvar : break
+            if err > maxerr : break
+            check = self.mmodel.get_values(v,pct=True).T 
+            check.columns = ['Before check','After calculation','Difference','Pct']
+            # breakpoint()
+            if (check.Difference.abs() >= tol).any():                
+                err=err+1
+                maxdiff = check.Difference.abs().max()
+                maxpct  = check.Pct.abs().max()
+                # breakpoint()
+                print('\nVariable with residuals above threshold')
+                print(f"{v}, Max difference:{maxdiff:15.8f} Max Pct {maxpct:15.10f}% It is number {i} in the solveorder and error number {err}")
+                if showall:
+                    print(f'\n{self.mmodel.allvar[v]["frml"]}')
+                    print(f'\nResult of equation \n {check}')
+                    print(f'\nEquation values before calculations: \n {self.mmodel.get_eq_values(v,last=False,showvar=1)} \n')
+        self.mmodel.oldkwargs = {}
+        
+        
+