LSMS_Library 0.2.9.dev0__py3-none-any.whl → 0.2.10.dev0__py3-none-any.whl

lsms_library/contributing.org

@@ -1,290 +0,0 @@
-
-* YAML File Development Instructions for LSMS Package
-
-This guide explains how to organize ~data_scheme.yml~, ~data_info.yml~, and optional Python transformation scripts to ensure proper functionality of the LSMS package.
-Maintaining correct indentation and formatting in ~data_info.yml~ is essential for the file to be processed correctly.
-
-* 1. Create the _/ Subdirectory and data_scheme.yml
-
-In each *country directory*, create a subdirectory named _:
-
-#+begin_src sh
-mkdir Mali/_
-#+end_src
-
-Inside the _/ folder, create a ~data_scheme.yml~ file to define the data structure and index for each table:
-
-#+begin_src yaml
-# Mali/_/data_scheme.yml
-
-Country: Mali
-
-Data Scheme:
-  cluster_features:
-    index: (t, v)
-    Region: str
-    Rural: str
-
-  household_roster:
-    index: (t, v, i, pid)
-    Sex: str
-    Age: int
-    Relationship: str
-
-  individual_education:
-    index: (t, v, i, pid)
-    Highest Education: str
-
-  food_acquired:
-    index: (t, v, visit, i, j, u)
-    Quantity: float
-    Price: float
-    Produced: float
-
-  interview_date:
-    index: (t, v, visit, i)
-    Int_t: str
-
-  panel_ids:
-    index: (t, i)
-    previous_i: str
-#+end_src
-
-* 2. Create the _/ Subdirectory in Each Wave and Define data_info.yml
-
-In each *wave directory*, also create an _ folder:
-
-#+begin_src sh
-mkdir Mali/2017-18/_
-#+end_src
-
-Then, inside _/, create a ~data_info.yml~ file to describe the data files and mappings.
-
-** 2.1 Basic Format
-
-#+begin_src yaml
-# Mali/2017-18/_/data_info.yml
-
-cluster_features:
-  file: ehcvm_conso_mli2018.dta
-  idxvars:
-    v: grappe
-  myvars:
-    Region: region
-    Rural: milieu
-#+end_src
-
-** 2.2 Composite Index or Variables with Functions 
-
-If the variable (e.g., i) requires multiple columns from the original dataset, list them together in a list format, in yml using hyphen (-). For example, if variable i maps to multiple source columns, define it as:
-#+begin_src yaml
-# Mali/2021-22/_/data_info.yml
-
-food_acquired:
-  file: s07b_me_mli2021.dta
-  idxvars:
-    v: grappe
-    visit: vague
-    i:          # Empty row is required here
-      - grappe
-      - menage
-    j: s07bq01
-    u: s07bq03b
-  myvars:
-  ...
-#+end_src
-
-* 3. Apply Transformation Functions in Python
-The `format_id` function from local_tools is automatically applied to all index variables by default.
-
-To apply a different formatting function to a specific index or column variable, you can define your own function and specify it in one of two ways:
-	•	Option 1: Name the function to match the variable name (e.g., i for variable i), and it will be recognized automatically.
-	•	Option 2: Add 'function: function_name' at the end of the mapping list (see bullet 7 for details). This explicitly assigns the function to mappings.
-
-This approach allows flexible customization of formatting logic for individual variables.
-** 3.1 Wave-Specific Column Transformation
-
-Create a Python file in the wave’s _ folder, e.g., ~Mali/2017-18/_/2017-18.py~:
-
-#+begin_src python
-# Mali/2017-18/_/2017-18.py
-
-def i(value):
-    from lsms_library.local_tools import format_id
-    return format_id(value[0]) + '0' + format_id(value[1], zeropadding=2)
-#+end_src
-
-** 3.2 Country-Wide Column Transformation
-
-To apply a transformation across all waves in a country, define the function in ~Mali/_/mali.py~.
-
-#+begin_src python
-# Mali/_/mali.py
-
-def i(value):
-    from lsms_library.local_tools import format_id
-    return format_id(value[0]) + '0' + format_id(value[1], zeropadding=2)
-#+end_src
-
-* 4. Value Mapping Dictionaries
-
-Define value mappings in ~data_info.yml~ for categorical variables.
-
-#+begin_src yaml
-# Malawi/2004-05/_/data_info.yml
-
-myvars:
-  Rural:
-    - reside
-    -       # Empty row is required here and the following should be in correct indention
-      'Rural': 1 # original_value: mapping_value
-      'Urban': 0
-#+end_src
-
-#+begin_src yaml
-# Mali/2021-22/_/data_info.yml
-
-myvars:
-  in_previous_wave:
-    - PanelHH
-    - mapping:         
-            'Oui': 1  # original_value: mapping_value
-            'Non': 0
-#+end_src
-
-Note: The empty line after the variable name is *required* to separate it from the mapping dictionary.
-
-* 5. Row-Level DataFrame Transformation
-
-Define a function named after the dataframe in either the wave- or country-level Python file:
-
-#+begin_src python
-# Mali/2021-22/_/2021-22.py
-
-def panel_ids(df):
-    from lsms_library.local_tools import format_id
-    df = df[df['visit'] == '2']
-    df['previous_i'] = df[['previous_v', 'previous_hid']].apply(
-        lambda x: format_id(x[0]) + '0' + format_id(x[1], zeropadding=2), axis=1)
-    return df[['i', 't', 'previous_i']].drop_duplicates().set_index(['i', 't'])
-#+end_src
-
-* 6. Multi-File DataFrames
-
-** 6.1 Vertical Merge: Stacking Data Row-Wise (data from multiple files)
-
-#+begin_src yaml
-# Mali/2017-18/_/data_info.yml
-
-interview_date:
-  file:
-    - eaci17_s00p1.dta
-    - eaci17_s00p2.dta
-  idxvars:
-    v: grappe
-    visit: passage
-    i:
-      - grappe
-      - exploitation
-  myvars:
-    Int_t:
-      - s0q21c
-      - s0q21b
-      - s0q21a
-  converted_categoricals: False
-#+end_src
-
-** 6.2 File-Specific Overrides
-
-#+begin_src yaml
-# Mali/2014-15/_/data_info.yml
-food_acquired:
-  file:
-    - EACIALI_p1.dta:       #A colon is required here.
-        Quantity: s13q03a   #Override defult Quantity column mapping
-    - EACIALI_p2.dta
-  idxvars:
-    v: grappe
-    visit: passage
-    i:
-      - grappe
-      - menage
-    j: s13q01
-    u: s13q03b
-  myvars:
-    Quantity: s13q02
-    Expenditure: s13q03c
-    Produced: s13q04a
-#+end_src
-
-
-* 7. File-Specific Function Applications
-To apply a function to a specific column mapping, add 'function: function_name' at the end of the variable mapping list (as shown in the example for variable i). This function will be applied only to files that use this particular column mapping. If an override mapping is present, the function will not be applied.
-
-If you want the function to apply across all mappings for column i, refer to bullet point 3, where you can simply define the function name directly after the variable name i.
-
-#+begin_src yaml
-# Malawi/2016-17/_/data_info.yml
-
-household_roster:
-  file:
-    - Cross_Sectional/hh_mod_b.dta
-    - Panel/hh_mod_b_16.dta:
-        i: y3_hhid
-        pid: id_code
-  idxvars:
-    i:
-      - case_id
-      - mapping: cs_i
-    pid: pid
-  myvars:
-    Sex: hh_b03
-    Age: hh_b05a
-    Relation: hh_b04
-#+end_src
-
-src python
-# Malawi/2016-17/_/2016-17.py
-
-def cs_i(value):
-    from lsms_library.local_tools import format_id
-    return 'cs-17-' + format_id(value[0])
-#+end_src
-
-* 8. 	Horizontal Merge = Combining Data Column-Wise (adding columns side by side).
-
-#+begin_src yaml
-# an example
-merged_dfs:
-    dfs:
-        - df_1
-        - df_2
-    df_1:
-        file: file_1
-        idxvars:
-            i: i_1
-            j: j_1
-        myvars:
-            var: var
-    df_2:
-        file: file_2
-        idxvars:
-            i: i_2
-            j: j_2
-        myvars:
-            var1: var
-    merged_on:  #list of variables to merge on
-        - i 
-        - j
-    final_index:
-        - i
-        - j
-#+end_src
-
-* 9. Hard-coded Case
-
-In some cases where dataframes require more complex logic than simple mappings, create a Python file named after the dataframe in the wave's _ folder (e.g., ~Malawi/2004-05/_/food_acquired.py~). 
-And create a Makefile on its country level/_ directory (e.g. ~Malawi/_/Makefile~).
-The system will automatically generate the dataframe by running Makefile.
-
-Note: The reason why not just simply run the python file is becasue some data's python files have dependencies.

lsms_library-0.2.9.dev0.dist-info/METADATA

@@ -1,138 +0,0 @@
-Metadata-Version: 2.3
-Name: LSMS_Library
-Version: 0.2.9.dev0
-Summary: Abstraction layer for Living Standards Measurement Survey data
-License: CC-BY-4.0
-Author: Ethan Ligon
-Author-email: ligon@berkeley.edu
-Requires-Python: >=3.11, <4.0
-Classifier: License :: Other/Proprietary License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Requires-Dist: CFEDemands (>=0.7.0)
-Requires-Dist: PyYAML
-Requires-Dist: dvc[s3] (>=3.61.0,<4.0.0)
-Requires-Dist: ligonlibrary (>=0.1.1,<0.2.0)
-Requires-Dist: lsms (>=0.4.7,<0.5.0)
-Requires-Dist: openai (>=1.99.5,<2.0.0)
-Requires-Dist: pandas (>=2.2,<3.0)
-Requires-Dist: pyarrow (>=21.0.0,<22.0.0)
-Requires-Dist: pyreadstat (>=1.3.0,<2.0.0)
-Requires-Dist: scikit-learn (>=1.7.0,<2.0.0)
-Requires-Dist: xarray (>=2025.7.1,<2026.0.0)
-Description-Content-Type: text/plain
-
-* Streaming dvc files
-   A =dvc pull= will download dvc files to your local repository.
-   But this may not be the best way to proceed!  In particular, =dvc=
-   offers an api which permits one to "stream" or cache files, leaving
-   your storage local to the working repository free of big data
-   files.
-
-   To illustrate,
-   #+begin_src python
-     import dvc.api
-     import pandas as pd
-
-     with dvc.api.open('BigRemoteFile.dta',mode='rb') as dta:
-         df = pd.read_stata(dta)
-   #+end_src
-   This will result in a =pandas.DataFrame= in RAM, but will use no
-   additional disk (except that, depending on what's being used as the
-   dvc store, the file may actually be stored in =.dvc/cache=; this
-   cache can be cleared with =dvc gc=).
-
-** Pulling dvc files
-   If you need the actual file instead of a "stream" you can instead
-   "pull" the dvc files, using
-   #+begin_src sh
-   dvc pull
-   #+end_src
-   and files should be added from the remote dvc data store to your
-   working repository. 
-
-* Adding New Data
-** Additional S3 Credentials
-Write access to the remote s3 repository requires additional credentials; contact =ligon@berkeley.edu= to obtain these.
-
-** Procedure to Add Data
-   To add a new LSMS-style survey to the repo, you'll follow the
-   following steps.  Here we give the example of adding a 2015--16
-   survey from Uganda, obtained from
-   https://microdata.worldbank.org/index.php/catalog/3460.  The same
-   steps should work for you /mutatis mutandis/:
-
-  1. Create a directory corresponding to the country or area; e.g., 
-     #+begin_src sh
-     mkdir Uganda
-     #+end_src
-  2. Create a /sub/-directory indicating the time period for the
-     survey; e.g., 
-     #+begin_src sh
-     mkdir Uganda/2015-16
-     #+end_src
-  3. Create a =Documentation= sub-directory for each survey; e.g.,
-     #+begin_src sh
-     mkdir Uganda/2015-16/Documentation
-     #+end_src
-     In this directory include the following files:
-     - SOURCE :: A text file giving both a url (if available) and
-       citation information for the dataset.
-     - LICENSE :: A text file containing a description of the license
-       or other terms under which you've obtained the data.
-  4. Add other documentation useful for understanding the data to the
-     =Documentation= sub-directory.
-
-  5. Add all the contents of the =Documentation= folder to the =git= repo;
-     e.g., 
-     #+begin_src sh
-     cd ./Uganda/2015-16/Documentation
-     git add .
-     git commit -m"Add Uganda 2015-16 documentation to repo."
-     git push
-     #+end_src
-
-  6. Create a =Data= sub-directory for each survey; e.g.,
-     #+begin_src sh
-     mkdir Uganda/2015-16/Data
-     #+end_src
-
-  7. Obtain a copy of the data you're interested in, perhaps as a zip
-     file or other archive.  Store this in some temporary place, and
-     unzip (or whatever) the files into the relevant Country/Year/Data
-     directory, taking care to preserve any useful directory structure
-     in the archive.  E.g.,
-     #+begin_src sh
-     cd Uganda/2015-16 && unzip -j /tmp/UGA_2015_UNPS_v01_M_STATA8.zip
-     #+end_src
-  8. Add the data you've unarchived to =dvc=, then add the /pointers/
-     (i.e., files with a .dvc extension to git).  For the Uganda case we assume that
-     all the relevant data comes in the form of =stata= *.dta files,
-     since this is what we downloaded from the World Bank.  For example,
-     #+begin_src sh
-     cd ../Data
-     dvc add *.dta
-     git commit -m"Add Uganda/2015-16/Data/*.dta files to dvc store."
-     git pull && git push
-     #+end_src
-  9. Push the data files to the dvc store. Make sure you have good
-     internet connection!  Then a simple
-     #+begin_src sh
-     dvc push
-     #+end_src
-     will copy the data to the remote data store.  NB: If this is the
-     first time you've done this for this repository, then you'll
-     first need to jump through some simple hoops to authenticate with
-     gdrive.
-  10. With the files pushed to the dvc store, you won't need them
-      locally anymore, so you can do something like
-      #+begin_src sh
-      cd ../Data && rm *.dta
-      #+end_src
-      or (if you have a more complex directory structure) perhaps
-      #+begin_src sh
-      find ../Data -name \*.dta -exec rm \{\} \;
-      #+end_src
-