syspop-v2 0.4.0__tar.gz

syspop_v2-0.4.0/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: syspop_v2
+Version: 0.4.0
+Summary: A package for stochastic population imputation
+Author: Sijin Zhang
+Author-email: Sijin Zhang <zsjzyhzp@gmail.com>
+Project-URL: Homepage, https://github.com/jzanetti/Syspop
+Project-URL: Bug Tracker, https://github.com/jzanetti/Syspop/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: matplotlib
+Requires-Dist: pyyaml
+Requires-Dist: pyarrow
+Dynamic: author
+Dynamic: requires-python
+
+# Stochastic Impute: Synthetic Data Generation Engine
+
+A Python utility designed to generate an **integreated synthetic unit record population data** using aggregated public data sources (e.g., from [Stats NZ Data Explorer](https://explore.data.stats.govt.nz)).
+
+This tool is optimized for large-scale microdata generation where individual attributes are assigned based on conditional probability distributions.
+
+The New Zealand unit records of synthetic population can be explored at [here](https://jzanetti1985.pythonanywhere.com)
+
+---
+
+## 🚀 Key Features
+
+* **Synthetic Unit Record Generation**: Transforms multiple aggregated data sources into a unified, granular unit-record dataset.
+* **Dynamic Column Matching**: Automatically identifies shared features between the base population and reference data.
+* **Missingness-Aware Logic**: Handles rows with `NaN` values by dynamically re-calculating probabilities based only on the available non-null features.
+* **Stochastic Selection**: Uses weighted random sampling to preserve the natural variance and distribution of the source data.
+* **Optimized Performance**: Processes millions rows in seconds by grouping identical "missingness patterns" rather than iterating row-by-row.
+
+> Please see the [FAQ](#-faq) for more details.
+
+---
+
+## 📋 Data Requirements
+
+### 1. Population Seed
+The starting point for your synthetic data. 
+* Contain existing columns you wish to "expand".
+
+### 2. Reference Distributions
+Each entry in the dictionary must be a DataFrame containing:
+* **Shared Features**: (e.g., `age`, `location`) to match against the seed.
+* **Target Column**: The attribute being generated.
+
+---
+
+## 💻 Example Usage
+
+```python
+from pandas import DataFrame
+from numpy import nan
+from syspop.model.stochastic_impute import stochastic_impute
+from syspop.postp.vis import plot_distribution
+
+# ---------------------------------
+# 1. Define base aggregated population data (e.g., from a census)
+#    For example, a total of 50 + 60 + 70 people in this example
+# ---------------------------------
+base_population_data = DataFrame(
+    {
+        "gender": [1, 2, 1],
+        "age": [25, 30, 40],
+        "value": [50, 60, 70],
+    }
+)
+
+# ---------------------------------
+# 2. Define reference aggregated data (e.g., Work Status distribution)
+#    For example, a total of 8 + 2 + 6 + 4 people in this reference data
+# ---------------------------------
+income_data = DataFrame(
+    {
+        "gender": [1, 1, 2, 2],
+        "age": [25, 30, 25, nan],
+        "work_status": [1, 2, 1, 2],
+        "income": [50000, 60000, 55000, 45000],
+        "value": [8, 2, 6, 4],
+    }
+)
+
+# ---------------------------------
+# 3. Combine data into a dictionary for the imputation process
+# ---------------------------------
+data = {"seed": base_population_data, "income": income_data}
+
+# -------------------------------------------------------------------------
+# 4. Imputation Task Configuration
+#
+# OBJECTIVE:
+#   Define imputation models where 'features' (age, gender) 
+#   predict 'targets' (work_status and income).
+#
+# HANDLING SAMPLE SIZE DISCREPANCIES (Sparsity Alignment):
+#   The source datasets (e.g., income) sometimes have smaller populations 
+#   than the seed population (180 records). To maintain statistical 
+#   consistency, we introduce NaNs into the output for matching 
+#   reference data in 'drop_list'. 
+#
+#   Example: If the income dataset only contains 20 records, we randomly 
+#   retain only 20 income/work_status in the output population and set the remaining 
+#   160 to NaN
+# -------------------------------------------------------------------------
+task_list = {
+    "income": {
+        "targets": {"work_status": "category", "income": "numeric"},
+        "features": ["age", "gender"],
+    }
+}
+drop_list = ["income"]
+# ---------------------------------
+# 5. Run the stochastic imputation process
+# ---------------------------------
+syn_pop = stochastic_impute(data, task_list)
+
+# ---------------------------------
+# 6. Plot distribution
+# ---------------------------------
+# 6.1 Plot distribution for age
+plot_distribution(syn_pop, ["age"])
+# 6.2 Plot joint distribution for gender + age
+plot_distribution(syn_pop, ["gender", "age"])
+```
+
+<a name="faq"></a>
+## 🧠 FAQ
+### Is generating synthetic unit-record data in this way actually accurate?
+Well, it depends on your use case and the quality of your inputs. Ideally, you should work with real unit-record data. However, in practice, this isn't always feasible (i.e., in New Zealand, if you live far away from a Stats NZ IDI data lab). This utility provides a method to statistically link different aggregated, published population benchmarks together. This allows you to build out full data pipelines and prototype products locally before taking your code into a restricted environment.
+
+### Maximizing accuracy?
+The accuracy of the synthetic output depends heavily on task design. The more shared variables (covariates) present in your reference data to condition the probabilities, the closer the synthetic distribution will mirror reality.
+
+Also, you can run the process multiple times to capture inherent uncertainties.
+
+### Are we doing any prediction modelling here ?
+It is out of scope at the moment. If certain covariate values are missing from the reference data to condition the probabilities, the process simply ignores those missing values when linking the reference data to the seed data (for example, if the reference data does not contain income information for children, when integrating the reference data into the seed population data, the integrated data will just set the income for children as NaN). The reason is that many covariates in these datasets are categorical, and applying simple prediction models can struggle to capture the nuances and introduce unwanted noise or uncertainty into the output data. However, you are welcome to apply your own predictive models to handle missing data prior to running this process if your use case requires it.

syspop_v2-0.4.0/README.md

@@ -0,0 +1,124 @@
+# Stochastic Impute: Synthetic Data Generation Engine
+
+A Python utility designed to generate an **integreated synthetic unit record population data** using aggregated public data sources (e.g., from [Stats NZ Data Explorer](https://explore.data.stats.govt.nz)).
+
+This tool is optimized for large-scale microdata generation where individual attributes are assigned based on conditional probability distributions.
+
+The New Zealand unit records of synthetic population can be explored at [here](https://jzanetti1985.pythonanywhere.com)
+
+---
+
+## 🚀 Key Features
+
+* **Synthetic Unit Record Generation**: Transforms multiple aggregated data sources into a unified, granular unit-record dataset.
+* **Dynamic Column Matching**: Automatically identifies shared features between the base population and reference data.
+* **Missingness-Aware Logic**: Handles rows with `NaN` values by dynamically re-calculating probabilities based only on the available non-null features.
+* **Stochastic Selection**: Uses weighted random sampling to preserve the natural variance and distribution of the source data.
+* **Optimized Performance**: Processes millions rows in seconds by grouping identical "missingness patterns" rather than iterating row-by-row.
+
+> Please see the [FAQ](#-faq) for more details.
+
+---
+
+## 📋 Data Requirements
+
+### 1. Population Seed
+The starting point for your synthetic data. 
+* Contain existing columns you wish to "expand".
+
+### 2. Reference Distributions
+Each entry in the dictionary must be a DataFrame containing:
+* **Shared Features**: (e.g., `age`, `location`) to match against the seed.
+* **Target Column**: The attribute being generated.
+
+---
+
+## 💻 Example Usage
+
+```python
+from pandas import DataFrame
+from numpy import nan
+from syspop.model.stochastic_impute import stochastic_impute
+from syspop.postp.vis import plot_distribution
+
+# ---------------------------------
+# 1. Define base aggregated population data (e.g., from a census)
+#    For example, a total of 50 + 60 + 70 people in this example
+# ---------------------------------
+base_population_data = DataFrame(
+    {
+        "gender": [1, 2, 1],
+        "age": [25, 30, 40],
+        "value": [50, 60, 70],
+    }
+)
+
+# ---------------------------------
+# 2. Define reference aggregated data (e.g., Work Status distribution)
+#    For example, a total of 8 + 2 + 6 + 4 people in this reference data
+# ---------------------------------
+income_data = DataFrame(
+    {
+        "gender": [1, 1, 2, 2],
+        "age": [25, 30, 25, nan],
+        "work_status": [1, 2, 1, 2],
+        "income": [50000, 60000, 55000, 45000],
+        "value": [8, 2, 6, 4],
+    }
+)
+
+# ---------------------------------
+# 3. Combine data into a dictionary for the imputation process
+# ---------------------------------
+data = {"seed": base_population_data, "income": income_data}
+
+# -------------------------------------------------------------------------
+# 4. Imputation Task Configuration
+#
+# OBJECTIVE:
+#   Define imputation models where 'features' (age, gender) 
+#   predict 'targets' (work_status and income).
+#
+# HANDLING SAMPLE SIZE DISCREPANCIES (Sparsity Alignment):
+#   The source datasets (e.g., income) sometimes have smaller populations 
+#   than the seed population (180 records). To maintain statistical 
+#   consistency, we introduce NaNs into the output for matching 
+#   reference data in 'drop_list'. 
+#
+#   Example: If the income dataset only contains 20 records, we randomly 
+#   retain only 20 income/work_status in the output population and set the remaining 
+#   160 to NaN
+# -------------------------------------------------------------------------
+task_list = {
+    "income": {
+        "targets": {"work_status": "category", "income": "numeric"},
+        "features": ["age", "gender"],
+    }
+}
+drop_list = ["income"]
+# ---------------------------------
+# 5. Run the stochastic imputation process
+# ---------------------------------
+syn_pop = stochastic_impute(data, task_list)
+
+# ---------------------------------
+# 6. Plot distribution
+# ---------------------------------
+# 6.1 Plot distribution for age
+plot_distribution(syn_pop, ["age"])
+# 6.2 Plot joint distribution for gender + age
+plot_distribution(syn_pop, ["gender", "age"])
+```
+
+<a name="faq"></a>
+## 🧠 FAQ
+### Is generating synthetic unit-record data in this way actually accurate?
+Well, it depends on your use case and the quality of your inputs. Ideally, you should work with real unit-record data. However, in practice, this isn't always feasible (i.e., in New Zealand, if you live far away from a Stats NZ IDI data lab). This utility provides a method to statistically link different aggregated, published population benchmarks together. This allows you to build out full data pipelines and prototype products locally before taking your code into a restricted environment.
+
+### Maximizing accuracy?
+The accuracy of the synthetic output depends heavily on task design. The more shared variables (covariates) present in your reference data to condition the probabilities, the closer the synthetic distribution will mirror reality.
+
+Also, you can run the process multiple times to capture inherent uncertainties.
+
+### Are we doing any prediction modelling here ?
+It is out of scope at the moment. If certain covariate values are missing from the reference data to condition the probabilities, the process simply ignores those missing values when linking the reference data to the seed data (for example, if the reference data does not contain income information for children, when integrating the reference data into the seed population data, the integrated data will just set the income for children as NaN). The reason is that many covariates in these datasets are categorical, and applying simple prediction models can struggle to capture the nuances and introduce unwanted noise or uncertainty into the output data. However, you are welcome to apply your own predictive models to handle missing data prior to running this process if your use case requires it.

syspop_v2-0.4.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "syspop_v2"
+version = "0.4.0"
+authors = [
+  { name="Sijin Zhang", email="zsjzyhzp@gmail.com" },
+]
+description = "A package for stochastic population imputation"
+readme = "README.md"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+# List the external libraries your package needs to run
+dependencies = [
+    "pandas",
+    "numpy",
+    "matplotlib",
+    "pyyaml",
+    "pyarrow"
+]
+
+[project.urls]
+"Homepage" = "https://github.com/jzanetti/Syspop"
+"Bug Tracker" = "https://github.com/jzanetti/Syspop/issues"
+
+[tool.setuptools.packages.find]
+where = ["."]          # Tells it to look in the current root directory
+include = ["syspop*"]  # Grabs your 'syspop' folder and everything inside it
+exclude = ["examples*"] # Ensures the 'examples' folder doesn't get published

syspop_v2-0.4.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

syspop_v2-0.4.0/setup.py

@@ -0,0 +1,17 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="syspop_v2",
+    version="0.4.0",  # Or whatever version is appropriate
+    packages=find_packages(),
+    install_requires=[
+        "pandas",
+        "numpy",
+        "pyarrow"
+        "pyyaml",
+        "matplotlib"
+    ],
+    author="Sijin Zhang",
+    description="Creating synthetic population",
+    python_requires='>=3.8',
+)

syspop_v2-0.4.0/syspop/data/data.py

@@ -0,0 +1,93 @@
+from syspop.data.query import obtain_stats_data
+from syspop.data.utils import stats_data_proc
+from pandas import DataFrame as pdDataFrame
+from sklearn.preprocessing import LabelEncoder
+
+
+def obtain_data(cfg: dict, api_key: str):
+    """
+    Obtains and processes population statistics data based on the provided configuration and API key.
+
+    The function performs the following steps:
+    1. Fetches raw statistics data using the API configuration and key.
+    2. Processes the raw data according to the configuration.
+    3. Applies inclusion and exclusion filters to the data based on specified criteria.
+    4. Maps the filtered data to the desired columns.
+    5. Expands the DataFrame by repeating rows according to the 'value' column.
+    6. Returns the final DataFrame with the 'value' column removed and index reset.
+
+    Args:
+        cfg (dict): Configuration dictionary containing API settings, mapping of column names,
+            inclusion and exclusion criteria.
+        api_key (str): API key for accessing the statistics data.
+
+    Returns:
+        pandas.DataFrame: Processed and filtered DataFrame with population statistics.
+    """
+
+    def _obtain_qc_key(cfg, proc_qc_type):
+        if proc_qc_type in cfg["map"]:
+            return cfg["map"][proc_qc_type]
+        return proc_qc_type
+
+    data_pop = obtain_stats_data(cfg["api"], api_key=api_key)
+    data_pop = stats_data_proc(data_pop, cfg)
+
+    try:
+        for proc_qc_type in cfg["inclusion"]:
+            proc_qc_key = _obtain_qc_key(cfg, proc_qc_type)
+            data_pop = data_pop[
+                data_pop[proc_qc_key].isin(cfg["inclusion"][proc_qc_type])
+            ]
+    except TypeError:
+        pass
+
+    try:
+        for proc_qc_type in cfg["exclusion"]:
+            proc_qc_key = _obtain_qc_key(cfg, proc_qc_type)
+            data_pop = data_pop[
+                ~data_pop[proc_qc_key].isin(cfg["exclusion"][proc_qc_type])
+            ]
+    except TypeError:
+        pass
+
+    df = data_pop[list(cfg["map"].values())]
+
+    return df
+
+
+def encode_weights(data_dict) -> dict:
+
+    for key in data_dict:
+
+        run_repeat = False
+        if key == "seed":
+            run_repeat = True
+
+        df = data_dict[key]
+
+        if run_repeat:
+            df = df.loc[df.index.repeat(df["value"])].copy()
+            df = df.reset_index(drop=True).drop(columns=["value"])
+        else:
+            group_cols = df.columns.drop("value").tolist()
+            df_grouped = df.groupby(group_cols, as_index=False)["value"].sum()
+            df_grouped["probability"] = df_grouped["value"] / df_grouped["value"].sum()
+            df = df_grouped.reset_index(drop=True).drop(columns=["value"])
+
+        data_dict[key] = df
+
+    return data_dict
+
+
+def encode_sum(data_dict, drop_list) -> dict:
+    
+    result_dict = {}
+
+    for key in data_dict:
+
+        if key not in drop_list:
+            continue
+        result_dict[key] = data_dict[key]["value"].sum()
+
+    return result_dict

syspop_v2-0.4.0/syspop/data/query.py

@@ -0,0 +1,73 @@
+from requests import get, exceptions
+import xml.etree.ElementTree as ET
+from pandas import DataFrame
+from pandas import to_numeric
+
+
+def stats_data_proc(data):
+    data["value"] = to_numeric(data["value"], errors="coerce")
+    data = data.dropna()
+    data["value"] = data["value"].astype(int)
+    return data
+
+
+def obtain_stats_data(api_url: str, api_key: str or None = None):
+
+    if api_key is None:
+        raise Exception("No proper Stats API is provided")
+
+    # Set headers with the API key
+    headers = {"Ocp-Apim-Subscription-Key": api_key}
+    # Make the GET request
+    try:
+        response = get(api_url, headers=headers)
+        response.raise_for_status()  # Check for HTTP errors
+    except exceptions.HTTPError as e:
+        print(f"HTTP Error: {e}")
+        if response.status_code == 401:
+            print(
+                "Authentication failed. Please check your API key or obtain a valid one from https://api.data.stats.govt.nz/"
+            )
+        elif response.status_code == 400:
+            print(
+                "Bad Request. The URL or dimensions may be invalid. Check the API documentation or simplify the query."
+            )
+        print(f"Raw response: {response.text}")
+        exit()
+    except exceptions.RequestException as e:
+        print(f"Error fetching data: {e}")
+        exit()
+
+    # Check if response is empty
+    if not response.text:
+        print("Error: Empty response received from the API.")
+        exit()
+    xml_text = response.text  # your XML string
+
+    # Parse XML
+    root = ET.fromstring(xml_text)
+
+    # Namespaces
+    ns = {
+        "generic": "http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/generic",
+        "message": "http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message",
+    }
+
+    rows = []
+    # Iterate over all Obs elements
+    for obs in root.findall(".//generic:Obs", ns):
+        row = {}
+
+        # Extract all key/value pairs from ObsKey
+        for val in obs.findall("./generic:ObsKey/generic:Value", ns):
+            row[val.attrib["id"]] = val.attrib["value"]
+
+        # Extract the observed value
+        obs_value = obs.find("./generic:ObsValue", ns)
+        if obs_value is not None:
+            row["OBS_VALUE"] = obs_value.attrib["value"]
+
+        rows.append(row)
+
+    # Convert to DataFrame
+    return DataFrame(rows)

syspop_v2-0.4.0/syspop/data/sample.py

@@ -0,0 +1,80 @@
+from process.data.data import obtain_data
+from logging import info as log_info
+from etc.sample_data.api_keys import STATS_API
+from yaml import safe_load
+from pickle import dump as pickle_dump
+from pickle import load as pickle_load
+from process.model.utils import obtain_all_tasks
+
+
+def obtain_sample_data_cfg():
+    with open("etc/sample_data/sample_data_cfg.yml", "r") as fid:
+        cfg = safe_load(fid)
+
+    return cfg["tables"]
+
+
+def obtain_sample_api_key(api_key: str or None = None):
+    if api_key is None:
+        api_key = STATS_API
+
+    return api_key
+
+
+def load_sample_data(
+    data_types: list = [
+        "seed",
+        "industry",
+        "occupation",
+        "occupation_income",
+        "industry_income",
+        "travel_to_work",
+        "work_hours",
+    ],
+    refresh: bool = False,
+):
+    """
+    Wrapper function to retrieve data for specified types using an API key.
+
+        cfg_data (dict): Dictionary containing configuration for each data type,
+            where keys are data type strings and values are configuration dicts.
+        api_key (str or None, optional): API key to use for data retrieval. If None,
+            the key will be obtained via `obtain_sample_api_key`. Defaults to None.
+        data_types (list, optional): List of data type strings to retrieve.
+            Defaults to ["pop"].
+
+        dict: A dictionary mapping each data type (from `data_types`) to its
+            corresponding data retrieved using the API.
+
+    Raises:
+        KeyError: If a specified data type is not present in `cfg_data`.
+        Exception: Propagates exceptions raised during API key retrieval or data fetching.
+
+    Example:
+        >>> cfg_data = {"seed": {...}, "income": {...}}
+        >>> result = obtain_data_wrapper(cfg_data, api_key="my_key", data_types=["pop", "income"])
+        >>> print(result)
+        {'seed': <pop_data>, 'income': <income_data>}
+    """
+
+    if not refresh:
+        data_dict = pickle_load(open("etc/sample_data/sample_data.pkl", "rb"))
+    else:
+        api_key = obtain_sample_api_key()
+        cfg_data = obtain_sample_data_cfg()
+
+        data_dict = {}
+        for data_type in data_types:
+
+            log_info(f"Obtaining data for type: {data_type}")
+
+            data_dict[data_type] = obtain_data(cfg_data[data_type], api_key)
+
+        pickle_dump(data_dict, open("etc/sample_data/sample_data.pkl", "wb"))
+
+    with open("etc/sample_data/sample_model_cfg.yml", "r") as fid:
+        model_cfg = safe_load(fid)
+
+    task_list = obtain_all_tasks(model_cfg["tasks"], model_cfg["cfg"])
+
+    return data_dict, task_list

syspop_v2-0.4.0/syspop/data/utils.py

@@ -0,0 +1,81 @@
+from pandas import to_numeric
+from pandas import DataFrame
+
+
+def stats_data_proc(data: DataFrame, cfg: dict):
+
+    data = data.rename(columns=cfg["map"])
+
+    data["value"] = to_numeric(data["value"], errors="coerce")
+    data = data.dropna()
+    data["value"] = data["value"].astype(int)
+
+    return data
+
+
+def check_data_consistency(
+    data_dict: dict,
+    check_err: bool = True,
+    throw_err: bool = False,
+    output_dir: str or None = None,
+):
+    rows = []
+
+    for key, df in data_dict.items():
+        for col in df.columns:
+            unique_list = df[col].unique()
+            unique_str = ", ".join(map(str, unique_list))
+            unique_str = [item.strip() for item in unique_str.split(",")]
+            unique_str.sort()
+            unique_str = ", ".join(unique_str)
+
+            unique_count = len(unique_list)
+            rows.append(
+                {
+                    "data_key": key,
+                    "cols": col,
+                    "unique_count": unique_count,
+                    "unique_value": unique_str,
+                }
+            )
+
+    summary_df = DataFrame(rows)
+    if output_dir is not None:
+        summary_df.to_csv(f"{output_dir}/data_consistency.csv", index=False)
+
+    if check_err:
+        all_cols = summary_df["cols"].unique()
+
+        for proc_col in all_cols:
+            proc_series = summary_df[summary_df["cols"] == proc_col]["unique_value"]
+
+            proc_series = [set(val.split(", ")) for val in proc_series]
+
+            all_same = all(s == proc_series[0] for s in proc_series)
+
+            if not all_same:
+                print(
+                    summary_df[summary_df["cols"] == proc_col][
+                        ["data_key", "cols", "unique_value"]
+                    ]
+                )
+
+                if throw_err:
+                    raise ValueError(
+                        f"Column '{proc_col}' has different unique values across datasets."
+                    )
+
+                # Ask the user if they want to continue
+                user_choice = (
+                    input(
+                        f"\nColumn '{proc_col}' has different unique values "
+                        + "across datasets. Continue? [Y/N]: "
+                    )
+                    .strip()
+                    .upper()
+                )
+                if user_choice != "Y":
+                    raise ValueError(
+                        f"Execution halted by user. Column '{proc_col}' has "
+                        + "different unique values across datasets."
+                    )

syspop_v2-0.4.0/syspop/model/__init__.py

@@ -0,0 +1 @@
+MODEL_TRAINING_TEST_RATIO = 0.1