healthdata 0.1.0__tar.gz

healthdata-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ismail Elouafiq
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

healthdata-0.1.0/MANIFEST.in

@@ -0,0 +1 @@
+exclude tests/test_geo*.py

healthdata-0.1.0/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: healthdata
+Version: 0.1.0
+Summary: Download and read NHANES health survey data.
+Author-email: Ismail Elouafiq <contact@ismail.bio>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/nidhog/healthdata
+Project-URL: Repository, https://github.com/nidhog/healthdata
+Project-URL: Documentation, https://github.com/nidhog/healthdata/tree/main/docs/nhanes
+Project-URL: Issues, https://github.com/nidhog/healthdata/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: tqdm
+Dynamic: license-file
+
+# Health Data Downloader Package
+The Health Data Downloader is a Python package that makes it easy to download and read real world health data.
+
+Current data sources supported:
+* **NHANES** National Health and Nutrition Examination Survey (CDC)
+## The NHANES Submodule
+If you want to download health data from NHANES this module can help you not only download the data but also add documentation from the official NHANES website, and read this data into a pandas dataframe.
+
+[NHANES](https://www.cdc.gov/nchs/nhanes/about/) is the National Health and Nutrition Examination Survey which collects data about the health of adults and children in the United States.
+
+_Note: This is not an official repository from the NHANES project nor the CDC_
+
+This project provides workflows that enable you to:
+- discover and download datasets from NHANES by component (meaning what type of data like Laboratory data, examination etc.) and by cycle (which years)
+- read the XPT files into a pandas DataFrames
+- extract NHANES variable documentation into CSV files and rename variable codes into human-readable names (such as `heart_rate` instead of `LF321`)
+- search and summarize local files if they are already downloaded
+
+## Quick start
+In thie quick start we will:
+1. Install required packages.
+2. Run a download.
+3. Read and inspect data.
+
+```bash
+pip install healthdata
+```
+You can choose which `components` you want to download (such as Demographics, Laboratory etc.) and which years (can be either "2017-2020" or "2016"). This will also download documentation about these fields from the NHANES website to add them when the data is read.
+If the data is already downloaded in the output directory the module will log this and skip the download.
+```python
+from healthdata.nhanes.downloader import download_nhanes_data
+
+download_nhanes_data(
+    components=["Demographics", "Examination"],
+    output_dir="nhanes_data",
+    years=["2017-2018"],
+)
+```
+
+You can read the data you already downloaded (you will be prompted to download it if it doesn't exist)
+```python
+from healthdata.nhanes.reader import read_nhanes_data, search_nhanes_local_data
+
+df = read_nhanes_data("nhanes_data/2017-2018/2017/Demographics/DEMO_J.XPT")
+print(df.head())
+
+inventory = search_nhanes_local_data(search_dir="nhanes_data")
+print(inventory.head())
+```
+That's all folks! Now you will see that your dataframe already includes documentation.
+## Current caveats
+- HTML codebook parsing depends on current NHANES page text patterns. CDC page structure changes may require parser updates.
+- Network errors and partial downloads are not retried automatically.
+
+## Upcoming Additions
+Planned additions include NCBI GEO and other open health and biomedical datasets. The current release supports NHANES only.
+
+## License
+This project is licensed under the [MIT License](LICENSE). Commercial use is permitted under its terms. Downloaded datasets may be subject to separate terms set by their respective providers.
+
+For paid support or consulting, contact the author at contact@ismail.bio.
+
+## Author
+[Ismail Elouafiq](https://ismail.bio)
+
+Contact: contact@ismail.bio

healthdata-0.1.0/README.md

@@ -0,0 +1,66 @@
+# Health Data Downloader Package
+The Health Data Downloader is a Python package that makes it easy to download and read real world health data.
+
+Current data sources supported:
+* **NHANES** National Health and Nutrition Examination Survey (CDC)
+## The NHANES Submodule
+If you want to download health data from NHANES this module can help you not only download the data but also add documentation from the official NHANES website, and read this data into a pandas dataframe.
+
+[NHANES](https://www.cdc.gov/nchs/nhanes/about/) is the National Health and Nutrition Examination Survey which collects data about the health of adults and children in the United States.
+
+_Note: This is not an official repository from the NHANES project nor the CDC_
+
+This project provides workflows that enable you to:
+- discover and download datasets from NHANES by component (meaning what type of data like Laboratory data, examination etc.) and by cycle (which years)
+- read the XPT files into a pandas DataFrames
+- extract NHANES variable documentation into CSV files and rename variable codes into human-readable names (such as `heart_rate` instead of `LF321`)
+- search and summarize local files if they are already downloaded
+
+## Quick start
+In thie quick start we will:
+1. Install required packages.
+2. Run a download.
+3. Read and inspect data.
+
+```bash
+pip install healthdata
+```
+You can choose which `components` you want to download (such as Demographics, Laboratory etc.) and which years (can be either "2017-2020" or "2016"). This will also download documentation about these fields from the NHANES website to add them when the data is read.
+If the data is already downloaded in the output directory the module will log this and skip the download.
+```python
+from healthdata.nhanes.downloader import download_nhanes_data
+
+download_nhanes_data(
+    components=["Demographics", "Examination"],
+    output_dir="nhanes_data",
+    years=["2017-2018"],
+)
+```
+
+You can read the data you already downloaded (you will be prompted to download it if it doesn't exist)
+```python
+from healthdata.nhanes.reader import read_nhanes_data, search_nhanes_local_data
+
+df = read_nhanes_data("nhanes_data/2017-2018/2017/Demographics/DEMO_J.XPT")
+print(df.head())
+
+inventory = search_nhanes_local_data(search_dir="nhanes_data")
+print(inventory.head())
+```
+That's all folks! Now you will see that your dataframe already includes documentation.
+## Current caveats
+- HTML codebook parsing depends on current NHANES page text patterns. CDC page structure changes may require parser updates.
+- Network errors and partial downloads are not retried automatically.
+
+## Upcoming Additions
+Planned additions include NCBI GEO and other open health and biomedical datasets. The current release supports NHANES only.
+
+## License
+This project is licensed under the [MIT License](LICENSE). Commercial use is permitted under its terms. Downloaded datasets may be subject to separate terms set by their respective providers.
+
+For paid support or consulting, contact the author at contact@ismail.bio.
+
+## Author
+[Ismail Elouafiq](https://ismail.bio)
+
+Contact: contact@ismail.bio

healthdata-0.1.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "healthdata"
+version = "0.1.0"
+description = "Download and read NHANES health survey data."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Ismail Elouafiq", email = "contact@ismail.bio" },
+]
+requires-python = ">=3.14"
+dependencies = [
+    "pandas",
+    "requests",
+    "beautifulsoup4",
+    "tqdm",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering :: Medical Science Apps.",
+]
+
+[project.urls]
+Homepage = "https://github.com/nidhog/healthdata"
+Repository = "https://github.com/nidhog/healthdata"
+Documentation = "https://github.com/nidhog/healthdata/tree/main/docs/nhanes"
+Issues = "https://github.com/nidhog/healthdata/issues"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+packages = ["healthdata", "healthdata.nhanes"]
+
+[tool.setuptools.package-data]
+healthdata = []
+"healthdata.nhanes" = []

healthdata-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

healthdata-0.1.0/src/healthdata/__init__.py

@@ -0,0 +1,10 @@
+"""healthdata package
+
+Provides tools to: download, read and preprocess health datasets.
+It is specific to open health datasets.
+Dataset implementations are under separate modules, such as ``healthdata.nhanes``
+"""
+
+from . import nhanes
+
+__all__ = ["nhanes"]

healthdata-0.1.0/src/healthdata/nhanes/__init__.py

@@ -0,0 +1,17 @@
+"""NHANES submodule for the healthdata package."""
+
+from .downloader import (
+    download_and_extract_docs,
+    download_nhanes_data,
+    get_file_links_for_component,
+)
+from .reader import get_column_doc, read_nhanes_data, search_nhanes_local_data
+
+__all__ = [
+    "download_nhanes_data",
+    "download_and_extract_docs",
+    "get_file_links_for_component",
+    "read_nhanes_data",
+    "get_column_doc",
+    "search_nhanes_local_data",
+]

healthdata-0.1.0/src/healthdata/nhanes/downloader.py

@@ -0,0 +1,282 @@
+import csv
+import logging
+import os
+import re
+import json
+from typing import List, Optional, Union
+from urllib.parse import urlparse
+
+import requests
+from bs4 import BeautifulSoup
+from tqdm import tqdm
+
+from .settings import DEFAULT_BASE_DOMAIN, DEFAULT_BASE_SEARCH_URL, DEFAULT_COMPONENTS
+
+logger = logging.getLogger(__name__)
+
+
+def ensure_dir(path: str):
+    if not os.path.exists(path):
+        os.makedirs(path)
+
+
+def extract_date_from_url(url: str) -> str:
+    """
+    Extract NHANES cycle or year from url patterns.
+    The patterns we look for are either "YYYY-YYYY" or "/YYYY/".
+    Returns 'nodate' if none found.
+    """
+    match = re.search(r"\d{4}-\d{4}", url)
+    if match:
+        return match.group(0)
+    match = re.search(r"/(19|20)\d{2}/", url)
+    if match:
+        return match.group(0).strip("/")
+    return "nodate"
+
+
+def download_file(url: str, out_path: str):
+    """Download an XPT file"""
+    r = requests.get(url, stream=True)
+    r.raise_for_status()
+    total = int(r.headers.get("content-length", 0))
+
+    with open(out_path, "wb") as f, tqdm(
+        desc=os.path.basename(out_path),
+        total=total if total > 0 else None,
+        unit="B",
+        unit_scale=True,
+    ) as bar:
+        for chunk in r.iter_content(chunk_size=8192):
+            if not chunk:
+                continue
+            f.write(chunk)
+            if total > 0:
+                bar.update(len(chunk))
+
+
+def parse_nhanes_doc_variables(doc_html: str, doc_url: str):
+    """
+    Parse NHANES doc HTML into: variable_name, label, description, value_meanings, link.
+
+    This extracts metadata that is not available in the XPT files.
+
+    TODO: add patterns as config instead?
+    """
+    soup = BeautifulSoup(doc_html, "html.parser")
+    text = soup.get_text("\n")
+    text = re.sub(r"\r", "", text)
+    text = re.sub(r"[ \t]+", " ", text)
+
+    rows = []
+    chunks = re.split(r"(?=\bVariable Name:\s*[A-Za-z0-9_]+)", text)
+    for chunk in chunks:
+        chunk = chunk.strip()
+        if not chunk.startswith("Variable Name:"):
+            continue
+
+        var_match = re.search(r"Variable Name:\s*(?P<var>[A-Za-z0-9_]+)", chunk)
+        if not var_match:
+            continue
+        var = var_match.group("var").strip()
+
+        label_match = re.search(
+            r"SAS Label:\s*(?P<label>.*?)(?=\n\s*English Text:|\n\s*English Instructions:|\n\s*Target:|\n\s*Code or Value|\Z)",
+            chunk,
+            re.DOTALL,
+        )
+        desc_match = re.search(
+            r"English Text:\s*(?P<desc>.*?)(?=\n\s*English Instructions:|\n\s*Target:|\n\s*Code or Value|\Z)",
+            chunk,
+            re.DOTALL,
+        )
+
+        label = re.sub(r"\s+", " ", label_match.group("label")).strip() if label_match else ""
+        desc = re.sub(r"\s+", " ", desc_match.group("desc")).strip() if desc_match else ""
+
+        value_meanings = []
+        value_section = re.search(
+            r"Code or Value\s*(?P<section>.*?)(?=\n\s*Variable Name:|\Z)",
+            chunk,
+            re.DOTALL,
+        )
+        if value_section:
+            section_text = value_section.group("section")
+            # Common NHANES rows are represented as: code whitespace value-description.
+            for line in [ln.strip() for ln in section_text.splitlines() if ln.strip()]:
+                if line.lower() in {
+                    "value description",
+                    "unweighted count",
+                    "weighted percent",
+                    "weighted frequency",
+                    "cumulative",
+                    "skip to item",
+                }:
+                    continue
+                m = re.match(r"^(?P<code>[-+A-Za-z0-9\.]+)\s{1,}(?P<meaning>.+)$", line)
+                if not m:
+                    continue
+                value_meanings.append(
+                    {
+                        "code": m.group("code").strip(),
+                        "meaning": re.sub(r"\s+", " ", m.group("meaning")).strip(),
+                    }
+                )
+
+        rows.append(
+            {
+                "variable_name": var,
+                "label": label,
+                "description": desc,
+                "value_meanings": json.dumps(value_meanings, ensure_ascii=False) if value_meanings else "",
+                "link": doc_url,
+            }
+        )
+    return rows
+
+
+def download_and_extract_docs(doc_url: str, out_dir: str):
+    r = requests.get(doc_url)
+    r.raise_for_status()
+    rows = parse_nhanes_doc_variables(r.text, doc_url)
+
+    doc_base = os.path.basename(urlparse(doc_url).path)
+    csv_name = os.path.splitext(doc_base)[0] + "_variables.csv"
+    out_csv = os.path.join(out_dir, csv_name)
+
+    
+    ensure_dir(os.path.dirname(out_csv))
+    with open(out_csv, "w", newline="", encoding="utf-8") as f:
+        w = csv.DictWriter(
+            f,
+            fieldnames=["variable_name", "label", "description", "value_meanings", "link"],
+        )
+        w.writeheader()
+        w.writerows(rows)
+    print(f"> [SAVED] Docs CSV saved: {out_csv}")
+
+
+def get_file_links_for_component(
+    component: str,
+    years: Optional[List[Union[int, str]]] = None,
+    base_search_url: str = DEFAULT_BASE_SEARCH_URL,
+    base_domain: str = DEFAULT_BASE_DOMAIN,
+):
+    """Return dataset links for an NHANES component, filtered by years if provided.
+        This is where most of the logic for parsing the website is defined.
+        TODO: perhaps this can be provided as a function to the downloader
+        in case the website changes its structure in the future?
+    """
+    params = {"Component": component}
+    print(f"> [FETCH] Fetching data page for Component = {component} ...")
+    resp = requests.get(base_search_url, params=params)
+    resp.raise_for_status()
+
+    soup = BeautifulSoup(resp.text, "html.parser")
+    rows = soup.select("table tr")
+    out = []
+
+    years_set = set()
+    if years:
+        for y in years:
+            years_set.add(str(y))
+
+    for tr_field in rows:
+        td_fields = tr_field.find_all("td")
+        if not td_fields:
+            continue
+
+        cycle_str = td_fields[0].get_text(strip=True) if len(td_fields) >= 1 else "unknown"
+
+        if years:
+            cycle_parts = re.findall(r"\d{4}", cycle_str)
+            is_match = False
+            for part in cycle_parts:
+                if part in years_set:
+                    is_match = True
+                    break
+            if cycle_str in years_set:
+                is_match = True
+
+            if not is_match:
+                continue
+
+        doc_url = None
+        xpt_url = None
+        data_file_name = None
+
+        for a in tr_field.select("a[href]"):
+            href = a.get("href", "").strip()
+            if not href:
+                continue
+
+            if href.startswith("http"):
+                full_url = href
+            elif href.startswith("/"):
+                full_url = base_domain + href
+            else:
+                full_url = base_domain + "/" + href.lstrip("/")
+
+            if href.lower().endswith(".htm"):
+                doc_url = full_url
+            elif href.lower().endswith(".xpt"):
+                xpt_url = full_url
+                data_file_name = a.get_text(strip=True) or None
+
+        if xpt_url and doc_url:
+            out.append(
+                {
+                    "years": cycle_str,
+                    "xpt_url": xpt_url,
+                    "doc_url": doc_url,
+                    "data_file_name": data_file_name,
+                }
+            )
+
+    return out
+
+
+def download_nhanes_data(
+    output_dir: str,
+    components: Optional[List[str]] = DEFAULT_COMPONENTS,
+    years: Optional[List[Union[int, str]]] = None,
+    with_docs: bool = True,
+    base_search_url: str = DEFAULT_BASE_SEARCH_URL,
+    base_domain: str = DEFAULT_BASE_DOMAIN,
+):
+    """Download NHANES XPT files and optional docs for components and years."""
+    ensure_dir(output_dir)
+
+    for comp in components:
+        entries = get_file_links_for_component(
+            comp,
+            years=years,
+            base_search_url=base_search_url,
+            base_domain=base_domain,
+        )
+        print(f"> [FOUND] Found {len(entries)} datasets for component '{comp}' (matching years: {years})")
+
+        for e in entries:
+            xpt_url = e["xpt_url"]
+            doc_url = e["doc_url"]
+            cycle = e["years"] or "unknown"
+
+            date = extract_date_from_url(doc_url)
+
+            out_subdir = os.path.join(output_dir, cycle, date, comp)
+            ensure_dir(out_subdir)
+
+            filename = xpt_url.split("/")[-1]
+            out_path = os.path.join(out_subdir, filename)
+
+            if os.path.exists(out_path):
+                print(f"> [CHECK] Already downloaded: {cycle}/{filename}")
+            else:
+                print(f"> [.....] Downloading {cycle}/{filename}")
+                download_file(xpt_url, out_path)
+
+            if with_docs:
+                try:
+                    download_and_extract_docs(doc_url, out_subdir)
+                except Exception as err:
+                    logger.warning("Failed to extract docs for %s (%s)", doc_url, err)

healthdata-0.1.0/src/healthdata/nhanes/reader.py

@@ -0,0 +1,444 @@
+import logging
+import os
+import re
+import json
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+import pandas as pd
+
+from .downloader import download_nhanes_data
+
+logger = logging.getLogger(__name__)
+
+
+def snake_case(s: str) -> str:
+	s = s.strip().lower()
+	s = re.sub(r"[^\w\s]", " ", s)
+	s = re.sub(r"\s+", "_", s)
+	s = re.sub(r"_+", "_", s).strip("_")
+	return s or "unnamed"
+
+
+def read_nhanes_data(
+	file_path: str,
+	use_csv_for_labels: bool = True,
+	local_doc_csv_dir: Optional[str] = None,
+	allow_download_prompt: bool = True,
+	attach_documentation: bool = True,
+	add_documentation_column: bool = False,
+) -> pd.DataFrame:
+	"""
+	Read an NHANES XPT file and optionally rename columns using <dataset>_variables.csv.
+
+	use_csv_for_labels: If True, attempts to find a corresponding <dataset>_variables.csv file
+		and uses it to rename columns based on the 'label' field.
+		These can be downloaded using the `download_nhanes_data` function with `with_docs=True`
+	attach_documentation: If True, attach per-column documentation metadata under
+		df.attrs['documentation'] and a docs table under df.attrs['documentation_df'].
+	add_documentation_column: If True, add a 'documentation' column where each row
+		contains the same per-column documentation dictionary.
+	"""
+	if not os.path.exists(file_path):
+		if allow_download_prompt:
+			parts = file_path.split(os.sep)
+			if len(parts) >= 4:
+				component_candidate = parts[-2]
+				cycle_candidate = parts[-4]
+
+				if re.match(r"\d{4}-\d{4}", cycle_candidate):
+					print(f"File not found: {file_path}")
+					print(
+						"It looks like you're missing data for "
+						f"Component='{component_candidate}', Cycle='{cycle_candidate}'"
+					)
+					response = input("Would you like to try downloading it now? [y/N] ").lower().strip()
+					if response == "y":
+						output_dir = os.sep.join(parts[:-4])
+						if not output_dir:
+							output_dir = "."
+
+						print(f"Downloading to {output_dir}...")
+						download_nhanes_data(
+							components=[component_candidate],
+							output_dir=output_dir,
+							years=[cycle_candidate],
+						)
+						if not os.path.exists(file_path):
+							raise FileNotFoundError(
+								f"Download completed but file is still missing: {file_path}"
+							)
+					else:
+						raise FileNotFoundError(f"File not found: {file_path}")
+				else:
+					raise FileNotFoundError(f"File not found: {file_path}")
+			else:
+				raise FileNotFoundError(f"File not found: {file_path}")
+		else:
+			raise FileNotFoundError(f"File not found: {file_path}")
+
+	try:
+		df = pd.read_sas(file_path)
+	except Exception as e:
+		raise IOError(f"Failed to read XPT file {file_path}: {e}")
+
+	if use_csv_for_labels or attach_documentation:
+		base_name = os.path.splitext(os.path.basename(file_path))[0]
+		doc_csv_name = f"{base_name}_variables.csv"
+
+		search_dirs = [os.path.dirname(file_path)]
+		if local_doc_csv_dir:
+			search_dirs.insert(0, local_doc_csv_dir)
+
+		doc_csv_path = None
+		for d in search_dirs:
+			candidate = os.path.join(d, doc_csv_name)
+			if os.path.exists(candidate):
+				doc_csv_path = candidate
+				break
+
+		if not doc_csv_path and allow_download_prompt:
+			parts = file_path.split(os.sep)
+			if len(parts) >= 4:
+				component_candidate = parts[-2]
+				cycle_candidate = parts[-4]
+				if re.match(r"\d{4}-\d{4}", cycle_candidate):
+					print(
+						"Documentation CSV not found. "
+						f"Try downloading docs for Component='{component_candidate}', Cycle='{cycle_candidate}'?"
+					)
+					response = input("Download docs now? [y/N] ").lower().strip()
+					if response == "y":
+						output_dir = os.sep.join(parts[:-4])
+						if not output_dir:
+							output_dir = "."
+						try:
+							download_nhanes_data(
+								components=[component_candidate],
+								output_dir=output_dir,
+								years=[cycle_candidate],
+								with_docs=True,
+							)
+						except Exception as e:
+							logger.warning("Failed to download docs for %s: %s", file_path, e)
+
+						for d in search_dirs:
+							candidate = os.path.join(d, doc_csv_name)
+							if os.path.exists(candidate):
+								doc_csv_path = candidate
+								break
+
+		if not doc_csv_path:
+			logger.warning(
+				"Variables CSV not found for %s. Returning DataFrame with original column names.",
+				base_name,
+			)
+			if attach_documentation:
+				df.attrs["documentation"] = {}
+				df.attrs["documentation_df"] = pd.DataFrame(
+					columns=["variable_name", "label", "description", "value_meanings", "link"]
+				)
+				if add_documentation_column:
+					df["documentation"] = [df.attrs["documentation"] for _ in range(len(df))]
+			return df
+		try:
+			doc_df = pd.read_csv(doc_csv_path)
+		except Exception as e:
+			logger.warning(
+				"Failed to read variables CSV for documentation %s: %s",
+				doc_csv_path,
+				e,
+			)
+			if attach_documentation:
+				df.attrs["documentation"] = {}
+				df.attrs["documentation_df"] = pd.DataFrame(
+					columns=["variable_name", "label", "description", "value_meanings", "link"]
+				)
+				if add_documentation_column:
+					df["documentation"] = [df.attrs["documentation"] for _ in range(len(df))]
+			return df
+
+		if "variable_name" not in doc_df.columns or "label" not in doc_df.columns:
+			logger.warning(
+				"Variables CSV for documentation %s missing 'variable_name' or 'label' columns.",
+				doc_csv_path,
+			)
+			if attach_documentation:
+				df.attrs["documentation"] = {}
+				df.attrs["documentation_df"] = pd.DataFrame(
+					columns=["variable_name", "label", "description", "value_meanings", "link"]
+				)
+				if add_documentation_column:
+					df["documentation"] = [df.attrs["documentation"] for _ in range(len(df))]
+			return df
+
+		label_map = doc_df.dropna(subset=["variable_name", "label"]).set_index("variable_name")[
+			"label"
+		].to_dict()
+
+		doc_records = {}
+		for _, row in doc_df.iterrows():
+			var = str(row.get("variable_name", "") or "").strip()
+			if not var:
+				continue
+
+			raw_value_meanings = row.get("value_meanings", "")
+			value_meanings: Any = None
+			if isinstance(raw_value_meanings, str) and raw_value_meanings.strip():
+				try:
+					value_meanings = json.loads(raw_value_meanings)
+				except Exception:
+					value_meanings = raw_value_meanings.strip()
+
+			doc_records[var] = {
+				"variable_name": var,
+				"label": None if pd.isna(row.get("label")) else str(row.get("label")),
+				"description": None if pd.isna(row.get("description")) else str(row.get("description")),
+				"value_meanings": value_meanings,
+				"link": None if pd.isna(row.get("link")) else str(row.get("link")),
+			}
+
+		rename_map = {}
+		used_names = set()
+
+		def get_unique_name(base, existing):
+			if base not in existing:
+				return base
+			i = 2
+			while f"{base}_{i}" in existing:
+				i += 1
+			return f"{base}_{i}"
+
+		for col in df.columns:
+			original_col = col
+			if original_col in label_map and str(label_map[original_col]).strip():
+				new_name = snake_case(str(label_map[original_col]))
+			else:
+				new_name = snake_case(original_col)
+
+			final_name = get_unique_name(new_name, used_names)
+			used_names.add(final_name)
+			rename_map[original_col] = final_name
+
+		if use_csv_for_labels:
+			df = df.rename(columns=rename_map)
+
+		if attach_documentation:
+			documentation: Dict[str, Dict[str, Any]] = {}
+			for original_col in rename_map:
+				final_col = rename_map[original_col] if use_csv_for_labels else original_col
+				rec = doc_records.get(original_col, {})
+				documentation[final_col] = {
+					"variable_name": original_col,
+					"label": rec.get("label"),
+					"description": rec.get("description"),
+					"value_meanings": rec.get("value_meanings"),
+					"link": rec.get("link"),
+				}
+
+			df.attrs["documentation"] = documentation
+			df.attrs["documentation_df"] = pd.DataFrame(list(documentation.values()))
+
+			if add_documentation_column:
+				df["documentation"] = [documentation for _ in range(len(df))]
+	return df
+
+
+def get_column_doc(df: pd.DataFrame, column_name: str) -> Optional[Dict[str, Any]]:
+	"""Return documentation metadata for a column from df.attrs['documentation']."""
+	docs = df.attrs.get("documentation")
+	if isinstance(docs, dict):
+		entry = docs.get(column_name)
+		if isinstance(entry, dict):
+			return entry
+
+	# Fallback to variable_name lookup in case docs are keyed differently.
+	if isinstance(docs, dict):
+		for _, entry in docs.items():
+			if isinstance(entry, dict) and entry.get("variable_name") == column_name:
+				return entry
+
+	return None
+
+
+def search_nhanes_local_data(
+	search_dir: str,
+	*,
+	recursive: bool = True,
+	include_extensions: Optional[List[str]] = None,
+	only_nhanes_like: bool = True,
+	print_results: bool = True,
+) -> pd.DataFrame:
+	"""Search a local directory for NHANES-relevant files and return an inventory DataFrame."""
+	root = Path(search_dir).expanduser().resolve()
+	if not root.exists() or not root.is_dir():
+		raise NotADirectoryError(f"search_dir is not a valid directory: {search_dir}")
+
+	exts = include_extensions or ["xpt", "csv"]
+	exts = {("." + e.lower().lstrip(".")) for e in exts}
+
+	cycle_re = re.compile(r"^\d{4}-\d{4}$")
+	year_re = re.compile(r"^\d{4}$")
+
+	def infer_parts(p: Path) -> Dict[str, Optional[str]]:
+		parts = list(p.parts)
+		cycle = None
+		year = None
+		component = None
+
+		cycle_idx = None
+		for i, name in enumerate(parts):
+			if cycle_re.match(name):
+				cycle = name
+				cycle_idx = i
+				break
+
+		if cycle_idx is not None:
+			if cycle_idx + 1 < len(parts) and year_re.match(parts[cycle_idx + 1]):
+				year = parts[cycle_idx + 1]
+				if cycle_idx + 2 < len(parts):
+					component = parts[cycle_idx + 2]
+			else:
+				if cycle_idx + 1 < len(parts):
+					component = parts[cycle_idx + 1]
+
+		return {"cycle": cycle, "year": year, "component": component}
+
+	def classify(p: Path) -> str:
+		name = p.name.lower()
+		if p.suffix.lower() == ".xpt":
+			return "data-xpt"
+		if name.endswith("_variables.csv"):
+			return "doc-variables-csv"
+		if p.suffix.lower() == ".csv":
+			return "csv"
+		return "other"
+
+	def nhanes_like_filter(p: Path) -> bool:
+		if p.suffix.lower() == ".xpt" or p.name.lower().endswith("_variables.csv"):
+			return True
+
+		if p.suffix.lower() != ".csv":
+			return False
+
+		meta = infer_parts(p)
+		if meta.get("cycle"):
+			return True
+
+		stem = p.stem
+		if stem.endswith("_variables"):
+			xpt_candidate = p.with_name(stem.replace("_variables", "") + ".xpt")
+			if xpt_candidate.exists():
+				return True
+
+		return False
+
+	it = root.rglob("*") if recursive else root.glob("*")
+
+	rows: List[Dict[str, Any]] = []
+	for p in it:
+		if not p.is_file():
+			continue
+		if p.suffix.lower() not in exts:
+			continue
+		if only_nhanes_like and not nhanes_like_filter(p):
+			continue
+
+		meta = infer_parts(p)
+		kind = classify(p)
+
+		try:
+			st = p.stat()
+			size_kb = st.st_size / 1024
+			mtime = datetime.fromtimestamp(st.st_mtime)
+		except OSError:
+			size_kb = None
+			mtime = None
+
+		rows.append(
+			{
+				"type": kind,
+				"cycle": meta.get("cycle"),
+				"year": meta.get("year"),
+				"component": meta.get("component"),
+				"file": p.name,
+				"stem": p.stem,
+				"ext": p.suffix.lower().lstrip("."),
+				"size_kb": None if size_kb is None else round(size_kb, 1),
+				"modified": None if mtime is None else mtime.strftime("%Y-%m-%d %H:%M"),
+				"path": str(p),
+			}
+		)
+
+	df = pd.DataFrame(rows)
+	if df.empty:
+		if print_results:
+			print(f"No NHANES-like files found in: {root}")
+		return df
+
+	sort_cols = ["cycle", "component", "year", "type", "file"]
+	sort_cols = [c for c in sort_cols if c in df.columns]
+	df = df.sort_values(sort_cols, na_position="last").reset_index(drop=True)
+
+	if print_results:
+		_print_nhanes_search_results(df, root=str(root))
+
+	return df
+
+
+def _print_nhanes_search_results(df: pd.DataFrame, root: str) -> None:
+	"""Pretty-printer using rich when available, otherwise pandas text output."""
+	title = f"NHANES local files found in: {root}  (n={len(df)})"
+
+	try:
+		from rich.console import Console
+		from rich.table import Table
+		from rich.text import Text
+
+		console = Console()
+
+		table = Table(title=title, show_lines=False, header_style="bold")
+		table.add_column("Type", style="cyan", no_wrap=True)
+		table.add_column("Cycle", style="magenta", no_wrap=True)
+		table.add_column("Year", style="magenta", no_wrap=True)
+		table.add_column("Component", style="green")
+		table.add_column("File", style="white")
+		table.add_column("Size KB", justify="right")
+		table.add_column("Modified", style="dim", no_wrap=True)
+		table.add_column("Path", style="dim")
+
+		for _, r in df.iterrows():
+			table.add_row(
+				str(r.get("type") or ""),
+				str(r.get("cycle") or ""),
+				str(r.get("year") or ""),
+				str(r.get("component") or ""),
+				str(r.get("file") or ""),
+				"" if pd.isna(r.get("size_kb")) else str(r.get("size_kb")),
+				str(r.get("modified") or ""),
+				str(r.get("path") or ""),
+			)
+
+		counts = df["type"].value_counts(dropna=False).to_dict()
+		summary = ", ".join([f"{k}={v}" for k, v in counts.items()])
+		console.print(table)
+		console.print(Text(f"Summary: {summary}", style="bold"))
+
+		return
+
+	except Exception:
+		pass
+
+	# TODO: log instead of print
+	print(title)
+	show_cols = ["type", "cycle", "year", "component", "file", "size_kb", "modified", "path"]
+	show_cols = [c for c in show_cols if c in df.columns]
+	df_out = df[show_cols].copy()
+
+	with pd.option_context("display.max_rows", 200, "display.max_colwidth", 120, "display.width", 200):
+		print(df_out.to_string(index=False))
+
+	counts = df["type"].value_counts(dropna=False).to_dict()
+	summary = ", ".join([f"{k}={v}" for k, v in counts.items()])
+	print(f"Summary: {summary}")

healthdata-0.1.0/src/healthdata/nhanes/settings.py

@@ -0,0 +1,10 @@
+DEFAULT_BASE_SEARCH_URL = "https://wwwn.cdc.gov/nchs/nhanes/search/datapage.aspx"
+DEFAULT_BASE_DOMAIN = "https://wwwn.cdc.gov"
+
+DEFAULT_COMPONENTS = [
+    "Demographics",
+    "Dietary",
+    "Examination",
+    "Laboratory",
+    "Questionnaire",
+]

healthdata-0.1.0/src/healthdata.egg-info/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: healthdata
+Version: 0.1.0
+Summary: Download and read NHANES health survey data.
+Author-email: Ismail Elouafiq <contact@ismail.bio>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/nidhog/healthdata
+Project-URL: Repository, https://github.com/nidhog/healthdata
+Project-URL: Documentation, https://github.com/nidhog/healthdata/tree/main/docs/nhanes
+Project-URL: Issues, https://github.com/nidhog/healthdata/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: tqdm
+Dynamic: license-file
+
+# Health Data Downloader Package
+The Health Data Downloader is a Python package that makes it easy to download and read real world health data.
+
+Current data sources supported:
+* **NHANES** National Health and Nutrition Examination Survey (CDC)
+## The NHANES Submodule
+If you want to download health data from NHANES this module can help you not only download the data but also add documentation from the official NHANES website, and read this data into a pandas dataframe.
+
+[NHANES](https://www.cdc.gov/nchs/nhanes/about/) is the National Health and Nutrition Examination Survey which collects data about the health of adults and children in the United States.
+
+_Note: This is not an official repository from the NHANES project nor the CDC_
+
+This project provides workflows that enable you to:
+- discover and download datasets from NHANES by component (meaning what type of data like Laboratory data, examination etc.) and by cycle (which years)
+- read the XPT files into a pandas DataFrames
+- extract NHANES variable documentation into CSV files and rename variable codes into human-readable names (such as `heart_rate` instead of `LF321`)
+- search and summarize local files if they are already downloaded
+
+## Quick start
+In thie quick start we will:
+1. Install required packages.
+2. Run a download.
+3. Read and inspect data.
+
+```bash
+pip install healthdata
+```
+You can choose which `components` you want to download (such as Demographics, Laboratory etc.) and which years (can be either "2017-2020" or "2016"). This will also download documentation about these fields from the NHANES website to add them when the data is read.
+If the data is already downloaded in the output directory the module will log this and skip the download.
+```python
+from healthdata.nhanes.downloader import download_nhanes_data
+
+download_nhanes_data(
+    components=["Demographics", "Examination"],
+    output_dir="nhanes_data",
+    years=["2017-2018"],
+)
+```
+
+You can read the data you already downloaded (you will be prompted to download it if it doesn't exist)
+```python
+from healthdata.nhanes.reader import read_nhanes_data, search_nhanes_local_data
+
+df = read_nhanes_data("nhanes_data/2017-2018/2017/Demographics/DEMO_J.XPT")
+print(df.head())
+
+inventory = search_nhanes_local_data(search_dir="nhanes_data")
+print(inventory.head())
+```
+That's all folks! Now you will see that your dataframe already includes documentation.
+## Current caveats
+- HTML codebook parsing depends on current NHANES page text patterns. CDC page structure changes may require parser updates.
+- Network errors and partial downloads are not retried automatically.
+
+## Upcoming Additions
+Planned additions include NCBI GEO and other open health and biomedical datasets. The current release supports NHANES only.
+
+## License
+This project is licensed under the [MIT License](LICENSE). Commercial use is permitted under its terms. Downloaded datasets may be subject to separate terms set by their respective providers.
+
+For paid support or consulting, contact the author at contact@ismail.bio.
+
+## Author
+[Ismail Elouafiq](https://ismail.bio)
+
+Contact: contact@ismail.bio

healthdata-0.1.0/src/healthdata.egg-info/SOURCES.txt

@@ -0,0 +1,17 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+src/healthdata/__init__.py
+src/healthdata.egg-info/PKG-INFO
+src/healthdata.egg-info/SOURCES.txt
+src/healthdata.egg-info/dependency_links.txt
+src/healthdata.egg-info/requires.txt
+src/healthdata.egg-info/top_level.txt
+src/healthdata/nhanes/__init__.py
+src/healthdata/nhanes/downloader.py
+src/healthdata/nhanes/reader.py
+src/healthdata/nhanes/settings.py
+tests/test_downloader.py
+tests/test_public_api.py
+tests/test_reader.py

healthdata-0.1.0/src/healthdata.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

healthdata-0.1.0/src/healthdata.egg-info/requires.txt

@@ -0,0 +1,4 @@
+pandas
+requests
+beautifulsoup4
+tqdm

healthdata-0.1.0/src/healthdata.egg-info/top_level.txt

@@ -0,0 +1 @@
+healthdata

healthdata-0.1.0/tests/test_downloader.py

@@ -0,0 +1,72 @@
+from healthdata.nhanes.downloader import (
+    extract_date_from_url,
+    get_file_links_for_component,
+    parse_nhanes_doc_variables,
+)
+
+
+class _DummyResponse:
+    def __init__(self, text):
+        self.text = text
+
+    def raise_for_status(self):
+        return None
+
+
+def test_extract_date_from_url_cycle_and_year_and_fallback():
+    # We expect the function to extract the cycle or year based on the URL similar to below
+    assert extract_date_from_url("https://example.org/2017-2018/doc.htm") == "2017-2018"
+    assert extract_date_from_url("https://example.org/data/2019/file.xpt") == "2019"
+    assert extract_date_from_url("https://example.org/no-date") == "nodate"
+
+
+def test_parse_nhanes_doc_variables_extracts_basic_fields():
+    # this is currently what we expect to find in the HTML
+    html = """
+    <html><body>
+    Variable Name: SEQN
+    SAS Label: Respondent sequence number
+    English Text: Unique participant identifier
+    Target: Both males and females
+    </body></html>
+    """
+
+    rows = parse_nhanes_doc_variables(html, "https://example.org/doc.htm")
+
+    assert len(rows) == 1
+    assert rows[0]["variable_name"] == "SEQN"
+    assert rows[0]["label"] == "Respondent sequence number"
+    assert rows[0]["description"] == "Unique participant identifier"
+    assert "value_meanings" in rows[0]
+    assert rows[0]["link"] == "https://example.org/doc.htm"
+
+
+def test_get_file_links_for_component_filters_years(monkeypatch):
+    # On NHANES website, the files are linked in this format:
+    expected_html = """
+    <table>
+      <tr>
+        <td>2017-2018</td>
+        <td><a href="/nchs/nhanes/2017-2018/P_DEMO.htm">P_DEMO</a></td>
+        <td><a href="/nchs/nhanes/2017-2018/P_DEMO.xpt">P_DEMO</a></td>
+      </tr>
+      <tr>
+        <td>2015-2016</td>
+        <td><a href="/nchs/nhanes/2015-2016/OLD_DEMO.htm">OLD_DEMO</a></td>
+        <td><a href="/nchs/nhanes/2015-2016/OLD_DEMO.xpt">OLD_DEMO</a></td>
+      </tr>
+    </table>
+    """
+
+    def _fake_get(url, params=None):
+        assert params == {"Component": "Demographics"}
+        return _DummyResponse(expected_html)
+
+    monkeypatch.setattr("healthdata.nhanes.downloader.requests.get", _fake_get)
+
+    links = get_file_links_for_component("Demographics", years=["2017-2018"])
+
+    assert len(links) == 1
+    assert links[0]["years"] == "2017-2018"
+    assert links[0]["doc_url"].endswith("/P_DEMO.htm")
+    assert links[0]["xpt_url"].endswith("/P_DEMO.xpt")

healthdata-0.1.0/tests/test_public_api.py

@@ -0,0 +1,10 @@
+import healthdata.nhanes as nhanes
+
+
+def test_nhanes_public_api_exports_main_functions():
+    assert callable(nhanes.download_nhanes_data)
+    assert callable(nhanes.download_and_extract_docs)
+    assert callable(nhanes.get_file_links_for_component)
+    assert callable(nhanes.read_nhanes_data)
+    assert callable(nhanes.get_column_doc)
+    assert callable(nhanes.search_nhanes_local_data)

healthdata-0.1.0/tests/test_reader.py

@@ -0,0 +1,87 @@
+from pathlib import Path
+
+import pandas as pd
+
+from healthdata.nhanes.reader import get_column_doc, read_nhanes_data, search_nhanes_local_data, snake_case
+
+
+def test_snake_case_handles_spacing_and_symbols():
+    assert snake_case(". Heart Rate (BPM)") == "heart_rate_bpm"
+    assert snake_case("___") == "unnamed"
+
+
+def test_read_nhanes_data_renames_columns_from_variables_csv(tmp_path, monkeypatch):
+    xpt_path = tmp_path / "P_DEMO.xpt"
+    xpt_path.write_bytes(b"placeholder")
+    # we create a csv file similar to what we have
+    csv_path = tmp_path / "P_DEMO_variables.csv"
+    csv_path.write_text(
+        "variable_name,label\n"
+        "SEQN,Participant ID\n"
+        "RIDAGEYR,Age\n"
+        "DUPCOL,Age\n",
+        encoding="utf-8",
+    )
+
+    source_df = pd.DataFrame({"SEQN": [1], "RIDAGEYR": [30], "DUPCOL": [31]})
+    monkeypatch.setattr("healthdata.nhanes.reader.pd.read_sas", lambda _: source_df)
+
+    result = read_nhanes_data(str(xpt_path), use_csv_for_labels=True)
+
+    assert list(result.columns) == ["participant_id", "age", "age_2"]
+    assert "documentation" in result.attrs
+    assert result.attrs["documentation"]["participant_id"]["variable_name"] == "SEQN"
+
+
+def test_read_nhanes_data_attaches_documentation_without_renaming(tmp_path, monkeypatch):
+    xpt_path = tmp_path / "P_DEMO.xpt"
+    xpt_path.write_bytes(b"placeholder")
+    csv_path = tmp_path / "P_DEMO_variables.csv"
+    csv_path.write_text(
+        "variable_name,label,description,value_meanings,link\n"
+        "SEQN,Participant ID,Unique id,\"[{\"\"code\"\": \"\"1\"\", \"\"meaning\"\": \"\"Yes\"\"}]\",https://example.org/doc.htm\n",
+        encoding="utf-8",
+    )
+
+    source_df = pd.DataFrame({"SEQN": [1]})
+    monkeypatch.setattr("healthdata.nhanes.reader.pd.read_sas", lambda _: source_df)
+
+    result = read_nhanes_data(
+        str(xpt_path),
+        use_csv_for_labels=False,
+        attach_documentation=True,
+    )
+
+    assert list(result.columns) == ["SEQN"]
+    assert "documentation" in result.attrs
+    assert result.attrs["documentation"]["SEQN"]["description"] == "Unique id"
+    assert result.attrs["documentation"]["SEQN"]["value_meanings"] == [{"code": "1", "meaning": "Yes"}]
+    assert get_column_doc(result, "SEQN")["label"] == "Participant ID"
+    assert get_column_doc(result, "MISSING") is None
+
+
+def test_search_nhanes_local_data_returns_expected_inventory(tmp_path):
+    base = tmp_path / "2017-2020" / "2017" / "Demographics"
+    base.mkdir(parents=True)
+
+    (base / "P_DEMO.xpt").write_bytes(b"xpt")
+    (base / "P_DEMO_variables.csv").write_text("variable_name,label\nSEQN,Participant ID\n", encoding="utf-8")
+    (tmp_path / "random.csv").write_text("a,b\n1,2\n", encoding="utf-8")
+
+    result = search_nhanes_local_data(str(tmp_path), print_results=False)
+
+    assert len(result) == 2
+    assert set(result["type"]) == {"data-xpt", "doc-variables-csv"}
+    assert set(result["cycle"]) == {"2017-2020"}
+    assert set(result["component"]) == {"Demographics"}
+
+
+def test_search_nhanes_local_data_invalid_dir_raises(tmp_path):
+    missing = tmp_path / "does-not-exist"
+
+    try:
+        search_nhanes_local_data(str(missing), print_results=False)
+    except NotADirectoryError as exc:
+        assert "not a valid directory" in str(exc)
+    else:
+        raise AssertionError("Expected NotADirectoryError")