hccinfhir 0.2.0__py3-none-any.whl → 0.2.2__py3-none-any.whl

{hccinfhir-0.2.0.dist-info → hccinfhir-0.2.2.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: hccinfhir
-Version: 0.2.0
+Version: 0.2.2
 Summary: HCC Algorithm for FHIR Resources
 Project-URL: Homepage, https://github.com/mimilabs/hccinfhir
 Project-URL: Issues, https://github.com/mimilabs/hccinfhir/issues
@@ -10,6 +10,7 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Requires-Python: >=3.8
 Requires-Dist: pydantic>=2.10.3
+Requires-Dist: typing-extensions>=4.6.0
 Description-Content-Type: text/markdown
 
 # HCCInFHIR
@@ -62,6 +63,7 @@ print(f"HCCs: {result.hcc_list}")
   - [Demographic Prefix Override](#demographic-prefix-override)
   - [Custom File Path Resolution](#custom-file-path-resolution)
   - [Batch Processing](#batch-processing)
+  - [Large-Scale Processing with Databricks](#large-scale-processing-with-databricks)
   - [Converting to Dictionaries](#converting-to-dictionaries)
 - [Sample Data](#sample-data)
 - [Testing](#testing)
@@ -78,7 +80,7 @@ print(f"HCCs: {result.hcc_list}")
 - **Custom Data Files**: Full support for custom coefficients, mappings, and hierarchies
 - **Flexible File Resolution**: Absolute paths, relative paths, or bundled data files
 - **Type-Safe**: Built on Pydantic with full type hints
-- **Well-Tested**: 155 comprehensive tests covering all features
+- **Well-Tested**: 181 comprehensive tests covering all features
 
 ## 📊 Data Sources & Use Cases
 
@@ -834,6 +836,146 @@ with open("risk_scores.json", "w") as f:
     json.dump(results, f, indent=2)
 ```
 
+### Large-Scale Processing with Databricks
+
+For processing millions of beneficiaries, use PySpark's `pandas_udf` for distributed computation. The hccinfhir logic is well-suited for batch operations with clear, simple transformations.
+
+**Performance Benchmark**:
+
+![Databricks Performance Chart](hccinfhir_pandas_udf_performance_chart.png)
+
+*Tested with ACO data on Databricks Runtime 17.3 LTS, Worker: i3.4xlarge (122GB, 16 cores)*
+
+The chart shows execution time varies based on condition complexity - members with more diagnoses require additional internal processing loops. While the relationship isn't perfectly linear, **1 million members can be processed in under 2 minutes** with this configuration.
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql.types import StructType, StructField, FloatType, ArrayType, StringType
+from pyspark.sql import functions as F
+from pyspark.sql.functions import pandas_udf
+import pandas as pd
+
+from hccinfhir import HCCInFHIR, Demographics
+
+# Define the return schema
+hcc_schema = StructType([
+    StructField("risk_score", FloatType(), True),
+    StructField("risk_score_demographics", FloatType(), True),
+    StructField("risk_score_chronic_only", FloatType(), True),
+    StructField("risk_score_hcc", FloatType(), True),
+    StructField("hcc_list", ArrayType(StringType()), True)
+])
+
+# Initialize processor (will be serialized to each executor)
+hcc_processor = HCCInFHIR(model_name="CMS-HCC Model V28")
+
+# Create the pandas UDF
+@pandas_udf(hcc_schema)
+def calculate_hcc(
+    age_series: pd.Series,
+    sex_series: pd.Series,
+    diagnosis_series: pd.Series
+) -> pd.DataFrame:
+    results = []
+
+    for age, sex, diagnosis_codes in zip(age_series, sex_series, diagnosis_series):
+        try:
+            demographics = Demographics(age=int(age), sex=sex)
+
+            # diagnosis_codes can be passed directly - accepts any iterable including numpy arrays
+            result = hcc_processor.calculate_from_diagnosis(diagnosis_codes, demographics)
+
+            results.append({
+                'risk_score': float(result.risk_score),
+                'risk_score_demographics': float(result.risk_score_demographics),
+                'risk_score_chronic_only': float(result.risk_score_chronic_only),
+                'risk_score_hcc': float(result.risk_score_hcc),
+                'hcc_list': result.hcc_list
+            })
+        except Exception as e:
+            # Log error and return nulls for failed rows
+            print(f"ERROR processing row: {e}")
+            results.append({
+                'risk_score': None,
+                'risk_score_demographics': None,
+                'risk_score_chronic_only': None,
+                'risk_score_hcc': None,
+                'hcc_list': None
+            })
+
+    return pd.DataFrame(results)
+
+# Apply the UDF to your DataFrame
+# Assumes df has columns: age, patient_gender, diagnosis_codes (array of strings)
+df = df.withColumn(
+    "hcc_results",
+    calculate_hcc(
+        F.col("age"),
+        F.col("patient_gender"),
+        F.col("diagnosis_codes")
+    )
+)
+
+# Expand the struct into separate columns
+df = df.select(
+    "*",
+    F.col("hcc_results.risk_score").alias("risk_score"),
+    F.col("hcc_results.risk_score_demographics").alias("risk_score_demographics"),
+    F.col("hcc_results.risk_score_chronic_only").alias("risk_score_chronic_only"),
+    F.col("hcc_results.risk_score_hcc").alias("risk_score_hcc"),
+    F.col("hcc_results.hcc_list").alias("hcc_list")
+).drop("hcc_results")
+```
+
+**Performance Tips**:
+- **Repartition** your DataFrame before applying the UDF to balance workload across executors
+- **Cache** the processor initialization by defining it at module level
+- **Batch size**: pandas_udf processes data in batches; Spark handles optimal batch sizing automatically
+- **Install hccinfhir** on all cluster nodes: `%pip install hccinfhir` in a notebook cell or add to cluster init script
+
+**Extended Schema with Demographics**:
+
+```python
+# Include additional demographic parameters
+@pandas_udf(hcc_schema)
+def calculate_hcc_full(
+    age_series: pd.Series,
+    sex_series: pd.Series,
+    dual_status_series: pd.Series,
+    diagnosis_series: pd.Series
+) -> pd.DataFrame:
+    results = []
+
+    for age, sex, dual_status, diagnosis_codes in zip(
+        age_series, sex_series, dual_status_series, diagnosis_series
+    ):
+        try:
+            demographics = Demographics(
+                age=int(age),
+                sex=sex,
+                dual_elgbl_cd=dual_status if dual_status else "00"
+            )
+            result = hcc_processor.calculate_from_diagnosis(diagnosis_codes, demographics)
+
+            results.append({
+                'risk_score': float(result.risk_score),
+                'risk_score_demographics': float(result.risk_score_demographics),
+                'risk_score_chronic_only': float(result.risk_score_chronic_only),
+                'risk_score_hcc': float(result.risk_score_hcc),
+                'hcc_list': result.hcc_list
+            })
+        except Exception as e:
+            results.append({
+                'risk_score': None,
+                'risk_score_demographics': None,
+                'risk_score_chronic_only': None,
+                'risk_score_hcc': None,
+                'hcc_list': None
+            })
+
+    return pd.DataFrame(results)
+```
+
 ### Converting to Dictionaries
 
 All Pydantic models support dictionary conversion for JSON serialization, database storage, or legacy code:
@@ -914,7 +1056,7 @@ hatch shell
 # Install in development mode
 pip install -e .
 
-# Run all tests (155 tests)
+# Run all tests (181 tests)
 pytest tests/
 
 # Run specific test file

{hccinfhir-0.2.0.dist-info → hccinfhir-0.2.2.dist-info}/RECORD

@@ -1,20 +1,21 @@
-hccinfhir/__init__.py,sha256=CKpYTUSzZdP3s1eB74w5JTe9OS3MtcvuUkv6ymgSyic,1085
-hccinfhir/datamodels.py,sha256=NULDYb57R61v4EklOI_AAIuC1-OkLFH1InbAad48dZM,10601
-hccinfhir/defaults.py,sha256=tMNym0R6Nr6ibKTqOu6N1vLcdekL0ZmHyDNIOCOMsP4,1292
+hccinfhir/__init__.py,sha256=3aFYtjTklZJg3wIlnMJNgfDBaDCfKXVlYsacdsZ9L4I,1113
+hccinfhir/constants.py,sha256=C4Vyjtzgyd4Jm2I2X6cTYQZLe-jAMC8boUcy-7OXQDQ,8473
+hccinfhir/datamodels.py,sha256=X9ZHPCfxI12o88AFmeomzSxh5fA6tsDOAmTlXbdncJ8,11471
+hccinfhir/defaults.py,sha256=aKdXPhf9bYUzpGvXM1GIXZaKxqkKInt3v9meLB9fWog,1394
 hccinfhir/extractor.py,sha256=xL9c2VT-e2I7_c8N8j4Og42UEgVuCzyn9WFp3ntM5Ro,1822
-hccinfhir/extractor_834.py,sha256=vODcD53iU5ZwQsSbBE8Gix9D-0fz-EhwkmjqRO6LML8,19541
-hccinfhir/extractor_837.py,sha256=D60gUFtMk2S0NrJ0iq3ENo35yIwBmBQvF5TurJgRIa8,15327
+hccinfhir/extractor_834.py,sha256=dIqovUOWm_7k_c6sUqTIzQua_kTQ8dLGy3-4-LECW3Y,18855
+hccinfhir/extractor_837.py,sha256=fGsvBTWIj9dsHLGGR67AdlYDSsFi5qnSVlTgwkL1f-E,15334
 hccinfhir/extractor_fhir.py,sha256=wUN3vTm1oTZ-KvfcDebnpQMxAC-7YlRKv12Wrv3p85A,8490
 hccinfhir/filter.py,sha256=j_yD2g6RBXVUV9trKkWzsQ35x3fRvfKUPvEXKUefI64,2007
-hccinfhir/hccinfhir.py,sha256=rCnExvxZGKi1vLD4cHQ0nzPAGV6e-8C15MtJ2p7zAAk,11160
-hccinfhir/model_calculate.py,sha256=KSeZjKYBCfBYYIWOIckDg941OC8050MX2F7BZ2l3V8g,7663
-hccinfhir/model_coefficients.py,sha256=--8Gh5gYJez1v0cBA1ZqDn0QI6On-ia1-wPr-mqbRFs,4680
-hccinfhir/model_demographics.py,sha256=CR4WC8XVq-CI1nYJoVFc5-KXTw-pKoVlHkHqfnXlnj0,9121
+hccinfhir/hccinfhir.py,sha256=NydnH3WBvuyskn76hY70LpUS6XuIEoax_kip1mgfpHw,11225
+hccinfhir/model_calculate.py,sha256=_TUWNVUsBym0pre3wltXvRuipQaONQ0QBfWPFNAeDsQ,8347
+hccinfhir/model_coefficients.py,sha256=5n3QzHX6FJ3MlO0cV9NS7Bqt-lxzVvT_M3zFaWq6Gng,4685
+hccinfhir/model_demographics.py,sha256=nImKtJCq1HkR9w2GU8aikybJFgow71CPufBRV8Jn7fM,8932
 hccinfhir/model_dx_to_cc.py,sha256=Yjc6xKI-jMXsbOzS_chc4NI15Bwagb7BwZZ8cKQaTbk,1540
 hccinfhir/model_hierarchies.py,sha256=cboUnSHZZfOxA8QZKV4QIE-32duElssML32OqYT-65g,1542
-hccinfhir/model_interactions.py,sha256=xdsTuc3ii8U_MaPpYv0SnR4eR_w72eHKUJn1mwmZHm4,21379
+hccinfhir/model_interactions.py,sha256=g6jK27Xu8RQUHS3lk4sk2v6w6wqd52mdbGn0BsnR7Pk,21394
 hccinfhir/samples.py,sha256=2VSWS81cv9EnaHqK7sd6CjwG6FUI9E--5wHgD000REI,9952
-hccinfhir/utils.py,sha256=9ki4o1wXyAYYr8BR9Skkz0PKL_1H_HYNV4LalEsASE0,8260
+hccinfhir/utils.py,sha256=hQgHjuOcEQcnxemTZwqFBHWvLC5-C1Gup9cDXEYlZjE,10770
 hccinfhir/data/__init__.py,sha256=SGiSkpGrnxbvtEFMMlk82NFHOE50hFXcgKwKUSuVZUg,45
 hccinfhir/data/hcc_is_chronic.csv,sha256=Bwd-RND6SdEsKP-assoBaXnjUJAuDXhSkwWlymux72Y,19701
 hccinfhir/data/hcc_is_chronic_without_esrd_model.csv,sha256=eVVI4_8mQNkiBiNO3kattfT_zfcV18XgmiltdzZEXSo,17720
@@ -28,6 +29,7 @@ hccinfhir/data/ra_eligible_cpt_hcpcs_2025.csv,sha256=-tMvv2su5tsSbGUh6fZZCMUEkXI
 hccinfhir/data/ra_eligible_cpt_hcpcs_2026.csv,sha256=EYGN7k_rgCpJe59lL_yNInUcCkdETDWGSFTXII3LZ0Y,40497
 hccinfhir/data/ra_hierarchies_2025.csv,sha256=HQSPNloe6mvvwMgv8ZwYAfWKkT2b2eUvm4JQy6S_mVQ,13045
 hccinfhir/data/ra_hierarchies_2026.csv,sha256=A6ZQZb0rpRWrySBB_KA5S4PGtMxWuzB2guU3aBE09v0,19596
+hccinfhir/data/ra_labels_2026.csv,sha256=YstfP7s-3ZwjP4I_GYPPj3_yn-PQK3Q0Q_MVYZhsfjY,50248
 hccinfhir/sample_files/__init__.py,sha256=SGiSkpGrnxbvtEFMMlk82NFHOE50hFXcgKwKUSuVZUg,45
 hccinfhir/sample_files/sample_834_01.txt,sha256=J2HMXfY6fAFpV36rvLQ3QymRRS2TPqf3TQY6CNS7TrE,1627
 hccinfhir/sample_files/sample_837_0.txt,sha256=eggrD259uHa05z2dfxWBpUDseSDp_AQcLyN_adpHyTw,5295
@@ -47,7 +49,7 @@ hccinfhir/sample_files/sample_eob_1.json,sha256=_NGSVR2ysFpx-DcTvyga6dFCzhQ8Vi9f
 hccinfhir/sample_files/sample_eob_2.json,sha256=FcnJcx0ApOczxjJ_uxVLzCep9THfNf4xs9Yf7hxk8e4,1769
 hccinfhir/sample_files/sample_eob_200.ndjson,sha256=CxpjeQ1DCMUzZILaM68UEhfxO0p45YGhDDoCZeq8PxU,1917986
 hccinfhir/sample_files/sample_eob_3.json,sha256=4BW4wOMBEEU9RDfJR15rBEvk0KNHyuMEh3e055y87Hc,2306
-hccinfhir-0.2.0.dist-info/METADATA,sha256=bRJ_IYdfpYNRnKLYeqyhVlKTob5K69SeCiC7FGzX8vM,31674
-hccinfhir-0.2.0.dist-info/WHEEL,sha256=C2FUgwZgiLbznR-k0b_5k3Ai_1aASOXDss3lzCUsUug,87
-hccinfhir-0.2.0.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
-hccinfhir-0.2.0.dist-info/RECORD,,
+hccinfhir-0.2.2.dist-info/METADATA,sha256=PS0FbHZTjJ4ALMCISUyDf9h2hWCzNZvgThIVI8rdey0,37122
+hccinfhir-0.2.2.dist-info/WHEEL,sha256=C2FUgwZgiLbznR-k0b_5k3Ai_1aASOXDss3lzCUsUug,87
+hccinfhir-0.2.2.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
+hccinfhir-0.2.2.dist-info/RECORD,,