londonaicentre-docsynth 1.1.0__tar.gz

londonaicentre_docsynth-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 London AI Centre
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

londonaicentre_docsynth-1.1.0/PKG-INFO

@@ -0,0 +1,15 @@
+Metadata-Version: 2.4
+Name: londonaicentre-docsynth
+Version: 1.1.0
+Summary: SchemaLlama docsynth
+Author-email: "Dr. Joe Zhang" <jzhang@nhs.net>
+License-Expression: CC-BY-NC-ND-4.0
+Requires-Python: >=3.12
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: google-generativeai>=0.8.5
+Requires-Dist: pydantic-settings>=2.12.0
+Requires-Dist: londonaicentre-mesa-utils>=1.6.0
+Requires-Dist: londonaicentre-mesa-types>=2.3.0
+Requires-Dist: londonaicentre-mesa-runtime>=0.4.0
+Dynamic: license-file

londonaicentre_docsynth-1.1.0/README.md

@@ -0,0 +1,118 @@
+# MESA Docsynth
+
+Configurable pipeline for generating high fidelity synthetic documents that can be turned into training data.
+
+## Installation
+
+```sh
+uv add londonaicentre-docsynth
+```
+
+## Getting started
+
+Using the format specified in the [`config`](src/docsynth/config.py), create `pipeline.yml` to configure the LLM provider (currently Anthropic (AWS Bedrock), Gemini, local or the MESA Runtime), profile sampling mode (random/sequential), prompt configuration, and output directory (see [`pipeline.yml.example`](pipeline.yml.example)).
+
+If using Gemini or Bedrock, configure `.env` with an API key (see [`.env.example`](.env.example)).
+Bedrock API keys can be obtained from your AWS account manager.
+If not using AWS Bedrock, obtain suitable credentials for another provider (e.g. Gemini Developer API).
+If using a local LLM, no/blank credentials will likely be sufficient.
+
+### Anthropic (AWS Bedrock) batch generation
+
+For batch generation, additional credentials are needed:
+
+  1. Obtain access to AWS from your account manager and follow the instructions [here](https://docs.commonfate.io/granted/getting-started) to set up SSO authentication for use of the AWS CLI.
+
+  2. Obtain information on a Bedrock Execution IAM Role with S3 and model access and information on the name of an S3 bucket to upload a batch specification to.
+
+### MESA Runtime batch generation
+
+The MESA Runtime only supports batch generation:
+
+  1. Set up AWS SSO authentication as above, and obtain the name of an S3 bucket the runtime can read from and write to.
+
+  2. Obtain the MESA Runtime API URL and credentials, and add them to `.env` (see [`.env.example`](.env.example)).
+
+Set `split_across_models: true` under `mesa_runtime` in `pipeline.yml` to split prompts across every model the runtime serves, in turn.
+
+## Usage
+
+1. Create a `Generator` object, and call the `generate` method:
+
+- For Anthropic (AWS Bedrock) batch generation:
+
+  ```python
+  Generator().generate_via_batch(<BATCH_BUCKET>, <BEDROCK_EXECUTION_ROLE>)
+  Generator().extract_batch_output(<BATCH_BUCKET>)
+  ```
+
+- For MESA Runtime batch generation:
+
+  ```python
+  Generator().generate_via_batch(<BATCH_BUCKET>)
+  Generator().extract_batch_output(<BATCH_BUCKET>)
+  ```
+
+- To add the documents in `prompts/supplement.yaml` to a batch before publishing it, with [Claude Code](https://code.claude.com) installed and logged in, replace `extract_batch_output` with:
+
+  ```python
+  generator = Generator()
+  generator.save_batch_output(<BATCH_BUCKET>)
+  generator.supplement()
+  generator.publish()
+  ```
+
+- For other LLM providers:
+
+  ```python
+  Generator().generate()
+  ```
+
+2. Generated documents are saved to `./output/{subdirectory}/{doc_id}.json`:
+
+  ```json
+  {
+    "doc_id": "9893532233caff98cd083a116b013c0b",
+    "document_name": "narrative",
+    "document_sourcedb": "DocSynth",
+    "profile": "lung_001",
+    "timestamp": "20251012_143022_477",
+    "prompt": "... complete prompt text ...",
+    "content": "... generated clinical document ..."
+  }
+  ```
+
+Note: when `llm.enabled: false` in `pipeline.yml`, only prompts are saved (no `content` field).
+
+### Flowchart
+
+<p align="center">
+  <img src="_assets/prompt.svg" width="700" alt="Flowchart of the document generation pipeline">
+</p>
+
+### Custom Assets
+
+To add custom assets, the [`DocsynthAssets`](src/docsynth/types/wrapper.py) base class should be extended to wrap assets for a particular domain (e.g. cancer) and passed to the `Generator` class, e.g.:
+
+```python
+Generator(assets=MyDocsynthAssets()).generate()
+```
+
+The base class assumes the presence of:
+
+- Primary profiles, one entry per synthetic case, holding whatever domain-specific fields that case needs (formatted according to [`Profile`](src/docsynth/types/profile.py)):
+  - `assets/<use case>/profiles/*.yml`
+
+- Probabilistic sampling from style and content requirements, with domain-defined sections (formatted according to [`Style`](src/docsynth/types/sampling.py) and [`Content`](src/docsynth/types/sampling.py)):
+  - `assets/<use case>/style.yml`
+  - `assets/<use case>/content.yml`
+
+- Example structures that are hand-crafted based on real clinical document formats:
+  - `assets/<use case>/structure/*.txt`
+
+- Prompt templates:
+  - `assets/<use case>/prompts/`
+
+In addition, concrete implementations should be provided for abstract methods to handle domain-specific asset creation logic.
+
+To scaffold a new use case, run `uvx copier copy src/docsynth/assets/_domain src/docsynth/assets/<use case>`, which also prints which parts of the generated files are editable.

londonaicentre_docsynth-1.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[project]
+name = "londonaicentre-docsynth" 
+description = "SchemaLlama docsynth"
+authors = [
+    { name = "Dr. Joe Zhang", email = "jzhang@nhs.net" }
+]
+version = "1.1.0"
+requires-python = ">=3.12"
+license = "CC-BY-NC-ND-4.0"
+dependencies = [
+    "pyyaml>=6.0.3",
+    "google-generativeai>=0.8.5",
+    "pydantic-settings>=2.12.0",
+    "londonaicentre-mesa-utils>=1.6.0",
+    "londonaicentre-mesa-types>=2.3.0",
+    "londonaicentre-mesa-runtime>=0.4.0",
+]
+[build-system]
+requires = ["setuptools>=80.9.0"]
+build-backend = "setuptools.build_meta"
+[tool.setuptools.package-data]
+docsynth = [
+    "assets/*/profiles/*",
+    "assets/*/prompts/*",
+    "assets/*/structure/*",
+    "assets/*/*.yml",
+    "config/*.yml",
+    "config/*.json",
+    "py.typed"
+]
+[tool.setuptools.exclude-package-data]
+docsynth = ["assets/_domain/*", "assets/_domain/*/*"]
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "pytest-cov>=7.0.0",
+    "pytest-mock>=3.15.1",
+    "ruff>=0.14.6",
+    "ty>=0.0.56",
+    "types-pyyaml>=6.0.12.20250915",
+]
+

londonaicentre_docsynth-1.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

londonaicentre_docsynth-1.1.0/src/docsynth/assets/cancer/content.yml

@@ -0,0 +1,297 @@
+# PROMPT GENERATION CONFIG - CONTENT
+## Each section contains elements that can be sampled probabilistically 
+## Descriptions are prompt-ready and will be inserted directly into LLM prompts
+## mutually_exclusive: true = exactly ONE item selected from section (weighted random choice)
+## mutually_exclusive: false = each item sampled independently (based on independent probability)
+
+###################
+# TIMELINE EVENTS #
+###################
+timeline_events:
+  mutually_exclusive: false
+
+  clinical_trial_consideration:
+    probability: 0.20
+    description: "The patient was discussed for potential clinical trial enrollment."
+
+  clinical_trial_enrollment:
+    probability: 0.15
+    description: "The patient was enrolled in a clinical trial."
+
+  patient_death:
+    probability: 0.05
+    description: "The patient died."
+
+  surgery:
+    probability: 0.65
+    description: "The patient has had one or more surgical resection of the primary tumor."
+
+  systemic_therapy_initiated:
+    probability: 0.75
+    description: "The patient has had one or more systemic therapies (chemotherapy, immunotherapy, or targeted therapy)."
+
+  treatment_toxicity:
+    probability: 0.35
+    description: "The patient has experienced one or more treatment-related toxicity or adverse effects."
+
+  treatment_reduction_or_stop:
+    probability: 0.25
+    description: "The patient has experienced one or more treatments being dose-reduced, delayed, or discontinued due to toxicity or other factors."
+
+  positive_treatment_response:
+    probability: 0.30
+    description: "There were one or more events where imaging or clinical assessment demonstrated positive response to treatment (partial response, stable disease, or symptomatic improvement)."
+
+  disease_progression_imaging:
+    probability: 0.30
+    description: "There were one or more events where radiological imaging showed evidence of disease progression."
+
+  disease_remission:
+    probability: 0.15
+    description: "The patient achieved complete or partial remission at any point."
+
+  disease_recurrence:
+    probability: 0.20
+    description: "Disease recurrence or relapse was detected after previous treatment at any point."
+
+  radiotherapy:
+    probability: 0.45
+    description: "The patient has had one or more radiotherapy treatments."
+
+  new_metastatic_spread:
+    probability: 0.25
+    description: "There were one or more events where new metastatic disease was identified at nodal or distant sites."
+
+####################
+# PATIENT FINDINGS #
+####################
+patient_findings:
+  mutually_exclusive: false
+
+  comorbidity:
+    probability: 0.60
+    description: "Include one or more comorbid conditions."
+
+  social_history:
+    probability: 0.50
+    description: "Include one or more social history details such as smoking status, alcohol use, living situation, or social support."
+
+  family_history:
+    probability: 0.15
+    description: "Include relevant family history of cancer or other conditions."
+
+  symptoms_present:
+    probability: 0.65
+    description: "Include reporting of one or more symptoms (e.g., pain, fatigue, dyspnea, weight loss, nausea)."
+
+  symptoms_resolved:
+    probability: 0.30
+    description: "Include reporting that previously reported one or more symptoms have now resolved or improved."
+
+  physical_exam_positive:
+    probability: 0.40
+    description: "Include one or more positive findings of physical examination (e.g., palpable mass, lymphadenopathy, organomegaly, cachexia)."
+
+#####################
+# FUNCTIONAL STATUS #
+#####################
+functional_status:
+  mutually_exclusive: true
+
+  good:
+    probability: 0.40
+    description: "The patient has good functional status (e.g. any of ECOG 0-1, Karnofsky 90-100, independent in ADLs)."
+
+  poor:
+    probability: 0.40
+    description: "The patient has impaired functional status (e.g. any of ECOG 2-3, reduced mobility, requires assistance)."
+
+  omit:
+    probability: 0.20
+    description: ""
+
+################
+# MENTAL STATE #
+################
+mental_state:
+  mutually_exclusive: true
+
+  positive:
+    probability: 0.40
+    description: "The patient demonstrates positive mental state (e.g., coping well, optimistic, good spirits)."
+
+  distressed:
+    probability: 0.30
+    description: "The patient shows signs of psychological distress (e.g., anxiety, depression, adjustment difficulties)."
+
+  omit:
+    probability: 0.30
+    description: ""
+
+##################
+# DIAGNOSIS DATE #
+##################
+diagnosis_date:
+  mutually_exclusive: true
+
+  include_year_and_month:
+    probability: 0.50
+    description: "Include specific diagnosis month and year."
+
+  include_year_only:
+    probability: 0.30
+    description: "Include diagnosis year only, without specific month."
+
+  omit:
+    probability: 0.20
+    description: ""
+
+######################
+# DISEASE TRAJECTORY #
+######################
+disease_trajectory:
+  mutually_exclusive: true
+
+  stable:
+    probability: 0.40
+    description: "The disease course has been stable with no significant progression."
+
+  improving:
+    probability: 0.30
+    description: "The patient's condition has been improving with treatment response evident."
+
+  slowly_progressing:
+    probability: 0.15
+    description: "The disease has been slowly progressing despite treatment."
+
+  rapidly_deteriorating:
+    probability: 0.10
+    description: "The patient's condition has deteriorated rapidly with progressive disease."
+
+  remission:
+    probability: 0.05
+    description: "The patient is in complete or partial remission."
+
+########################
+# TREATMENT COMPLEXITY #
+########################
+treatment_complexity:
+  mutually_exclusive: true
+
+  simple:
+    probability: 0.35
+    description: "Treatment history is straightforward with single or first-line therapy."
+
+  moderate:
+    probability: 0.40
+    description: "Treatment involves multiple modalities or second-line therapy."
+
+  complex:
+    probability: 0.25
+    description: "Treatment history is complex with multiple lines of therapy, combination treatments, or clinical trial involvement."
+
+#####################
+# HISTORICAL CANCER #
+#####################
+historical_cancer:
+  mutually_exclusive: true
+
+  no_historical_cancer:
+    probability: 0.6
+    description: "The primary cancer is the patient's only cancer diagnosis"
+
+  historical_cancer_mentioned:
+    probability: 0.25
+    description: "In addition to the primary cancer, the patient has had another unrelated cancer diagnosis"
+
+  competing_historical_cancer:
+    probability: 0.15
+    description: "The patient has another active cancer with a described treatment timeline, although it is not the main subject of this document"
+    
+###############
+# TNM STAGING #
+###############
+tnm_staging:
+  mutually_exclusive: true
+
+  include_full_tnm:
+    probability: 0.50
+    description: "Include complete TNM staging in any format (e.g., pT2N1M0, cT3N2M0, pT2 N2 M1)."
+
+  include_partial_tnm:
+    probability: 0.35
+    description: "Include incomplete or partial TNM information in any format (e.g., T2N0, stage unknown, M3)."
+
+  omit:
+    probability: 0.15
+    description: ""
+
+#################
+# OTHER STAGING #
+#################
+other_staging:
+  mutually_exclusive: true
+
+  include_numeric_stage:
+    probability: 0.50
+    description: "Include numeric/group staging in any format (e.g., Stage IIA, Stage 4)."
+
+  omit:
+    probability: 0.50
+    description: ""
+
+#################
+# CANCER SCORES #
+#################
+cancer_scores:
+  mutually_exclusive: false
+
+  pathological_grade:
+    probability: 0.30
+    description: "Include an appropriate pathological grade (e.g., Grade 2, poorly differentiated, G3)."
+
+  other_score:
+    probability: 0.30
+    description: "Include at least one clinically relevant scoring system (e.g., Dukes, Binet, Child-Pugh, Clark level)."
+
+  tmb:
+    probability: 0.30
+    description: "Include an appropriate mention of tumour mutational burden."
+
+##############
+# METASTASES #
+##############
+metastatic_spread:
+  mutually_exclusive: true
+
+  no_metastases:
+    probability: 0.40
+    description: "Patient has no evidence of metastatic disease - mention explicitly, or do not mention metastases at all."
+
+  lymph_node_only:
+    probability: 0.25
+    description: "Patient has regional lymph node involvement only."
+
+  liver_metastases:
+    probability: 0.10
+    description: "Patient has hepatic metastases present."
+
+  lung_metastases:
+    probability: 0.10
+    description: "Patient has pulmonary metastases present."
+
+  bone_metastases:
+    probability: 0.10
+    description: "Patient has bone or spinal metastases present."
+
+  brain_metastases:
+    probability: 0.10
+    description: "Patient has brain metastases present."
+
+  peritoneal_spread:
+    probability: 0.04
+    description: "Patient has peritoneal metastases present."
+
+  multiple_sites:
+    probability: 0.15
+    description: "Patient has widespread metastatic disease involving multiple sites."

londonaicentre_docsynth-1.1.0/src/docsynth/assets/cancer/profiles/README.md

@@ -0,0 +1 @@
+This folder contains hand-crafted (LLM assisted) profiles that represent real combinations of topography/morphology and clinically relevant biomarkers. A descriptive name is given for each cancer diagnosis that represents different real-world ways of expressing the combination, including abbreviations.