vivarium-validation 0.1.0__tar.gz

vivarium_validation-0.1.0/.readthedocs.yaml

@@ -0,0 +1,37 @@
+# Paths are repo-root-relative even when this file lives in a subdirectory,
+# so the package path must be qualified with libs/validation/.
+#
+# This file is read by RTD only when the vivarium-validation RTD
+# project's Admin > Advanced Settings > "Path for .readthedocs.yaml" points
+# here. The root-level /.readthedocs.yaml is a placeholder used by the
+# import wizard and ignored for any project that has overridden the path.
+version: 2
+
+sphinx:
+  configuration: libs/validation/docs/source/conf.py
+  fail_on_warning: true
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.13"
+  jobs:
+    post_checkout:
+      - git fetch --unshallow origin 2>/dev/null || git fetch origin main
+      # Cancel PR (external) builds that don't touch this lib (Exit 183 is RTD's
+      # signal to cancel the build immediately).
+      - |
+        if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet "origin/main...HEAD" -- libs/validation/; then
+          echo "Skipping docs build: this PR changes nothing under libs/validation/."
+          exit 183
+        fi
+
+python:
+  install:
+    - method: pip
+      path: libs/validation
+      extra_requirements:
+        - docs
+
+formats:
+  - pdf

vivarium_validation-0.1.0/CHANGELOG.rst

@@ -0,0 +1,4 @@
+**0.1.0 - 07/21/26**
+
+- Initial release: automated verification and validation (V&V) tooling split out
+  of ``vivarium-testing-utils`` into its own package.

vivarium_validation-0.1.0/Jenkinsfile

@@ -0,0 +1,40 @@
+/* This Jenkinsfile simply loads the `reusable_pipeline` pipeline from the
+vivarium-build-utils package, which lives at ``libs/build-utils/`` inside the
+``ihmeuw/vivarium-suite`` monorepo (https://github.com/ihmeuw/vivarium-suite).
+
+vivarium-build-utils is loaded as a Jenkins shared library
+(https://www.jenkins.io/doc/book/pipeline/shared-libraries/).
+Jenkins shared library convention dictates that importable modules must be stored
+in the 'vars' folder.
+
+Jenkins shared libraries can be configured in the Jenkins UI:
+* Manage Jenkins
+  * Configure System
+    * Global Pipeline Libraries section
+      * Library subsection
+        * Name: The Name for the lib
+        * Version: The branch you want to use. Throws an error
+                   for nonexistent branches.
+        * Project Repository: Url to the shared lib
+        * Credentials: SSH key to access the repo
+
+Note that updating the shared repo will take effect on the next pipeline invocation.
+*/
+
+// Load the get_vbu_version function from build-utils/bootstrap/
+// (the directory to load from is defined in the Jenkins shared library configuration)
+@Library("get_vbu_version@main") _
+
+// Load the full vivarium_build_utils library at the expected version
+library("vivarium_build_utils@${get_vbu_version()}")
+
+reusable_pipeline(
+    scheduled_branches: [
+        // Add additional branches for cron jobs below.
+        // Note: Newly added branches will require a manual trigger of the first build
+        // for subsequent scheduled builds to run.
+        "main",
+    ],
+    skip_doc_build: true,
+    env_reqs: "ci_jenkins",
+)

vivarium_validation-0.1.0/LICENSE

@@ -0,0 +1,27 @@
+BSD 3-Clause License
+
+Copyright 2024 Institute for Health Metrics and Evaluation
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list
+   of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this
+   list of conditions and the following disclaimer in the documentation and/or other
+   materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may
+   be used to endorse or promote products derived from this software without specific
+   prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vivarium_validation-0.1.0/Makefile

@@ -0,0 +1,101 @@
+# Check if we're running in Jenkins
+ifdef JENKINS_URL
+# 	Files are already in workspace from shared library
+	MAKE_INCLUDES := .
+else
+# 	For local dev, use the installed vivarium.build_utils package if it exists
+# 	First, check if we can import vivarium.build_utils and assign 'yes' or 'no'.
+# 	We do this by importing the package in python and redirecting stderr to the null device.
+# 	If the import is successful (&&), it will print 'yes', otherwise (||) it will print 'no'.
+	VIVARIUM_BUILD_UTILS_AVAILABLE := $(shell python -c "import vivarium.build_utils" 2>/dev/null && echo "yes" || echo "no")
+# 	If vivarium.build_utils is available, get the makefiles path or else set it to empty
+	ifeq ($(VIVARIUM_BUILD_UTILS_AVAILABLE),yes)
+		MAKE_INCLUDES := $(shell python -c "from vivarium.build_utils.resources import get_makefiles_path; print(get_makefiles_path())")
+	else
+		MAKE_INCLUDES :=
+	endif
+endif
+
+# Set the package name as the last part of this file's parent directory path
+PACKAGE_NAME = $(notdir $(CURDIR))
+
+ifneq ($(MAKE_INCLUDES),) # not empty
+# Include makefiles from vivarium.build_utils
+include $(MAKE_INCLUDES)/base.mk
+include $(MAKE_INCLUDES)/test.mk
+else # empty
+# Use this help message (since the vivarium.build_utils version is not available)
+help:
+	@echo
+	@echo "For Make's standard help, run 'make --help'."
+	@echo
+	@echo "Most of our Makefile targets are provided by the vivarium.build_utils"
+	@echo "package. To access them, you need to create a development environment first."
+	@echo
+	@echo "make build-env"
+	@echo
+	@echo "USAGE:"
+	@echo "  make build-env [name=<environment name>] [py=<python version>] [artifactory_access=<yes|no>]"
+	@echo
+	@echo "ARGUMENTS:"
+	@echo "  name [optional]"
+	@echo "      Name of the conda environment to create (defaults to <PACKAGE_NAME>)"
+	@echo "  py [optional]"
+	@echo "      Python version (defaults to latest supported)"
+	@echo "  artifactory_access [optional]"
+	@echo "      Whether the build host can reach the IHME artifactory. Either 'yes'"
+	@echo "      (default) or 'no'; use 'no' to build off-VPN without artifactory."
+	@echo
+	@echo "After creating the environment:"
+	@echo "  1. Activate it: 'conda activate <environment_name>'"
+	@echo "  2. Run 'make help' again to see all newly available targets"
+	@echo
+endif
+
+build-env: # Create a new environment with installed packages
+#	Validate arguments - exit if unsupported arguments are passed
+	@allowed="name py artifactory_access"; \
+	for arg in $(filter-out build-env,$(MAKECMDGOALS)) $(MAKEFLAGS); do \
+		case $$arg in \
+			*=*) \
+				arg_name=$${arg%%=*}; \
+				if ! echo " $$allowed " | grep -q " $$arg_name "; then \
+					allowed_list=$$(echo $$allowed | sed 's/ /, /g'); \
+					echo "Error: Invalid argument '$$arg_name'. Allowed arguments are: $$allowed_list" >&2; \
+					exit 1; \
+				fi \
+				;; \
+		esac; \
+	done
+	
+#   Handle arguments and set defaults
+#	name
+	@$(eval name ?= $(PACKAGE_NAME))
+#	python version - validate if specified, else get default from json file
+	@supported_versions=$$(python -c "import json; print(' '.join(json.load(open('python_versions.json'))))" 2>/dev/null || echo ""); \
+	if [ -n "$(py)" ] && ! echo "$$supported_versions" | grep -q "$(py)"; then \
+		echo "Error: Python version '$(py)' is not supported. Available: $$(echo $$supported_versions | sed 's/ /, /g')" >&2; \
+		exit 1; \
+	fi
+	@$(eval py ?= $(shell python -c "import json; print(max(json.load(open('python_versions.json')), key=lambda x: tuple(map(int, x.split('.')))))"))
+#	artifactory_access
+	@$(eval artifactory_access ?= yes)
+	@case "$(artifactory_access)" in \
+		yes|no) ;; \
+		*) echo "Error: 'artifactory_access' must be 'yes' or 'no', got '$(artifactory_access)'" >&2; exit 1 ;; \
+	esac
+
+	conda create -n $(name) python=$(py) --yes
+# 	Bootstrap vivarium.build_utils into the new environment
+	conda run -n $(name) pip install vivarium-build-utils
+# 	artifactory_access=no passes an empty IHME_PYPI, which clears base.mk's EXTRA_INDEX_FLAGS
+	conda run -n $(name) make install $(if $(filter no,$(artifactory_access)),IHME_PYPI=)
+
+	@echo
+	@echo "Finished building environment"
+	@echo "  name: $(name)"
+	@echo "  python version: $(py)"
+	@echo
+	@echo "Don't forget to activate it with:"
+	@echo "conda activate $(name)"
+	@echo

vivarium_validation-0.1.0/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.4
+Name: vivarium-validation
+Version: 0.1.0
+Summary: Automated verification and validation tooling for Vivarium simulations.
+Author-email: The vivarium developers <vivarium.dev@gmail.com>
+License: BSD-3-Clause
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Natural Language :: English
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: POSIX
+Classifier: Operating System :: POSIX :: BSD
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Topic :: Education
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/x-rst
+License-File: LICENSE
+Requires-Dist: vivarium-dependencies[loguru,numpy,pandas,pyyaml]
+Requires-Dist: vivarium-build-utils<5.0.0,>=4.0.0
+Requires-Dist: vivarium-config-tree
+Requires-Dist: vivarium-fuzzy-checker
+Provides-Extra: cluster
+Provides-Extra: test
+Requires-Dist: vivarium-dependencies[pytest]; extra == "test"
+Requires-Dist: pytest-check; extra == "test"
+Requires-Dist: vivarium-testing-utils; extra == "test"
+Provides-Extra: docs
+Requires-Dist: vivarium-dependencies[docutils,ipython,matplotlib,sphinx,sphinx-click]; extra == "docs"
+Provides-Extra: interactive
+Requires-Dist: vivarium-dependencies[interactive]; extra == "interactive"
+Provides-Extra: pandera
+Requires-Dist: pandera<0.23.0; extra == "pandera"
+Provides-Extra: lint
+Requires-Dist: vivarium-dependencies[lint]; extra == "lint"
+Requires-Dist: vivarium-validation[pandera]; extra == "lint"
+Provides-Extra: validation
+Requires-Dist: vivarium-artifact; extra == "validation"
+Requires-Dist: vivarium-gbd-mapping; extra == "validation"
+Requires-Dist: vivarium-inputs<9.0.0,>=8.0.0; extra == "validation"
+Requires-Dist: vivarium-validation[pandera]; extra == "validation"
+Requires-Dist: pyarrow; extra == "validation"
+Requires-Dist: seaborn; extra == "validation"
+Requires-Dist: jinja2>=3.0.0; extra == "validation"
+Requires-Dist: vivarium-dependencies[ipython,matplotlib]; extra == "validation"
+Provides-Extra: dev
+Requires-Dist: vivarium-validation[docs,interactive,lint,test,validation]; extra == "dev"
+Provides-Extra: ci-github
+Requires-Dist: vivarium-validation[docs,lint,test]; extra == "ci-github"
+Provides-Extra: ci-jenkins
+Requires-Dist: vivarium-validation[ci_github,cluster,validation]; extra == "ci-jenkins"
+Dynamic: license-file
+
+============================
+Vivarium Validation
+============================
+
+.. image:: https://badge.fury.io/py/vivarium-validation.svg
+    :target: https://badge.fury.io/py/vivarium-validation
+
+.. image:: https://readthedocs.org/projects/vivarium-validation/badge/?version=latest
+    :target: https://vivarium-validation.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
+
+This library provides tooling for automated verification and validation (V&V) of
+Vivarium simulations, including data loading, measure comparison, and reporting.
+
+**Supported Python versions: 3.10, 3.11**
+
+You can install ``vivarium-validation`` from PyPI with pip:
+
+.. code-block:: bash
+
+   pip install vivarium-validation
+
+or build it from source by cloning the monorepo and installing this package:
+
+.. code-block:: bash
+
+   git clone https://github.com/ihmeuw/vivarium-suite.git
+   cd vivarium-suite
+   pip install libs/validation
+
+Note that the validation feature depends on ``vivarium-inputs``, which is only
+available from the IHME artifactory.
+
+For broader monorepo development setup, see the monorepo README at
+https://github.com/ihmeuw/vivarium-suite.
+
+`Check out the docs! <https://vivarium-validation.readthedocs.io/en/latest/>`_
+-----------------------------------------------------------------------------------

vivarium_validation-0.1.0/README.rst

@@ -0,0 +1,38 @@
+============================
+Vivarium Validation
+============================
+
+.. image:: https://badge.fury.io/py/vivarium-validation.svg
+    :target: https://badge.fury.io/py/vivarium-validation
+
+.. image:: https://readthedocs.org/projects/vivarium-validation/badge/?version=latest
+    :target: https://vivarium-validation.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
+
+This library provides tooling for automated verification and validation (V&V) of
+Vivarium simulations, including data loading, measure comparison, and reporting.
+
+**Supported Python versions: 3.10, 3.11**
+
+You can install ``vivarium-validation`` from PyPI with pip:
+
+.. code-block:: bash
+
+   pip install vivarium-validation
+
+or build it from source by cloning the monorepo and installing this package:
+
+.. code-block:: bash
+
+   git clone https://github.com/ihmeuw/vivarium-suite.git
+   cd vivarium-suite
+   pip install libs/validation
+
+Note that the validation feature depends on ``vivarium-inputs``, which is only
+available from the IHME artifactory.
+
+For broader monorepo development setup, see the monorepo README at
+https://github.com/ihmeuw/vivarium-suite.
+
+`Check out the docs! <https://vivarium-validation.readthedocs.io/en/latest/>`_
+-----------------------------------------------------------------------------------

vivarium_validation-0.1.0/docs/Makefile

@@ -0,0 +1,22 @@
+# Minimal makefile for Sphinx documentation
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = vivarium.validation
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+clean:
+	rm -rf build/*

vivarium_validation-0.1.0/docs/nitpick-exceptions

@@ -0,0 +1,4 @@
+# Add "py:class <target>" entries here for cross-references that Sphinx's
+# nitpicky mode cannot resolve. Empty for now: these docs are narrative-only,
+# so no autodoc-generated type references need suppressing. Populate this when
+# the package starts auto-documenting its API.

vivarium_validation-0.1.0/docs/source/automated_v_and_v/index.rst

@@ -0,0 +1,168 @@
+..
+  Section title decorators for this document:
+  
+  ==============
+  Document Title
+  ==============
+  Section Level 1
+  ---------------
+  Section Level 2
+  +++++++++++++++
+  Section Level 3
+  ~~~~~~~~~~~~~~~
+  Section Level 4
+  ^^^^^^^^^^^^^^^
+  Section Level 5
+  '''''''''''''''
+
+  The depth of each section level is determined by the order in which each
+  decorator is encountered below. If you need an even deeper section level, just
+  choose a new decorator symbol from the list here:
+  https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#sections
+  And then add it to the list of decorators above.
+
+.. _automated_v_and_v:
+
+.. role:: underline
+    :class: underline
+
+=========================================================
+Automated verification and validation (V&V)
+=========================================================
+
+.. contents::
+   :local:
+
+Background/Motivation
+---------------------
+
+Our team uses :ref:`a process called verification and validation (V&V) <vivarium_best_practices_results_processing>`
+to ensure that our models are behaving as we expect.
+:ref:`The current process for V&V <vivarium_v_and_v_process>`
+works well but can be quite labor intensive.
+Due to the long runtime of simulations and the number of people involved,
+each V&V cycle generally takes *at least* one day, which makes it difficult to find and fix issues quickly.
+
+Automating more of the V&V process would help us catch issues faster and with less work.
+We can **incrementally** automate V&V by building tools to do the repetitive parts,
+and over time stitching these tools together to make the process more and more automatic.
+
+As we develop the tools, there are also some process questions to work out about this transformation:
+
+* Who investigates the cause of automated V&V failures?
+  If the answer is "it depends on the failure," how do we route the right failures to the right people?
+* How and when do we run automated V&V?
+
+Automated V&V comparisons
+-------------------------
+
+Automated V&V plots
+-------------------
+
+Automated V&V checks
+--------------------
+
+Automating V&V checks refers to replacing the manual process of looking at graphs (to check that simulation
+results are close enough to known targets) with an automatic equivalent.
+This requires transforming the manual process into something that can be executed by a computer
+and generate an unambiguous verdict on whether or not there is an issue that needs to be investigated.
+
+.. note:: 
+  In the future, we could explore having a result more complex than just "investigate" or
+  "don't investigate," such as multiple levels of priority for potential issues to be reviewed by a human.
+  For now, we stick to a binary result.
+
+For checks where the expectation is that the simulation result should exactly match a target or respect
+a constraint (e.g. 0% coverage of an intervention among those not eligible) it is fairly straightforward
+to use a Python :code:`assert` statement in addition to, or instead of, a plot.
+This approach can be used in notebooks or in automated tests run with :code:`pytest`.
+
+The statistical methodology behind fuzzy checking lives in the
+`vivarium-fuzzy-checker documentation
+<https://vivarium-fuzzy-checker.readthedocs.io/en/latest/fuzzy_checking.html>`_.
+
+Automated V&V runs
+------------------
+
+.. todo::
+  Due to technical limitations in the :code:`pytest` tool, integration tests currently must select a simulation
+  "size" (population, draws, time span), run it completely, and then check the results.
+  It would likely lead to a much quicker iteration cycle if we ran a small simulation, checked the results,
+  then added more population/draws/time and checked the results again, etc, similar to how we expand runs with
+  :code:`psimulate`.
+  This way, egregious bugs could be caught very quickly.
+
+History/case studies
+--------------------
+
+Probabilistic record linkage (PRL) synthetic population simulation
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+The :ref:`PRL synthetic population simulation <vivarium_census_prl_synth_data>`
+used an atypical V&V process,
+and was used as a testbed for some of these automation ideas.
+
+Rather than researchers performing V&V in notebooks, V&V for this simulation
+was entirely done via `integration tests <https://en.wikipedia.org/wiki/Integration_testing>`_,
+which ran the simulation for a given number of time steps and checked certain conditions.
+When the model exhibited unexpected behavior,
+the tests failed.
+For example, there was a test that checked that each non-GQ household in the simulation had exactly one
+living reference person, a condition which should always be true.
+
+These tests have access to intermediate simulation states, not only simulation outputs, and can check
+conditions at each time step.
+Therefore, they are quite similar to :ref:`interactive simulation V&V <vivarium_interactive_simulation>` we have done in the past.
+Because not much changes over time in the PRL simulation (there is nothing like an intervention scale-up),
+we test only the first 10 time steps.
+
+Test coverage was not complete; notably, none of the observers had integration tests.
+The test suite can be found `in the simulation repository's integration tests directory <https://github.com/ihmeuw/vivarium_census_prl_synth_pop/tree/main/tests/integration>`_.
+The tests all ran in just a few minutes, so engineers were expected to run them before merging any pull request,
+though we did not have continuous integration (CI).
+
+These integration tests were the first place we used fuzzy checking,
+specifically for rates of simulant migration into and within the US (fuzzy checking was not implemented for emigration).
+These rates are stratified by a number
+of demographic factors, and some of these factors (e.g. race/ethnicity) have highly imbalanced categories.
+Therefore, verifying rates within each demographic combination would require a large population size.
+
+Instead, the integration tests do a combination of verification and validation by checking
+**population-level** migration rates against the corresponding rates in our data source (the American Communities Survey).
+These should be similar, since the simulation's rates are calculated using this data source,
+and the demographic composition of the population is initialized from the same data.
+However, simulation rates can drift slightly from population-level rates in the data, without being indicative of a bug,
+due to demographic change over the course of the simulation.
+All these checks were implemented as proportion checks on the proportion of simulants experiencing the vent.
+Checking at the population level makes use of the binomial approximation to the Poisson binomial,
+as described in the proportions section of the `vivarium-fuzzy-checker documentation
+<https://vivarium-fuzzy-checker.readthedocs.io/en/latest/fuzzy_checking.html>`_.
+
+For rates of migration within the US, we check the migration rate at each time step, and overall.
+We set the target range for each time step by assuming with 95% certainty that the drift will be at most 1% per time step that has elapsed
+since initialization.
+Overall, we set a UI of +/-10% the ACS value.
+
+Migration into the US is a bit different; it is not an event with a rate of occurrence among
+an at-risk population.
+The only stochastic part of determining the number of immigration events is the
+:ref:`"stochastic rounding" used <census_prl_international_immigration>`.
+We check this rounding as a set of Bernoulli trials, one per time step:
+whether to round up or down.
+
+The PRL integration tests are run very frequently by the software engineering team.
+Due to how frequently they are run and the difficulty of debugging a failed test
+(perhaps requiring researcher input in some cases),
+it is important for these tests to be highly **specific**;
+they should very rarely fail by chance.
+That is the main reason we have set the default Bayes factor cutoff to 100, commonly called "decisive,"
+in *addition* to the generally conservative approximations described above.
+In practice, by manually introducing bugs in the simulation, we have found that even with this very conservative approach, automated V&V is quite sensitive.
+
+pseudopeople noise tests
+++++++++++++++++++++++++
+
+Our `pseudopeople <https://pseudopeople.readthedocs.io/>`_ package applies random noise to data derived from the PRL synthetic population simulation.
+We used fuzzy checking in both the `unit <https://github.com/ihmeuw/pseudopeople/tree/main/tests/unit>`_ and `integration <https://github.com/ihmeuw/pseudopeople/tree/main/tests/integration>`_ tests of pseudopeople to check that
+noise was happening at the expected rates.
+We did not have any false alarms, and fuzzy checking caught some `very subtle bugs <https://github.com/ihmeuw/pseudopeople/pull/373>`_.