ob-metaflow 2.15.0.1__tar.gz → 2.15.5.1__tar.gz

ob_metaflow-2.15.5.1/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.2
+Name: ob-metaflow
+Version: 2.15.5.1
+Summary: Metaflow: More AI and ML, Less Engineering
+Author: Netflix, Outerbounds & the Metaflow Community
+Author-email: help@outerbounds.co
+License: Apache License 2.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: boto3
+Requires-Dist: pylint
+Requires-Dist: kubernetes
+Provides-Extra: stubs
+Requires-Dist: metaflow-stubs==2.15.5.1; extra == "stubs"
+Dynamic: author
+Dynamic: author-email
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: summary
+
+![Metaflow_Logo_Horizontal_FullColor_Ribbon_Dark_RGB](https://user-images.githubusercontent.com/763451/89453116-96a57e00-d713-11ea-9fa6-82b29d4d6eff.png)
+
+# Metaflow
+
+[Metaflow](https://metaflow.org) is a human-centric framework designed to help scientists and engineers **build and manage real-life AI and ML systems**. Serving teams of all sizes and scale, Metaflow streamlines the entire development lifecycle—from rapid prototyping in notebooks to reliable, maintainable production deployments—enabling teams to iterate quickly and deliver robust systems efficiently.
+
+Originally developed at [Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9) and now supported by [Outerbounds](https://outerbounds.com), Metaflow is designed to boost the productivity for research and engineering teams working on [a wide variety of projects](https://netflixtechblog.com/supporting-diverse-ml-systems-at-netflix-2d2e6b6d205d), from classical statistics to state-of-the-art deep learning and foundation models. By unifying code, data, and compute at every stage, Metaflow ensures seamless, end-to-end management of real-world AI and ML systems.
+
+Today, Metaflow powers thousands of AI and ML experiences across a diverse array of companies, large and small, including Amazon, Doordash, Dyson, Goldman Sachs, Ramp, and [many others](ADOPTERS.md). At Netflix alone, Metaflow supports over 3000 AI and ML projects, executes hundreds of millions of data-intensive high-performance compute jobs processing petabytes of data and manages tens of petabytes of models and artifacts for hundreds of users across its AI, ML, data science, and engineering teams.
+
+## From prototype to production (and back)
+
+Metaflow provides a simple and friendly pythonic [API](https://docs.metaflow.org) that covers foundational needs of AI and ML systems:
+<img src="./docs/prototype-to-prod.png" width="800px">
+
+1. [Rapid local prototyping](https://docs.metaflow.org/metaflow/basics), [support for notebooks](https://docs.metaflow.org/metaflow/managing-flows/notebook-runs), and built-in support for [experiment tracking, versioning](https://docs.metaflow.org/metaflow/client) and [visualization](https://docs.metaflow.org/metaflow/visualizing-results).
+2. [Effortlessly scale horizontally and vertically in your cloud](https://docs.metaflow.org/scaling/remote-tasks/introduction), utilizing both CPUs and GPUs, with [fast data access](https://docs.metaflow.org/scaling/data) for running [massive embarrassingly parallel](https://docs.metaflow.org/metaflow/basics#foreach) as well as [gang-scheduled](https://docs.metaflow.org/scaling/remote-tasks/distributed-computing) compute workloads [reliably](https://docs.metaflow.org/scaling/failures) and [efficiently](https://docs.metaflow.org/scaling/checkpoint/introduction).
+3. [Easily manage dependencies](https://docs.metaflow.org/scaling/dependencies) and [deploy with one-click](https://docs.metaflow.org/production/introduction) to highly available production orchestrators with built in support for [reactive orchestration](https://docs.metaflow.org/production/event-triggering).
+
+For full documentation, check out our [API Reference](https://docs.metaflow.org/api) or see our [Release Notes](https://github.com/Netflix/metaflow/releases) for the latest features and improvements. 
+
+
+## Getting started
+
+Getting up and running is easy. If you don't know where to start, [Metaflow sandbox](https://outerbounds.com/sandbox) will have you running and exploring in seconds.
+
+### Installing Metaflow
+
+To install Metaflow in your Python environment from [PyPI](https://pypi.org/project/metaflow/):
+
+```sh
+pip install metaflow
+```
+Alternatively, using [conda-forge](https://anaconda.org/conda-forge/metaflow):
+
+```sh
+conda install -c conda-forge metaflow
+```
+
+Once installed, a great way to get started is by following our [tutorial](https://docs.metaflow.org/getting-started/tutorials). It walks you through creating and running your first Metaflow flow step by step.  
+
+For more details on Metaflow’s features and best practices, check out:
+- [How Metaflow works](https://docs.metaflow.org/metaflow/basics)  
+- [Additional resources](https://docs.metaflow.org/introduction/metaflow-resources)  
+
+If you need help, don’t hesitate to reach out on our [Slack community](http://slack.outerbounds.co/)!
+
+
+### Deploying infrastructure for Metaflow in your cloud
+<img src="./docs/multicloud.png" width="800px">
+
+
+While you can get started with Metaflow easily on your laptop, the main benefits of Metaflow lie in its ability to [scale out to external compute clusters](https://docs.metaflow.org/scaling/remote-tasks/introduction) 
+and to [deploy to production-grade workflow orchestrators](https://docs.metaflow.org/production/introduction). To benefit from these features, follow this [guide](https://outerbounds.com/engineering/welcome/) to 
+configure Metaflow and the infrastructure behind it appropriately.
+
+
+## Get in touch
+We'd love to hear from you. Join our community [Slack workspace](http://slack.outerbounds.co/)!
+
+## Contributing
+We welcome contributions to Metaflow. Please see our [contribution guide](https://docs.metaflow.org/introduction/contributing-to-metaflow) for more details.
+

ob_metaflow-2.15.5.1/README.md

@@ -0,0 +1,63 @@
+![Metaflow_Logo_Horizontal_FullColor_Ribbon_Dark_RGB](https://user-images.githubusercontent.com/763451/89453116-96a57e00-d713-11ea-9fa6-82b29d4d6eff.png)
+
+# Metaflow
+
+[Metaflow](https://metaflow.org) is a human-centric framework designed to help scientists and engineers **build and manage real-life AI and ML systems**. Serving teams of all sizes and scale, Metaflow streamlines the entire development lifecycle—from rapid prototyping in notebooks to reliable, maintainable production deployments—enabling teams to iterate quickly and deliver robust systems efficiently.
+
+Originally developed at [Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9) and now supported by [Outerbounds](https://outerbounds.com), Metaflow is designed to boost the productivity for research and engineering teams working on [a wide variety of projects](https://netflixtechblog.com/supporting-diverse-ml-systems-at-netflix-2d2e6b6d205d), from classical statistics to state-of-the-art deep learning and foundation models. By unifying code, data, and compute at every stage, Metaflow ensures seamless, end-to-end management of real-world AI and ML systems.
+
+Today, Metaflow powers thousands of AI and ML experiences across a diverse array of companies, large and small, including Amazon, Doordash, Dyson, Goldman Sachs, Ramp, and [many others](ADOPTERS.md). At Netflix alone, Metaflow supports over 3000 AI and ML projects, executes hundreds of millions of data-intensive high-performance compute jobs processing petabytes of data and manages tens of petabytes of models and artifacts for hundreds of users across its AI, ML, data science, and engineering teams.
+
+## From prototype to production (and back)
+
+Metaflow provides a simple and friendly pythonic [API](https://docs.metaflow.org) that covers foundational needs of AI and ML systems:
+<img src="./docs/prototype-to-prod.png" width="800px">
+
+1. [Rapid local prototyping](https://docs.metaflow.org/metaflow/basics), [support for notebooks](https://docs.metaflow.org/metaflow/managing-flows/notebook-runs), and built-in support for [experiment tracking, versioning](https://docs.metaflow.org/metaflow/client) and [visualization](https://docs.metaflow.org/metaflow/visualizing-results).
+2. [Effortlessly scale horizontally and vertically in your cloud](https://docs.metaflow.org/scaling/remote-tasks/introduction), utilizing both CPUs and GPUs, with [fast data access](https://docs.metaflow.org/scaling/data) for running [massive embarrassingly parallel](https://docs.metaflow.org/metaflow/basics#foreach) as well as [gang-scheduled](https://docs.metaflow.org/scaling/remote-tasks/distributed-computing) compute workloads [reliably](https://docs.metaflow.org/scaling/failures) and [efficiently](https://docs.metaflow.org/scaling/checkpoint/introduction).
+3. [Easily manage dependencies](https://docs.metaflow.org/scaling/dependencies) and [deploy with one-click](https://docs.metaflow.org/production/introduction) to highly available production orchestrators with built in support for [reactive orchestration](https://docs.metaflow.org/production/event-triggering).
+
+For full documentation, check out our [API Reference](https://docs.metaflow.org/api) or see our [Release Notes](https://github.com/Netflix/metaflow/releases) for the latest features and improvements. 
+
+
+## Getting started
+
+Getting up and running is easy. If you don't know where to start, [Metaflow sandbox](https://outerbounds.com/sandbox) will have you running and exploring in seconds.
+
+### Installing Metaflow
+
+To install Metaflow in your Python environment from [PyPI](https://pypi.org/project/metaflow/):
+
+```sh
+pip install metaflow
+```
+Alternatively, using [conda-forge](https://anaconda.org/conda-forge/metaflow):
+
+```sh
+conda install -c conda-forge metaflow
+```
+
+Once installed, a great way to get started is by following our [tutorial](https://docs.metaflow.org/getting-started/tutorials). It walks you through creating and running your first Metaflow flow step by step.  
+
+For more details on Metaflow’s features and best practices, check out:
+- [How Metaflow works](https://docs.metaflow.org/metaflow/basics)  
+- [Additional resources](https://docs.metaflow.org/introduction/metaflow-resources)  
+
+If you need help, don’t hesitate to reach out on our [Slack community](http://slack.outerbounds.co/)!
+
+
+### Deploying infrastructure for Metaflow in your cloud
+<img src="./docs/multicloud.png" width="800px">
+
+
+While you can get started with Metaflow easily on your laptop, the main benefits of Metaflow lie in its ability to [scale out to external compute clusters](https://docs.metaflow.org/scaling/remote-tasks/introduction) 
+and to [deploy to production-grade workflow orchestrators](https://docs.metaflow.org/production/introduction). To benefit from these features, follow this [guide](https://outerbounds.com/engineering/welcome/) to 
+configure Metaflow and the infrastructure behind it appropriately.
+
+
+## Get in touch
+We'd love to hear from you. Join our community [Slack workspace](http://slack.outerbounds.co/)!
+
+## Contributing
+We welcome contributions to Metaflow. Please see our [contribution guide](https://docs.metaflow.org/introduction/contributing-to-metaflow) for more details.
+

{ob_metaflow-2.15.0.1 → ob_metaflow-2.15.5.1}/devtools/Makefile

@@ -8,6 +8,7 @@ help:
 	@echo "  ui            - Open Metaflow UI"
 	@echo "  dashboard     - Open Minikube dashboard"
 	@echo "  down          - Stop and clean up the environment"
+	@echo "  all-up        - Start the development environment with all services"
 	@echo "  help          - Show this help message"
 
 HELM_VERSION := v3.14.0
@@ -21,13 +22,14 @@ DEVTOOLS_DIR := $(MKFILE_DIR).devtools
 PICK_SERVICES := $(MKFILE_DIR)pick_services.sh
 MINIKUBE_DIR := $(DEVTOOLS_DIR)/minikube
 MINIKUBE := $(MINIKUBE_DIR)/minikube
+HELM_DIR := $(DEVTOOLS_DIR)/helm
 TILT_DIR := $(DEVTOOLS_DIR)/tilt
 TILT := $(TILT_DIR)/tilt
 TILTFILE := $(MKFILE_DIR)/Tiltfile
-MAKE_CMD := $(MAKE) -C "$(MKFILE_DIR)"
+MAKE_CMD := $(MAKE) -f "$(MKFILE_PATH)"
 
 MINIKUBE_CPUS ?= 4
-MINIKUBE_MEMORY ?= 6000
+MINIKUBE_MEMORY ?= 6144
 MINIKUBE_DISK_SIZE ?= 20g
 
 ifeq ($(shell uname), Darwin)
@@ -50,9 +52,18 @@ endif
 
 install-helm:
 	@if ! command -v helm >/dev/null 2>&1; then \
-		echo "📥 Installing Helm $(HELM_VERSION) (may require sudo access)..."; \
-		curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | HELM_INSTALL_VERSION=$(HELM_VERSION) bash; \
+		echo "📥 Installing Helm $(HELM_VERSION)..."; \
+		mkdir -p "$(HELM_DIR)"; \
+		curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \
+			| HELM_INSTALL_VERSION="$(HELM_VERSION)" \
+			  USE_SUDO="false" \
+			  PATH="$(HELM_DIR):$$PATH" \
+			  HELM_INSTALL_DIR="$(HELM_DIR)" \
+			  bash; \
+		chmod +x "$(HELM_DIR)/helm"; \
 		echo "✅ Helm installation complete"; \
+	else \
+		echo "✅ Helm is already installed at $$(command -v helm)"; \
 	fi
 
 check-docker:
@@ -98,7 +109,7 @@ install-gum:
 	@if ! command -v gum >/dev/null 2>&1; then \
 		echo "📥 Installing gum..."; \
 		if [ "$(shell uname)" = "Darwin" ]; then \
-			HOMEBREW_NO_AUTO_UPDATE=1 brew install gum; \
+			HOMEBREW_NO_AUTO_UPDATE=1 brew install gum|| { echo "❌ Failed to install gum via Homebrew"; exit 1; }; \
 		elif command -v apt-get >/dev/null 2>&1; then \
 			curl -fsSL -o /tmp/gum.deb \
 			"https://github.com/charmbracelet/gum/releases/download/$(GUM_VERSION)/gum_$(GUM_VERSION:v%=%)_$(arch).deb"; \
@@ -129,7 +140,8 @@ setup-minikube:
 			--cpus $(MINIKUBE_CPUS) \
 			--memory $(MINIKUBE_MEMORY) \
 			--disk-size $(MINIKUBE_DISK_SIZE) \
-			--driver docker; \
+			--driver docker \
+		|| { echo "❌ Failed to start Minikube (check if Docker is running)"; exit 1; }; \
 		echo "🔌 Enabling metrics-server and dashboard (quietly)..."; \
 		$(MINIKUBE) addons enable metrics-server >/dev/null 2>&1; \
 		$(MINIKUBE) addons enable dashboard >/dev/null 2>&1; \
@@ -170,7 +182,6 @@ up: install-brew check-docker install-curl install-gum setup-minikube install-he
 	@echo 'trap "exit" INT TERM' >> $(DEVTOOLS_DIR)/start.sh
 	@echo 'trap "kill 0" EXIT' >> $(DEVTOOLS_DIR)/start.sh
 	@echo 'eval $$($(MINIKUBE) docker-env)' >> $(DEVTOOLS_DIR)/start.sh
-	
 	@echo 'if [ -n "$$SERVICES_OVERRIDE" ]; then' >> "$(DEVTOOLS_DIR)/start.sh"
 	@echo '    echo "🌐 Using user-provided list of services: $$SERVICES_OVERRIDE"' >> "$(DEVTOOLS_DIR)/start.sh"
 	@echo '    SERVICES="$$SERVICES_OVERRIDE"' >> "$(DEVTOOLS_DIR)/start.sh"
@@ -181,16 +192,19 @@ up: install-brew check-docker install-curl install-gum setup-minikube install-he
 	@echo 'PATH="$(MINIKUBE_DIR):$(TILT_DIR):$$PATH" $(MINIKUBE) tunnel &' >> $(DEVTOOLS_DIR)/start.sh
 	@echo 'echo -e "🚀 Starting Tilt with selected services..."' >> $(DEVTOOLS_DIR)/start.sh
 	@echo 'echo -e "\033[1;38;5;46m\n🔥 \033[1;38;5;196mNext Steps:\033[0;38;5;46m Use \033[3mmetaflow-dev shell\033[23m to switch to the development\n   environment'\''s shell and start executing your Metaflow flows.\n\033[0m"' >> "$(DEVTOOLS_DIR)/start.sh"
-	@echo 'PATH="$(MINIKUBE_DIR):$(TILT_DIR):$$PATH" SERVICES="$$SERVICES" tilt up -f $(TILTFILE)' >> $(DEVTOOLS_DIR)/start.sh
+	@echo 'PATH="$(HELM_DIR):$(MINIKUBE_DIR):$(TILT_DIR):$$PATH" SERVICES="$$SERVICES" tilt up -f $(TILTFILE)' >> $(DEVTOOLS_DIR)/start.sh
 	@echo 'wait' >> $(DEVTOOLS_DIR)/start.sh
 	@chmod +x $(DEVTOOLS_DIR)/start.sh
 	@$(DEVTOOLS_DIR)/start.sh
 
+all-up:
+	@echo "🚀 Starting up all services..."
+	SERVICES_OVERRIDE=all $(MAKE_CMD) up
+
 down:
 	@echo "🛑 Stopping all services..."
 	@-pkill -f "$(MINIKUBE) tunnel" 2>/dev/null || true
 	@echo "⏹️  Stopping Tilt..."
-	-PATH="$(MINIKUBE_DIR):$(TILT_DIR):$$PATH" tilt down -f $(TILTFILE)
 	@echo "🧹 Cleaning up Minikube..."
 	$(MAKE_CMD) teardown-minikube
 	@echo "🗑️  Removing Tilt binary and directory..."
@@ -201,7 +215,7 @@ down:
 
 shell: setup-tilt
 	@echo "⏳ Checking if development environment is up..."
-	@set -e; \
+	@set -eu; \
 	for i in $$(seq 1 90); do \
 		if "$(TILT)" get session >/dev/null 2>&1; then \
 			found_session=1; \
@@ -210,7 +224,7 @@ shell: setup-tilt
 			sleep 2; \
 		fi; \
 	done; \
-	if [ -z "$${found_session}" ]; then \
+	if [ -z "$${found_session:-}" ]; then \
 		echo "❌ Development environment is not up."; \
 		echo "   Please run 'metaflow-dev up' in another terminal, then re-run 'metaflow-dev shell'."; \
 		exit 1; \
@@ -220,7 +234,10 @@ shell: setup-tilt
 		"$(TILT)" get uiresource generate-configs >/dev/null 2>&1; \
 		status=$$?; \
 		if [ $$status -eq 0 ]; then \
-			"$(TILT)" wait --for=condition=Ready uiresource/generate-configs; \
+			if ! "$(TILT)" wait --for=condition=Ready uiresource/generate-configs --timeout=300s; then \
+				echo "❌ Timed out waiting for development environment to be ready."; \
+				exit 1; \
+			fi; \
 			break; \
 		elif [ $$status -eq 127 ]; then \
 			echo "❌ Development environment is not up."; \
@@ -250,8 +267,7 @@ shell: setup-tilt
 				"$$user_shell" -i; \
 		fi'
 
-# TODO: This can be done away with in a while since we now have metaflow-dev command.
-#
+
 # @echo '$(MAKE_CMD) create-dev-shell' >> $(DEVTOOLS_DIR)/start.sh
 # @echo 'rm -f /tmp/metaflow-devshell-*' >> $(DEVTOOLS_DIR)/start.sh
 create-dev-shell: setup-tilt
@@ -268,7 +284,10 @@ create-dev-shell: setup-tilt
 		echo "fi" >> $$SHELL_PATH && \
 		echo "" >> $$SHELL_PATH && \
 		echo "echo \"⏳ Waiting for development environment to be ready...\"" >> $$SHELL_PATH && \
-		echo "$(TILT) wait --for=condition=Ready uiresource/generate-configs" >> $$SHELL_PATH && \
+		echo "if ! $(TILT) wait --for=condition=Ready uiresource/generate-configs --timeout=300s; then" >> $$SHELL_PATH && \
+		echo "  echo \"❌ Timed out waiting for development environment to be ready.\"" >> $$SHELL_PATH && \
+		echo "  exit 1" >> $$SHELL_PATH && \
+		echo "fi" >> $$SHELL_PATH && \
 		echo "" >> $$SHELL_PATH && \
 		echo "echo \"🔧 Starting a new shell for development environment...\"" >> $$SHELL_PATH && \
 		echo "if [ -n \"\$$SHELL\" ]; then" >> $$SHELL_PATH && \
@@ -318,6 +337,6 @@ ui: setup-tilt
 	@echo "🔗 Opening Metaflow UI at http://localhost:3000"
 	@open http://localhost:3000
 
-.PHONY: install-helm setup-minikube setup-tilt teardown-minikube tunnel up down check-docker install-curl install-gum install-brew up down dashboard shell ui help
+.PHONY: install-helm setup-minikube setup-tilt teardown-minikube tunnel up down check-docker install-curl install-gum install-brew up down dashboard shell ui all-up help
 
-.DEFAULT_GOAL := up
+.DEFAULT_GOAL := help

{ob_metaflow-2.15.0.1 → ob_metaflow-2.15.5.1}/devtools/Tiltfile

@@ -23,8 +23,13 @@ components = {
     "argo-events": ["argo-workflows"],
 }
 
-if os.getenv("SERVICES", "").strip():
-    requested_components = os.getenv("SERVICES", "").split(",")
+services_env = os.getenv("SERVICES", "").strip().lower()
+
+if services_env:
+    if services_env == "all":
+        requested_components = list(components.keys())
+    else:
+        requested_components = services_env.split(",")
 else:
     requested_components = list(components.keys())
 
@@ -78,7 +83,7 @@ for component in requested_components:
         if result not in enabled_components:
             enabled_components.append(result)
 
-# Print a friendly summary when running `tilt up`.            
+# Print a friendly summary when running `tilt up`.
 if config.tilt_subcommand == 'up':
     print("\n📦 Components to install:")
     for component in enabled_components:
@@ -99,6 +104,7 @@ if "minio" in enabled_components:
         set=[
             'rootUser=rootuser',
             'rootPassword=rootpass123',
+            # TODO: perturb the bucket name to avoid conflicts
             'buckets[0].name=metaflow-test',
             'buckets[0].policy=none',
             'buckets[0].purge=false',
@@ -498,13 +504,14 @@ if "argo-events" in enabled_components:
         'argo-events-controller-manager',
         labels=['argo-events'],
     )
-    
+
     metaflow_config["METAFLOW_ARGO_EVENTS_EVENT"] = "metaflow-event"
     metaflow_config["METAFLOW_ARGO_EVENTS_EVENT_BUS"] = "default"
     metaflow_config["METAFLOW_ARGO_EVENTS_EVENT_SOURCE"] = "argo-events-webhook"
     metaflow_config["METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT"] = "operate-workflow-sa"
     metaflow_config["METAFLOW_ARGO_EVENTS_WEBHOOK_AUTH"] = "service"
-    metaflow_config["METAFLOW_ARGO_EVENTS_WEBHOOK_URL"] = "http://argo-events-webhook-eventsource-svc:12000/metaflow-event"
+    metaflow_config["METAFLOW_ARGO_EVENTS_INTERNAL_WEBHOOK_URL"] = "http://argo-events-webhook-eventsource-svc:12000/metaflow-event"
+    metaflow_config["METAFLOW_ARGO_EVENTS_WEBHOOK_URL"] = "http://localhost:12000/metaflow-event"
 
     config_resources.append('argo-events-controller-manager')
     config_resources.append('argo-events-webhook-eventsource-svc')
@@ -522,8 +529,8 @@ if "metadata-service" in enabled_components:
             'metadatadb.password=metaflow123',
             'metadatadb.database=metaflow',
             'metadatadb.host=postgresql',
-            'image.repository=public.ecr.aws/p7g1e3j4/metaflow-service',
-            'image.tag=2.4.13-fbcc7d04',
+            'image.repository=public.ecr.aws/outerbounds/metaflow_metadata_service',
+            'image.tag=2.4.13-2-g70af4ed',
             'resources.requests.cpu=25m',
             'resources.requests.memory=64Mi',
             'resources.limits.cpu=50m',
@@ -560,8 +567,8 @@ if "ui" in enabled_components:
             'uiBackend.metadatadb.host=postgresql',
             'uiBackend.metaflowDatastoreSysRootS3=s3://metaflow-test',
             'uiBackend.metaflowS3EndpointURL=http://minio.default.svc.cluster.local:9000',
-            'uiBackend.image.name=public.ecr.aws/p7g1e3j4/metaflow-service',
-            'uiBackend.image.tag=2.4.13-fbcc7d04',
+            'uiBackend.image.name=public.ecr.aws/outerbounds/metaflow_metadata_service',
+            'uiBackend.image.tag=2.4.13-2-g70af4ed',
             'uiBackend.env[0].name=AWS_ACCESS_KEY_ID',
             'uiBackend.env[0].value=rootuser',
             'uiBackend.env[1].name=AWS_SECRET_ACCESS_KEY',
@@ -600,8 +607,20 @@ if "ui" in enabled_components:
     config_resources.append('metaflow-ui')
     config_resources.append('metaflow-ui-static')
 
+cmd = '''
+ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}')
+case "$ARCH" in
+  arm64)   echo linux-aarch64 ;;
+  amd64)   echo linux-64 ;;
+  *)       echo linux-64 ;;
+esac
+'''
+
+# For @conda/@pypi emulation
+metaflow_config["METAFLOW_KUBERNETES_CONDA_ARCH"] = str(local(cmd)).strip()
+
 local_resource(
     name="generate-configs",
     cmd=write_config_files(),
     resource_deps=config_resources,
-)
+)

{ob_metaflow-2.15.0.1 → ob_metaflow-2.15.5.1}/metaflow/__init__.py

@@ -119,6 +119,12 @@ from .includefile import IncludeFile
 # Decorators
 from .decorators import step, _import_plugin_decorators
 
+
+# Parsers (for configs) for now
+from .plugins import _import_tl_plugins
+
+_import_tl_plugins(globals())
+
 # this auto-generates decorator functions from Decorator objects
 # in the top-level metaflow namespace
 _import_plugin_decorators(globals())

ob_metaflow-2.15.5.1/metaflow/cmd/code/__init__.py

@@ -0,0 +1,230 @@
+import os
+import shutil
+import sys
+from subprocess import PIPE, CompletedProcess, run
+from tempfile import TemporaryDirectory
+from typing import Any, Callable, List, Mapping, Optional, cast
+
+from metaflow import Run
+from metaflow._vendor import click
+from metaflow.cli import echo_always
+
+
+@click.group()
+def cli():
+    pass
+
+
+@cli.group(help="Access, compare, and manage code associated with Metaflow runs.")
+def code():
+    pass
+
+
+def echo(line: str) -> None:
+    echo_always(line, err=True, fg="magenta")
+
+
+def extract_code_package(runspec: str) -> TemporaryDirectory:
+    try:
+        mf_run = Run(runspec, _namespace_check=False)
+        echo(f"✅  Run *{runspec}* found, downloading code..")
+    except Exception as e:
+        echo(f"❌  Run **{runspec}** not found")
+        raise e
+
+    if mf_run.code is None:
+        echo(
+            f"❌  Run **{runspec}** doesn't have a code package. Maybe it's a local run?"
+        )
+        raise RuntimeError("no code package found")
+
+    return mf_run.code.extract()
+
+
+def perform_diff(
+    source_dir: str,
+    target_dir: Optional[str] = None,
+    output: bool = False,
+    **kwargs: Mapping[str, Any],
+) -> Optional[List[str]]:
+    if target_dir is None:
+        target_dir = os.getcwd()
+
+    diffs = []
+    for dirpath, dirnames, filenames in os.walk(source_dir, followlinks=True):
+        for fname in filenames:
+            # NOTE: the paths below need to be set up carefully
+            # for the `patch` command to work. Better not to touch
+            # the directories below. If you must, test that patches
+            # work after your changes.
+            #
+            # target_file is the git repo in the current working directory
+            rel = os.path.relpath(dirpath, source_dir)
+            target_file = os.path.join(rel, fname)
+            # source_file is the run file loaded in a tmp directory
+            source_file = os.path.join(dirpath, fname)
+
+            if sys.stdout.isatty() and not output:
+                color = ["--color"]
+            else:
+                color = ["--no-color"]
+
+            if os.path.exists(os.path.join(target_dir, target_file)):
+                cmd = (
+                    ["git", "diff", "--no-index", "--exit-code"]
+                    + color
+                    + [
+                        target_file,
+                        source_file,
+                    ]
+                )
+                result: CompletedProcess = run(
+                    cmd, text=True, stdout=PIPE, cwd=target_dir
+                )
+                if result.returncode == 0:
+                    if not output:
+                        echo(f"✅  {target_file} is identical, skipping")
+                    continue
+
+                if output:
+                    diffs.append(result.stdout)
+                else:
+                    run(["less", "-R"], input=result.stdout, text=True)
+            else:
+                if not output:
+                    echo(f"❗  {target_file} not in the target directory, skipping")
+    return diffs if output else None
+
+
+def run_op(
+    runspec: str, op: Callable[..., Optional[List[str]]], **op_args: Mapping[str, Any]
+) -> Optional[List[str]]:
+    tmp = None
+    try:
+        tmp = extract_code_package(runspec)
+        return op(tmp.name, **op_args)
+    finally:
+        if tmp and os.path.exists(tmp.name):
+            shutil.rmtree(tmp.name)
+
+
+def run_op_diff_runs(
+    source_run_pathspec: str, target_run_pathspec: str, **op_args: Mapping[str, Any]
+) -> Optional[List[str]]:
+    source_tmp = None
+    target_tmp = None
+    try:
+        source_tmp = extract_code_package(source_run_pathspec)
+        target_tmp = extract_code_package(target_run_pathspec)
+        return perform_diff(source_tmp.name, target_tmp.name, **op_args)
+    finally:
+        for d in [source_tmp, target_tmp]:
+            if d and os.path.exists(d.name):
+                shutil.rmtree(d.name)
+
+
+def op_diff(tmpdir: str, **kwargs: Mapping[str, Any]) -> Optional[List[str]]:
+    kwargs_dict = dict(kwargs)
+    target_dir = cast(Optional[str], kwargs_dict.pop("target_dir", None))
+    output: bool = bool(kwargs_dict.pop("output", False))
+    op_args: Mapping[str, Any] = {**kwargs_dict}
+    return perform_diff(tmpdir, target_dir=target_dir, output=output, **op_args)
+
+
+def op_pull(tmpdir: str, dst: str, **op_args: Mapping[str, Any]) -> None:
+    if os.path.exists(dst):
+        echo(f"❌  Directory *{dst}* already exists")
+    else:
+        shutil.move(tmpdir, dst)
+        echo(f"Code downloaded to *{dst}*")
+
+
+def op_patch(tmpdir: str, dst: str, **kwargs: Mapping[str, Any]) -> None:
+    diffs = perform_diff(tmpdir, output=True) or []
+    with open(dst, "w", encoding="utf-8") as f:
+        for out in diffs:
+            out = out.replace(tmpdir, "/.")
+            out = out.replace("+++ b/./", "+++ b/")
+            out = out.replace("--- b/./", "--- b/")
+            out = out.replace("--- a/./", "--- a/")
+            out = out.replace("+++ a/./", "+++ a/")
+            f.write(out)
+    echo(f"Patch saved in *{dst}*")
+    path = run(
+        ["git", "rev-parse", "--show-prefix"], text=True, stdout=PIPE
+    ).stdout.strip()
+    if path:
+        diropt = f" --directory={path.rstrip('/')}"
+    else:
+        diropt = ""
+    echo("Apply the patch by running:")
+    echo_always(
+        f"git apply --verbose{diropt} {dst}", highlight=True, bold=True, err=True
+    )
+
+
+@code.command()
+@click.argument("run_pathspec")
+def diff(run_pathspec: str, **kwargs: Mapping[str, Any]) -> None:
+    """
+    Do a 'git diff' of the current directory and a Metaflow run.
+    """
+    _ = run_op(run_pathspec, op_diff, **kwargs)
+
+
+@code.command()
+@click.argument("source_run_pathspec")
+@click.argument("target_run_pathspec")
+def diff_runs(
+    source_run_pathspec: str, target_run_pathspec: str, **kwargs: Mapping[str, Any]
+) -> None:
+    """
+    Do a 'git diff' between two Metaflow runs.
+    """
+    _ = run_op_diff_runs(source_run_pathspec, target_run_pathspec, **kwargs)
+
+
+@code.command()
+@click.argument("run_pathspec")
+@click.option(
+    "--dir", help="Destination directory (default: {run_pathspec}_code)", default=None
+)
+def pull(
+    run_pathspec: str, dir: Optional[str] = None, **kwargs: Mapping[str, Any]
+) -> None:
+    """
+    Pull the code of a Metaflow run.
+    """
+    if dir is None:
+        dir = run_pathspec.lower().replace("/", "_") + "_code"
+    op_args: Mapping[str, Any] = {**kwargs, "dst": dir}
+    run_op(run_pathspec, op_pull, **op_args)
+
+
+@code.command()
+@click.argument("run_pathspec")
+@click.option(
+    "--file_path",
+    help="Patch file name. If not provided, defaults to a sanitized version of RUN_PATHSPEC "
+    "with slashes replaced by underscores, plus '.patch'.",
+    show_default=False,
+)
+@click.option(
+    "--overwrite", is_flag=True, help="Overwrite the patch file if it exists."
+)
+def patch(
+    run_pathspec: str,
+    file_path: Optional[str] = None,
+    overwrite: bool = False,
+    **kwargs: Mapping[str, Any],
+) -> None:
+    """
+    Create a patch by comparing current dir with a Metaflow run.
+    """
+    if file_path is None:
+        file_path = run_pathspec.lower().replace("/", "_") + ".patch"
+    if os.path.exists(file_path) and not overwrite:
+        echo(f"File *{file_path}* already exists. To overwrite, specify --overwrite.")
+        return
+    op_args: Mapping[str, Any] = {**kwargs, "dst": file_path}
+    run_op(run_pathspec, op_patch, **op_args)

{ob_metaflow-2.15.0.1 → ob_metaflow-2.15.5.1}/metaflow/cmd/develop/stub_generator.py

@@ -1133,13 +1133,16 @@ class StubGenerator:
             result = result[1:]
         # Add doc to first and last overloads. Jedi uses the last one and pycharm
         # the first one. Go figure.
+        result_docstring = docs["func_doc"]
+        if docs["param_doc"]:
+            result_docstring += "\nParameters\n----------\n" + docs["param_doc"]
         result[0] = (
             result[0][0],
-            docs["func_doc"] + "\nParameters\n----------\n" + docs["param_doc"],
+            result_docstring,
         )
         result[-1] = (
             result[-1][0],
-            docs["func_doc"] + "\nParameters\n----------\n" + docs["param_doc"],
+            result_docstring,
         )
         return result