odctl 1.2.0__tar.gz → 1.2.1__tar.gz

{odctl-1.2.0 → odctl-1.2.1}/.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,7 +14,7 @@ A clear and concise description of what the bug is.
 
 Steps to reproduce the behavior:
 
-1. What profiles were you trying to launch? (e.g., `odctl up airflow spark-lite ch-lite`)
+1. What profiles were you trying to launch? (for example, `odctl up airflow spark-lite ch-lite`)
 2. Run this CLI command: '...'
 3. See error
 
@@ -34,13 +34,13 @@ If applicable, paste the full CLI error or the specific container logs here (you
 
 **Environment (please complete the following information):**
 
-* OS: [e.g. Ubuntu 24.04, macOS Sonoma, Windows 11 / WSL2]
-* Python Version: [e.g. 3.12]
-* odctl version (`odctl info`): [e.g. 1.0.0]
-* Docker Version: [e.g. Docker Desktop 4.28.0, Docker Engine 25.0]
+* OS: [for example Ubuntu 24.04, macOS Sonoma, Windows 11 / WSL2]
+* Python Version: [for example 3.12]
+* odctl version (`odctl info`): [for example 1.0.0]
+* Docker Version: [for example Docker Desktop 4.28.0, Docker Engine 25.0]
 
 ## **Additional context**
 
-* Did you customize any files in your local `./.odctl/` workspace? (e.g., changing ports in `compose-orch.yml` or editing `registry.yml`)
+* Did you customize any files in your local `./.odctl/` workspace? (for example, changing ports in `compose-orch.yml` or editing `registry.yml`)
 * How much RAM/CPU is allocated to your Docker daemon? (Many Open Data profiles are resource-heavy)
 * Add any other context about the problem here.

{odctl-1.2.0 → odctl-1.2.1}/.github/ISSUE_TEMPLATE/feature_request.md

@@ -12,7 +12,7 @@ A clear and concise description of what the problem is. Ex. I'm always frustrate
 
 **Describe the solution you'd like**
 
-A clear and concise description of what you want to happen. (e.g., "I want a new profile added to `compose-orch.yml` for Dagster", or "Add a dedicated connector script for MongoDB CDC.")
+A clear and concise description of what you want to happen. (for example, "I want a new profile added to `compose-orch.yml` for Dagster", or "Add a dedicated connector script for MongoDB CDC.")
 
 **Describe alternatives you've considered**
 

{odctl-1.2.0 → odctl-1.2.1}/.github/scripts/select-profiles.py

@@ -70,12 +70,12 @@ REGISTRY = Path("src/odctl/resources/registry.yml")
 
 # smoke.sh starts some profiles from another group's case: the kafka case starts
 # catalog, the flink case starts kafka-lite and feast, the spark case starts
-# kafka-lite, and the fluss case starts flink-lite. A change to that compose file
-# needs these groups as well as the ones the registry maps to it.
+# kafka-lite, and the fluss and feast cases start flink-lite. A change to that
+# compose file needs these groups as well as the ones the registry maps to it.
 ALSO_RUNS = {
     "compose-infra.yml": ["kafka-lite kafka-full"],
     "compose-kafka.yml": ["flink-lite flink-full", "spark-lite spark-full"],
-    "compose-flink.yml": ["fluss"],
+    "compose-flink.yml": ["fluss", "feast"],
     "compose-mlops.yml": ["flink-lite flink-full"],
 }
 
@@ -84,21 +84,23 @@ ALSO_RUNS = {
 # map runs everything, and a test fails until it is added.
 SNIPPET_PREFIX = "docs/snippets/"
 SNIPPET_GROUPS = {
+    "getting-started": ["kafka-lite kafka-full"],
+    "how-it-works": ["storage", "catalog", "trino", "airflow"],
     "kafka": ["kafka-lite kafka-full"],
     "kafka-connect": ["kafka-lite kafka-full"],
     "iceberg-ingestion": ["kafka-lite kafka-full", "flink-lite flink-full"],
-    "flink-sql": ["flink-lite flink-full", "mlflow"],
+    "flink-sql": ["flink-lite flink-full"],
+    "rolling-windows": ["flink-lite flink-full"],
     "fluss": ["fluss"],
     "spark": ["spark-lite spark-full"],
     "clickhouse": ["ch-lite ch-full"],
     "trino-metabase": ["trino", "metabase"],
     "hybrid-search": ["postgres"],
-    "storage": ["storage"],
     "valkey": ["valkey"],
     "airflow-dags": ["airflow"],
     "temporal-workers": ["temporal"],
     "mlflow": ["mlflow"],
-    "feast-iceberg": ["feast"],
+    "feast-iceberg": ["feast", "flink-lite flink-full", "mlflow"],
     "evidently-reports": ["evidently"],
     "lineage": ["lineage"],
     "openmetadata": ["metadata"],
@@ -106,8 +108,16 @@ SNIPPET_GROUPS = {
 }
 
 
-def profile_to_file(registry_path: Path = REGISTRY) -> dict:
-    """Map each profile to the compose file the registry says provides it."""
+def _profile_to_file(registry_path: Path = REGISTRY) -> dict:
+    """
+    Map each profile to the compose file the registry says provides it.
+
+    Args:
+        registry_path (Path): The registry.yml to read.
+
+    Returns:
+        dict: The compose file name of each profile.
+    """
     data = yaml.safe_load(registry_path.read_text())
     mapping = {}
     for stack in data["stacks"].values():
@@ -117,12 +127,30 @@ def profile_to_file(registry_path: Path = REGISTRY) -> dict:
 
 
 def select(changed: list, registry_path: Path = REGISTRY) -> list:
-    """Return the profile groups that the changed files require."""
+    """
+    Return the profile groups that the changed files require.
+
+    Args:
+        changed (list): The paths of the changed files, relative to the repository.
+        registry_path (Path): The registry.yml that maps profiles to compose files.
+
+    Returns:
+        list: The groups to run, every group when a change affects them all, or none.
+    """
     changed = [c for c in changed if c.strip()]
     if not changed:
         return []
 
     def is_compose(path: str) -> bool:
+        """
+        Tell whether a path is one of odctl's compose files.
+
+        Args:
+            path (str): A changed file's path, relative to the repository.
+
+        Returns:
+            bool: True when the path is under the compose files' folder.
+        """
         return path.startswith(COMPOSE_PREFIX)
 
     broad = [
@@ -141,7 +169,7 @@ def select(changed: list, registry_path: Path = REGISTRY) -> list:
                 return list(GROUPS)
             snippet_groups.update(SNIPPET_GROUPS[guide])
 
-    mapping = profile_to_file(registry_path)
+    mapping = _profile_to_file(registry_path)
     touched_files = {Path(c).name for c in changed if is_compose(c)}
 
     selected = []
@@ -154,6 +182,7 @@ def select(changed: list, registry_path: Path = REGISTRY) -> list:
 
 
 def main() -> None:
+    """Print the groups for the files named on the command line, or every group with `--all`, as JSON."""
     args = sys.argv[1:]
     if "--all" in args:
         print(json.dumps(list(GROUPS)))

{odctl-1.2.0 → odctl-1.2.1}/.github/scripts/smoke.sh

@@ -72,6 +72,65 @@ run_snippet() {
   bash -o pipefail -c "$code"
 }
 
+# Run a section and check its output: snippet_says kafka/commands.sh list -F 'orders'.
+# The whole output is read before grep looks at it. A grep -q at the end of a pipe
+# stops at the first match, the command then fails writing its next line, and
+# pipefail reports the check as failed.
+snippet_says() {
+  local file="$1" section="$2" out
+  shift 2
+  out=$(run_snippet "$file" "$section" 2>/dev/null) || return 1
+  grep -q "$@" <<<"$out"
+}
+
+# Run a guide's Runnable Example from the current directory, where the .odctl
+# workspace is, and check that its output matches an extended regular
+# expression: run_example kafka '"n":5'. The optional third argument is the
+# command that runs the file, such as a Python with the client installed.
+run_example() {
+  local guide="$1" want="$2" runner="${3:-bash}" file out
+  file=$(ls "$SNIPPETS/$guide"/example.* 2>/dev/null | head -1)
+  [ -n "$file" ] || fail "no runnable example in docs/snippets/$guide"
+  out=$($runner "$file" 2>&1) || { echo "$out" | tail -20; fail "the $guide example failed"; }
+  grep -qE -- "$want" <<<"$out" || { echo "$out" | tail -20; fail "the $guide example printed no '$want'"; }
+  pass "the $guide guide's runnable example printed '$want'"
+}
+
+# The commands that run the MLflow and Evidently examples: a Python 3.12 with
+# the client versions the images pin, as the guides tell the reader to install.
+mlflow_runner() {
+  local dockerfile="$SNIPPETS/../../src/odctl/resources/docker/mlflow/Dockerfile" mlflow xgboost
+  mlflow=$(grep -oE 'mlflow/mlflow:v[0-9.]+' "$dockerfile" | head -1 | cut -dv -f2)
+  xgboost=$(grep -oE "xgboost~=[0-9.]+" "$dockerfile" | head -1 | cut -d= -f2)
+  echo "uv run --no-project --quiet --python 3.12 --with mlflow==$mlflow --with xgboost~=$xgboost"
+}
+evidently_runner() {
+  local version
+  version=$(grep -oE 'evidently/evidently-service:[0-9.]+' \
+    "$SNIPPETS/../../src/odctl/resources/compose-mlops.yml" | head -1 | cut -d: -f2)
+  echo "uv run --no-project --quiet --python 3.12 --with evidently==$version --with pandas<3.1 python"
+}
+
+# How It Works edits a workspace file and applies it with odctl recreate. Each
+# check runs the guide's section, puts the file back and recreates again, so
+# later profiles in the same job see the shipped settings, and matches the
+# output against an extended regular expression.
+how_it_works_edit() {
+  local section="$1" file="$2" profile="$3" want="$4" out
+  out=$(run_snippet how-it-works/commands.sh "$section" 2>&1) || { echo "$out" | tail -20; fail "the $section example failed"; }
+  mv "$file.bak" "$file" || fail "the $section example left no $file.bak"
+  odctl recreate "$profile" >/dev/null 2>&1 || fail "could not recreate $profile after the $section example"
+  grep -qE -- "$want" <<<"$out" || { echo "$out" | tail -20; fail "the $section example printed no '$want'"; }
+  pass "How It Works' $section example printed '$want'"
+}
+how_it_works_trino() {
+  how_it_works_edit trino-port .odctl/compose-analytics.yml trino '8080/tcp -> 127\.0\.0\.1:8180'
+  retry 30 5 http_ok "http://127.0.0.1:8080/v1/info" || fail "Trino did not answer on 8080 after the port was put back"
+}
+how_it_works_airflow() {
+  how_it_works_edit airflow-deps .odctl/.env airflow '^1\.4\.3$'
+}
+
 # Metrics: nothing else here reads a metrics endpoint, so a change that breaks
 # one goes unseen. In 0.8.0 the JMX agent in KAFKA_OPTS broke every Kafka CLI
 # run, and only an unrelated topic check caught it. So each profile reads its
@@ -176,6 +235,50 @@ connect_plugins() {
   pass "Connect lists the Iceberg, Debezium and ClickHouse plugins"
 }
 
+# The Kafka Connect guide registers the MSK data generator, which needs only
+# Kafka, reads three of its records, then checks and removes the connector.
+connect_datagen() {
+  (cd "$SNIPPETS/kafka-connect" && run_snippet kafka-connect/commands.sh register) >/dev/null 2>&1 \
+    || fail "Connect rejected the data generator configuration"
+  local got status
+  got=$(run_snippet kafka-connect/commands.sh read 2>/dev/null | grep -c '"name"')
+  [ "$got" -eq 3 ] || fail "read $got of 3 data generator records from customers"
+  retry 12 5 sh -c "curl -fsS http://127.0.0.1:8083/connectors/demo-datagen/status | grep -q '\"state\":\"RUNNING\"'" \
+    || fail "the data generator was not RUNNING"
+  status=$(run_snippet kafka-connect/commands.sh status-remove 2>/dev/null)
+  grep -q '"state":"RUNNING"' <<<"$status" || { echo "$status"; fail "the data generator status was not RUNNING"; }
+  curl -fsS "http://127.0.0.1:8083/connectors/demo-datagen" >/dev/null 2>&1 \
+    && fail "the data generator is still registered after the delete"
+  pass "the data generator wrote records to customers, and was checked and removed"
+}
+
+# Getting Started runs each read-only command on kafka-lite, then restarts it.
+getting_started() {
+  local out
+  out=$(run_snippet getting-started/commands.sh list 2>&1) || fail "odctl list failed"
+  grep -q "kafka-lite" <<<"$out" || fail "odctl list does not name kafka-lite"
+  out=$(run_snippet getting-started/commands.sh list-details 2>&1) || fail "odctl list -d failed"
+  grep -q "kafka-ui" <<<"$out" || fail "odctl list -d does not name the kafka-ui service"
+  out=$(run_snippet getting-started/commands.sh explain 2>&1) || fail "odctl explain failed"
+  grep -q "karapace" <<<"$out" || fail "odctl explain kafka-lite does not name karapace"
+  out=$(run_snippet getting-started/commands.sh info 2>&1) || fail "odctl info failed"
+  grep -q "Reachable" <<<"$out" || fail "odctl info does not say Docker is reachable"
+  out=$(run_snippet getting-started/commands.sh ps 2>&1) || fail "odctl ps failed"
+  grep -q "connect" <<<"$out" || fail "odctl ps kafka-lite does not list connect"
+  out=$(run_snippet getting-started/commands.sh ps-all 2>&1) || fail "odctl ps --all failed"
+  grep -q "karapace" <<<"$out" || fail "odctl ps --all does not list karapace"
+  out=$(run_snippet getting-started/commands.sh logs 2>&1) || fail "odctl logs failed"
+  grep -q "^connect" <<<"$out" || fail "odctl logs printed no connect lines"
+  out=$(run_snippet getting-started/commands.sh logs-time 2>&1) || fail "odctl logs with --since and --until failed"
+  grep -qE "^connect +\| [0-9]{4}-" <<<"$out" || fail "odctl logs -t printed no timestamped connect lines: $out"
+  out=$(run_snippet getting-started/commands.sh down-dry-run 2>&1) || fail "odctl down --dry-run failed"
+  docker inspect connect >/dev/null 2>&1 || fail "odctl down --dry-run removed a container"
+  out=$(run_snippet getting-started/commands.sh restart 2>&1) || fail "odctl restart failed"
+  grep -q "Restart complete" <<<"$out" || fail "odctl restart did not complete"
+  retry 40 5 http_ok "http://127.0.0.1:8083/connectors" || fail "Connect never answered after the restart"
+  pass "Getting Started's list, explain, info, ps, logs, restart and down --dry-run ran on kafka-lite"
+}
+
 # The Iceberg sink is built from source, so prove it commits a snapshot rather
 # than only loading. It needs the catalog, which kafka-lite does not start.
 smoke_iceberg_sink() {
@@ -186,7 +289,7 @@ smoke_iceberg_sink() {
   run_snippet iceberg-ingestion/commands.sh topics-produce >/dev/null 2>&1 \
     || fail "could not create the topics or produce to demo-iceberg"
   # From the snippet folder, where the guide's @demo-iceberg.json is.
-  (cd "$SNIPPETS/kafka-connect" && run_snippet kafka-connect/commands.sh register) >/dev/null 2>&1 \
+  (cd "$SNIPPETS/iceberg-ingestion" && run_snippet iceberg-ingestion/commands.sh register) >/dev/null 2>&1 \
     || fail "Connect rejected the Iceberg sink configuration"
 
   # The first commit waits for the control consumer to join its group, which
@@ -208,7 +311,9 @@ print(snap[0]["summary"].get("total-records", "") if snap else "")' 2>/dev/null)
   fi
   run_snippet kafka-connect/commands.sh logs >/dev/null 2>&1 || fail "could not read the Connect log"
   local status
-  status=$(run_snippet kafka-connect/commands.sh status-remove 2>/dev/null)
+  snippet_says iceberg-ingestion/commands.sh records -F '"total-records":"5"' \
+    || fail "the guide's records step does not show 5 records"
+  status=$(run_snippet iceberg-ingestion/commands.sh status-remove 2>/dev/null)
   grep -q '"state":"RUNNING"' <<<"$status" || { echo "$status"; fail "the connector status was not RUNNING"; }
   curl -fsS "http://127.0.0.1:8083/connectors/demo-iceberg" >/dev/null 2>&1 \
     && fail "the connector is still registered after the delete"
@@ -270,11 +375,11 @@ smoke_flink_kafka_jdbc() {
   # Two different OVER windows in one streaming query are refused, so the guide
   # runs one statement for each window size, each writing its own table.
   local refused
-  refused=$(snippet flink-sql/steps.sql windows-refused | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
+  refused=$(snippet rolling-windows/steps.sql windows-refused | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
   grep -qF '[ERROR]' <<<"$refused" || fail "Flink did not refuse two different OVER windows: $(tail -5 <<<"$refused")"
-  run_snippet flink-sql/commands.sh create-window-tables >/dev/null 2>&1 || fail "could not create the window tables"
-  out=$(snippet flink-sql/steps.sql windows-one-per-size | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
-  run_snippet flink-sql/commands.sh check-windows >/dev/null 2>&1 || fail "the guide's window check query failed"
+  run_snippet rolling-windows/commands.sh create-window-tables >/dev/null 2>&1 || fail "could not create the window tables"
+  out=$(snippet rolling-windows/steps.sql windows-one-per-size | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
+  run_snippet rolling-windows/commands.sh check-windows >/dev/null 2>&1 || fail "the guide's window check query failed"
   rows=$(docker exec postgres psql -U user -d odctl -tAc \
     "SELECT (SELECT count(*) FROM card_sum_10m) || ',' || (SELECT count(*) FROM card_sum_1h)" 2>/dev/null)
   [ "$rows" = "20,20" ] || { echo "$out" | grep -iE "error|exception" | tail -10; fail "one statement for each OVER window gave '$rows' rows, expected 20,20"; }
@@ -288,11 +393,11 @@ smoke_flink_kafka_jdbc() {
 smoke_flink_http() {
   odctl up feast >/dev/null 2>&1 || fail "could not start feast for the Flink HTTP check"
   retry 40 5 http_ok "http://127.0.0.1:6566/health" || fail "Feast never answered on :6566"
-  (cd "$SNIPPETS/flink-sql" && run_snippet flink-sql/commands.sh feast-apply) >/dev/null 2>&1 \
+  (cd "$SNIPPETS/feast-iceberg" && run_snippet feast-iceberg/commands.sh push-apply) >/dev/null 2>&1 \
     || fail "could not apply the Feast push source"
   local out
-  out=$(snippet flink-sql/steps.sql feast-push | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
-  feast_value() { run_snippet flink-sql/commands.sh feast-read 2>/dev/null | grep -F 0.8999999761581421; }
+  out=$(snippet feast-iceberg/steps.sql feast-push | docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh 2>&1)
+  feast_value() { run_snippet feast-iceberg/commands.sh push-read 2>/dev/null | grep -F 0.8999999761581421; }
   retry 24 5 feast_value \
     || { echo "$out" | grep -iE "error|exception" | tail -10; fail "Feast never returned the value the Flink HTTP sink pushed"; }
   odctl down feast >/dev/null 2>&1 || fail "could not stop feast after the Flink HTTP check"
@@ -421,7 +526,8 @@ smoke_infra() {
       [ "$metres" = "714" ] || fail "PostGIS distance returned '$metres' km, expected 714"
       pass "PostGIS: geography distance works"
       # Read by the telemetry collector's postgresql receiver.
-      assert_pushed_metrics 'postgresql_.+' PostgreSQL ;;
+      assert_pushed_metrics 'postgresql_.+' PostgreSQL
+      run_example hybrid-search '^ +2$' ;;
     storage)
       retry 30 5 http_reachable "http://127.0.0.1:8333" || fail "S3 API never answered"
       # A port that answers is not a bucket that stores anything. Spark, Flink,
@@ -457,10 +563,11 @@ smoke_infra() {
       pass "three uploads of 268 MB passed and seaweed was not killed"
       # How it works shows how to create a bucket before its profile exists.
       local buckets
-      buckets=$(run_snippet storage/commands.sh create-bucket 2>&1) || fail "the create-bucket example failed: $buckets"
+      buckets=$(run_snippet how-it-works/commands.sh create-bucket 2>&1) || fail "the create-bucket example failed: $buckets"
       grep -q "'desflow'" <<<"$buckets" || fail "the bucket list has no desflow: $buckets"
       docker exec seaweed sh -c "curl -fsS $sig '$cred' -X DELETE 'http://localhost:8333/desflow'" >/dev/null 2>&1 || true
       pass "the create-bucket example created a bucket with the S3 login"
+      how_it_works_edit seaweed-memory .odctl/compose-infra.yml storage '^2147483648$'
       assert_metrics http://seaweed:9327/metrics ;;
     catalog)
       retry 30 5 http_ok "http://127.0.0.1:8181/v1/config" || fail "REST catalog never answered"
@@ -481,6 +588,22 @@ smoke_infra() {
         | grep -q 'metadata-location' || fail "table metadata did not read back"
       curl -fsS -X DELETE "http://127.0.0.1:8181/v1/namespaces/smoke/tables/t" >/dev/null 2>&1 || true
       pass "table created through the catalog and its metadata read back"
+      # How It Works reaches the catalog from a container on the odctl network,
+      # previews a start and pulls the catalog's images.
+      snippet_says how-it-works/commands.sh reach '"defaults"' \
+        || fail "a container on the odctl network did not reach catalog:8181"
+      snippet_says how-it-works/commands.sh dry-run 'compose-flink.yml' \
+        || fail "odctl up flink-lite --dry-run did not list compose-flink.yml"
+      snippet_says how-it-works/commands.sh pull 'All images pulled successfully' \
+        || fail "odctl pull catalog did not finish"
+      run_snippet how-it-works/commands.sh up-pull >/dev/null 2>&1 || fail "odctl up catalog --pull failed"
+      snippet_says how-it-works/commands.sh tag-override 'odctl/postgres:1.1.0' \
+        || fail "a TAG in the shell did not change the image odctl explain shows"
+      snippet_says how-it-works/commands.sh workspace-flag 'catalog' \
+        || fail "odctl --workspace ps catalog listed no catalog container"
+      run_snippet how-it-works/commands.sh recreate-pull >/dev/null 2>&1 || fail "odctl recreate catalog --pull failed"
+      retry 30 5 http_ok "http://127.0.0.1:8181/v1/config" || fail "REST catalog never answered after the recreate"
+      pass "How It Works' reach, dry-run, pull, up --pull, TAG, --workspace and recreate --pull examples ran"
       assert_metrics http://catalog:9404/metrics ;;
     valkey)
       retry 30 5 docker exec valkey valkey-cli --no-auth-warning -u "redis://user:password@localhost:6379" ping \
@@ -492,7 +615,8 @@ smoke_infra() {
       [ "$got" -eq 2 ] || fail "batch MGET returned $got of 2 values, expected 2"
       pass "authenticated, multi-key write and batch MGET round-tripped"
       # Read by the telemetry collector's redis receiver.
-      assert_pushed_metrics 'redis_.+' Valkey ;;
+      assert_pushed_metrics 'redis_.+' Valkey
+      run_example valkey '^\[5\]$' ;;
   esac
 }
 
@@ -544,7 +668,7 @@ for db in (d.get("data") if isinstance(d, dict) else d) or []:
 smoke_metabase() {
   local sid
   retry 60 5 http_ok "http://127.0.0.1:3000/api/health" || fail "no HTTP response from :3000"
-  run_snippet trino-metabase/commands.sh metabase-health 2>/dev/null | grep -q '"status":"ok"' \
+  snippet_says trino-metabase/commands.sh metabase-health '"status":"ok"' \
     || fail "Metabase health did not report ok"
   assert_metrics http://metabase:9191/metrics
   sid=$(retry_out 12 5 mb_session) || fail "could not obtain a Metabase session"
@@ -676,17 +800,17 @@ smoke_model_server() {
 smoke_model_features() {
   odctl up feast >/dev/null 2>&1 || fail "could not start feast for the features model"
   retry 40 5 http_ok "http://127.0.0.1:6566/health" || fail "Feast never answered on :6566"
-  (cd "$SNIPPETS/flink-sql" && run_snippet flink-sql/commands.sh feast-apply) >/dev/null 2>&1 \
+  (cd "$SNIPPETS/feast-iceberg" && run_snippet feast-iceberg/commands.sh push-apply) >/dev/null 2>&1 \
     || fail "could not apply the Feast push source"
   local now
   now=$(date -u +%Y-%m-%dT%H:%M:%S)
   retry 12 5 curl -fsS -X POST http://127.0.0.1:6566/push -H 'Content-Type: application/json' \
-    -d "{\"push_source_name\": \"driver_stats_push\", \"allow_registry_cache\": false, \"df\": {\"driver_id\": [1001], \"conv_rate\": [0.9], \"event_timestamp\": [\"$now\"]}}" \
+    -d "{\"push_source_name\": \"driver_push\", \"allow_registry_cache\": false, \"df\": {\"driver_id\": [1001], \"conv_rate\": [0.9], \"event_timestamp\": [\"$now\"]}}" \
     >/dev/null 2>&1 || fail "could not push a feature value to Feast"
   mlflow_snippet features-model >/dev/null 2>&1 || fail "could not register the features model"
   sed -i'' -e '/^MODEL_URI=/d' .odctl/.env
   cat "$SNIPPETS/mlflow/model-uri-features.env" >> .odctl/.env
-  odctl up mlflow >/dev/null 2>&1 || fail "could not start mlflow-serve with the features model"
+  odctl recreate mlflow >/dev/null 2>&1 || fail "could not recreate mlflow-serve with the features model"
   retry 30 5 http_ok "http://127.0.0.1:5003/ping" || fail "no HTTP response from :5003 for the features model"
   local got
   # feast-serve caches its registry for up to a minute after apply, and until it
@@ -1109,6 +1233,11 @@ smoke_feast() {
   # catalogs, and Iceberg's REST fixture server serves the spec with no prefix,
   # so a real warehouse gives HTTP 400 "Ambiguous URI empty segment".
   cp "$SNIPPETS/feast-iceberg/feature_repo/"* "$work/feature_repo/"
+  # Its own project, because the guide's Runnable Example registers the same
+  # feature view in demo afterwards. feast teardown leaves the view's row in
+  # feature_view_version_history, so a second apply in demo fails on its key.
+  sed 's/^project: demo$/project: smoke/' "$work/feature_repo/feature_store.yaml" \
+    > "$work/feature_store.yaml" && mv "$work/feature_store.yaml" "$work/feature_repo/feature_store.yaml"
 
   cat > "$work/check.py" <<'PYCHECK'
 import datetime, sys, pandas as pd
@@ -1174,7 +1303,8 @@ PYONLINE
   curl -fsS --max-time 10 -X DELETE \
     "http://127.0.0.1:8181/v1/namespaces/demo/tables/driver_stats?purgeRequested=true" >/dev/null 2>&1 || true
   curl -fsS --max-time 10 -X DELETE "http://127.0.0.1:8181/v1/namespaces/demo" >/dev/null 2>&1 || true
-  rm -rf "$work"
+  # The venv stays: the guide's Runnable Example runs with it next, and the feast
+  # case removes $work after that.
 }
 
 # Temporal: a healthy server proves only that it answers. Workflows run in the
@@ -1193,29 +1323,14 @@ smoke_temporal() {
   VIRTUAL_ENV="$work/.venv" uv pip install -q "temporalio==1.34.0" >/dev/null 2>&1 \
     || fail "could not install the temporalio SDK"
 
-  # Worker script from the guide as written, then the same script with its Approval
-  # workflow added and the body of main replaced by its approval step, as the
-  # guide tells the reader to do.
+  # The worker step as the guide writes it, then the Runnable Example, which
+  # adds the Approval workflow and its signal to the same script.
   snippet temporal-workers/steps.py worker > "$work/greet.py"
-  snippet temporal-workers/steps.py approval-workflow > "$work/approval.part"
-  snippet temporal-workers/steps.py approval-run > "$work/run.part"
-  python3 - "$work" <<'PYEOF' || fail "could not combine the guide's approval steps"
-import pathlib, sys, textwrap
-work = pathlib.Path(sys.argv[1])
-script = (work / "greet.py").read_text()
-head, tail = script.split("async def main() -> None:\n")
-body, rest = tail.split("\n\n", 1)
-connect = body.splitlines()[0]
-run = textwrap.indent((work / "run.part").read_text().rstrip(), "    ")
-approval = (work / "approval.part").read_text()
-(work / "approval.py").write_text(
-    f"{head}{approval}\n\nasync def main() -> None:\n{connect}\n{run}\n\n{rest}")
-PYEOF
   local got
   got=$(cd "$work" && "$work/.venv/bin/python" greet.py 2>&1 | tail -1)
   [ "$got" = "hello odctl" ] || fail "the guide's worker printed '$got', expected hello odctl"
-  got=$(cd "$work" && "$work/.venv/bin/python" approval.py 2>&1 | tail -1)
-  [ "$got" = "approved by alice" ] || fail "the guide's approval step printed '$got', expected approved by alice"
+  got=$(cd "$work" && "$work/.venv/bin/python" "$SNIPPETS/temporal-workers/example.py" 2>&1 | tail -1)
+  [ "$got" = "approved by alice" ] || fail "the guide's example printed '$got', expected approved by alice"
   pass "the guide's worker retried its activity, and its approval workflow completed on a signal"
 
   cat > "$work/workflows.py" <<'PYEOF'
@@ -1390,9 +1505,11 @@ SPARK_MASTER=http://spark-master:8080/metrics/master/prometheus/
 
 case "$PROFILE" in
   kafka-lite)
-    guide_kafka; connect_plugins
+    guide_kafka; connect_plugins; connect_datagen; getting_started
     assert_metrics http://kafka:9404/metrics "${KAFKA_SIDE[@]}"
-    smoke_iceberg_sink ;;
+    smoke_iceberg_sink
+    run_example kafka 'Processed a total of 5 messages'
+    run_example kafka-connect 'Processed a total of 3 messages' ;;
   kafka-full)
     smoke_kafka kafka-1; connect_plugins
     assert_metrics http://kafka-{1,2,3}:9404/metrics "${KAFKA_SIDE[@]}" ;;
@@ -1400,31 +1517,42 @@ case "$PROFILE" in
     smoke_flink
     smoke_flink_kafka_jdbc
     smoke_flink_http
-    assert_metrics http://jobmanager:9249/metrics http://taskmanager-a:9249/metrics ;;
+    assert_metrics http://jobmanager:9249/metrics http://taskmanager-a:9249/metrics
+    run_example flink-sql 'b +\| +1 +\| +7\.5'
+    run_example rolling-windows '20 +\| +20'
+    run_example iceberg-ingestion 'records written by the Kafka Connect sink' ;;
   flink-full)
     smoke_flink
     assert_metrics http://jobmanager:9249/metrics http://taskmanager-{a,b,c}:9249/metrics ;;
   spark-lite)
     smoke_spark
-    assert_metrics "$SPARK_MASTER" http://spark-worker-1:8081/metrics/prometheus/ ;;
+    assert_metrics "$SPARK_MASTER" http://spark-worker-1:8081/metrics/prometheus/
+    run_example spark '"n":3' ;;
   spark-full)
     smoke_spark
     assert_metrics "$SPARK_MASTER" http://spark-worker-{1,2,3}:8081/metrics/prometheus/ ;;
-  ch-lite)               smoke_ch_lite ;;
+  ch-lite)               smoke_ch_lite; run_example clickhouse 'ch-12 holds 50 rows' ;;
   ch-full)               smoke_ch_full ;;
-  trino)                 smoke_trino ;;
+  trino)                 smoke_trino; how_it_works_trino ;;
   deps)       smoke_deps ;;
   postgres|storage|catalog|valkey) smoke_infra ;;
-  metabase)   smoke_metabase ;;
-  airflow)    smoke_airflow ;;
+  metabase)   smoke_metabase; run_example trino-metabase '"status":"ok"' ;;
+  airflow)    smoke_airflow; run_example airflow-dags 'success'; how_it_works_airflow ;;
   temporal)   smoke_temporal ;;
-  mlflow)     smoke_mlflow ;;
-  lineage)    smoke_lineage ;;
-  telemetry)  smoke_telemetry ;;
-  metadata)   smoke_metadata ;;
-  fluss)      smoke_fluss ;;
-  feast)      smoke_feast ;;
-  evidently)  smoke_evidently ;;
+  mlflow)
+    smoke_mlflow
+    how_it_works_edit mlops-deps .odctl/.env mlflow '^0\.9\.0$'
+    how_it_works_edit timezone .odctl/.env mlflow '^AE[DS]T$'
+    run_example mlflow '"predictions": \[0, 0\]' "$(mlflow_runner) bash" ;;
+  lineage)    smoke_lineage; run_example lineage 'COMPLETED' ;;
+  telemetry)  smoke_telemetry; run_example telemetry '"42"' ;;
+  metadata)   smoke_metadata; run_example openmetadata 'removed demo_pg' ;;
+  fluss)      smoke_fluss; run_example fluss '\| +3 +\| +c +\|' ;;
+  feast)
+    smoke_feast
+    run_example feast-iceberg '0.8999999761581421' "env PATH=${TMPDIR:-/tmp}/odctl-feast-smoke/.venv/bin:$PATH bash"
+    rm -rf "${TMPDIR:-/tmp}/odctl-feast-smoke" ;;
+  evidently)  smoke_evidently; run_example evidently-reports 'report pushed to project' "$(evidently_runner)" ;;
   *)
     echo "ℹ️  $PROFILE: no functional assertion defined, checking containers only"
     [ "$(docker ps -q | wc -l)" -ge 1 ] || fail "no containers running"

{odctl-1.2.0 → odctl-1.2.1}/CLAUDE.md

@@ -8,14 +8,17 @@ A CLI that starts a local open data stack with Docker Compose profiles: Kafka, F
 |---|---|
 | `src/odctl/main.py` | the Typer CLI (`list`, `explain`, `ps`, `info`, `init`, `pull`, `up`, `down`, `logs`, `restart`, `recreate`) |
 | `src/odctl/registry.py`, `resources/registry.yml` | profiles, the services in each, dependencies, ports and usage text |
-| `src/odctl/planner.py`, `docker.py`, `workspace.py`, `config.py`, `ui.py` | startup order, Docker Compose calls, the `.odctl` workspace, settings, terminal output |
+| `src/odctl/planner.py`, `docker.py`, `projects.py`, `logs.py`, `workspace.py`, `config.py`, `ui.py` | startup order, Docker Compose calls, `restart` and `recreate`, and `logs`, each with one Compose project per file as `up` runs them, the `.odctl` workspace, settings, terminal output |
 | `src/odctl/resources/compose-*.yml` | one compose file per area (kafka, flink, spark, store, analytics, mlops, orch, obsv, metadata, infra, deps) |
 | `src/odctl/resources/docker/` | odctl's own images (postgres, mlflow, airflow, spark), published to `ghcr.io/jaehyeon-kim/odctl/<name>:<version>` |
 | `src/odctl/resources/deps/` | the `deps` image: connector and engine jars built or downloaded into the `odctl-shared-deps` volume; versions in `versions.env` |
-| `scripts/gen_pages.py` | builds a page per technology and the CLI reference at docs build time, from `registry.yml`, the compose files and `--help`; run directly, it writes `skills/odctl/SKILL.md`, which a test keeps in step |
-| `scripts/diagrams.py`, `docs/images/*.drawio` | exports each draw.io diagram to the PNG, or GIF for steps, that the docs use; `_legend.drawio` holds the shared shapes |
-| `.github/scripts/smoke.sh` | the e2e assertions per profile group, which run the guide snippets; `select-profiles.py` chooses the groups |
-| `docs/snippets/` | the code each guide shows, one folder per guide |
+| `scripts/gen_pages.py` | builds a page per technology and the CLI reference at docs build time, from `registry.yml`, the compose files and `--help`; run directly, it writes `skills/odctl/SKILL.md`, with every command, option and `.env` key, which a test keeps in step |
+| `scripts/local_links.py`, `scripts/llmstxt_sections.py`, `scripts/release_links.py` | MkDocs hooks: clickable local addresses, the `llms.txt` sections from the nav, and links to files on `main` pointed at the release tag `v<version>` |
+| `scripts/diagrams.py`, `docs/assets/diagrams/*.drawio` | exports each draw.io diagram to the PNG, or GIF for steps, that the docs use, and writes a still PNG of each GIF's last frame beside it; `_legend.drawio` holds the shared shapes; `tests/test_diagrams.py` checks the exports and the stills |
+| `docs/images/` | screenshots of the services' web UIs and the terminal |
+| `image/` | the README's diagram and recording, with the recording's still; they stay here because the PyPI page of every released version links them on `main` |
+| `.github/scripts/smoke.sh` | the e2e assertions per profile group, which run the guide snippets and each guide's Runnable Example; `select-profiles.py` chooses the groups |
+| `docs/snippets/` | the code each guide, Getting Started and How It Works show, one folder per page |
 | `skills/odctl/SKILL.md` | the odctl skill for AI assistants, generated by `scripts/gen_pages.py` |
 | `docs/` | MkDocs Material site, versioned with mike; `docs/about/` holds the public Roadmap and the Changelog |
 | `scripts/release_notes.py` | prints one version's Changelog section, the text of its GitHub release |
@@ -40,7 +43,9 @@ Run the unit suite, `pytest tests/`, in full once before each commit, never afte
 - **One profile per tool**, with `-lite` and `-full` variants where a tool has a small and a clustered form (`kafka-lite`, `kafka-full`).
 - **Fixed container names.** Every service has a `container_name`, so `odctl up` cannot scale a service; several copies are separate named services (Flink's `taskmanager-a` and `taskmanager-b`).
 - **Docs pages that list versions or ports are generated** by `scripts/gen_pages.py`; do not copy those values into hand-written guides.
-- **Guide code lives in `docs/snippets/<guide>/`**, one section per step (`# --8<-- [start:name]`). A guide block labelled `title="path:section"` or `title="path"` is a copy of that section or file, checked by `tests/test_docs.py`; use copies, not `--8<--` includes, so the Markdown reads correctly on GitHub. `smoke.sh` runs the same sections, and `select-profiles.py` maps each snippet folder to the e2e groups that run it.
+- **Guide code lives in `docs/snippets/<guide>/`**, one section per step (`# --8<-- [start:name]`), and Getting Started and How It Works follow the same rule. A block labelled `title="path:section"` or `title="path"` is a copy of that section or file, checked by `tests/test_docs.py`; use copies, not `--8<--` includes, so the Markdown reads correctly on GitHub. Every guide ends with a Runnable Example, `example.sh` or `example.py` in its folder. `smoke.sh` runs the same sections and examples, and `select-profiles.py` maps each snippet folder to the e2e groups that run it.
+- **A feature merges only with its docs.** Each command, option, profile and `.env` key needs a tested example on a guide, Getting Started or How It Works, a reference entry (the CLI reference, a profile page or How It Works), and a place in the skill; a command also needs a line in the README's Commands section. `test_every_feature_has_its_docs` checks this, and its short list of exceptions gives a reason for each.
+- **Each guide says which pages it builds on**, the output of every step, and where a value it quotes is set. A page has at most 10 `##` headings and 15 in all, and headings use title case, keeping product names such as `odctl` as they are.
 - **e2e runs only on tags and manual dispatch**, because a PR's images for an unreleased version do not exist yet. A PR gets lint, unit tests, image builds when image inputs change, and the strict docs build.
 - **Renovate PRs** open on Mondays, image updates grouped by compose file. Show any Renovate config change to the owner before making it.
 - **Writing.** Plain English for non-native readers. No em or en dashes. Headings do not start with "The". Markdown paragraphs are one line each, never hard-wrapped.