conlink 2.2.0 → 2.4.0

package/.github/workflows/push.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: npm install
         run: npm install
@@ -19,6 +19,6 @@ jobs:
       - name: compose build of conlink
         run: docker compose -f examples/test1-compose.yaml build
 
-      - name: "./run-tests.sh"
+      - name: "dctest test/test*.yaml"
         timeout-minutes: 5
-        run: time ./run-tests.sh
+        run: time node_modules/.bin/dctest --verbose-commands conlink-test $(ls -v test/test*.yaml)

package/Dockerfile

@@ -24,7 +24,7 @@ FROM node:16-slim as run
 RUN apt-get -y update
 # Runtime deps and utilities
 RUN apt-get -y install libpcap-dev tcpdump iproute2 iputils-ping curl \
-                       iptables bridge-utils ethtool \
+                       iptables bridge-utils ethtool jq \
                        openvswitch-switch openvswitch-testcontroller
 
 COPY --from=build /app/ /app/

package/README.md

@@ -1,9 +1,46 @@
 # conlink: Declarative Low-Level Networking for Containers
 
-
 Create (layer 2 and layer 3) networking between containers using
 a declarative configuration.
 
+Conlink also includes scripts that make docker compose a much more
+powerful development and testing environment (refer to
+[Compose scripts](#compose-scripts-mdc-waitsh-and-copysh) for
+details):
+
+* [mdc](#mdc): modular management of multiple compose configurations
+* [wait.sh](#waitsh): wait for network and file conditions before continuing
+* [copy.sh](#copysh): recursively copy files with variable templating
+
+## Why conlink?
+
+There are a number of limitations of docker-compose networking that
+conlink addresses:
+
+* Operates at layer 3 of the network stack (i.e. IP).
+* Has a fixed model of container interface naming: first interface is
+  `eth0`, second is `eth1`, etc.
+* For containers with multiple interfaces, the mapping between docker
+  compose networks and container interface is not controllable
+  (https://github.com/docker/compose/issues/4645#issuecomment-701351876,
+  https://github.com/docker/compose/issues/8561#issuecomment-1872510968)
+* If a container uses the scale property, then IPs cannot be
+  assigned and user assigned MAC addresses will be the same for every
+  instance of that service.
+* Docker bridge networking interferes with switch and bridge protocol
+  traffic (BPDU, STP, LLDP, etc). Conlink supports the "patch" link
+  mode that allows this type of traffic to pass correctly.
+
+Conlink has the following features:
+
+- Declarative network configuration (links, bridges, patches, etc)
+- Event driven (container restarts and scale changes)
+- Low-level control of network interfaces/links: MTU, routes, port
+  forwarding, netem properties, etc
+- Automatic IP and MAC address incrementing for scaled containers
+- Central network container for easy monitoring and debug
+- Composable configuration from multiple sources/locations
+
 ## Prerequisites
 
 General:
@@ -74,12 +111,12 @@ same bridge (broadcast domain).
 Network configuration can either be loaded directly from configuration
 files using the `--network-config` option or it can be loaded from
 `x-network` properties contained in docker-compose files using the
-`--compose-file`. Multiple of each option may be specified and all the
-network configuration will be merged into a final network
-configuration.
+`--compose-file` option. Multiple of each option may be specified and
+all the network configuration will be merged into a final network
+configuration. Both options also support colon separated lists.
 
-The network configuration can have three top level keys: `links`,
-`tunnels`, and `commands`.
+The network configuration can have four top level keys: `links`,
+`bridges`, `tunnels`, and `commands`.
 
 ### Links
 
@@ -103,12 +140,12 @@ The following table describes the link properties:
 | ip        | *          | CIDR           |         | IP CIDR 7                |
 | mac       | 3          | MAC            |         | MAC addr 7               |
 | mtu       | *          | number 4       | 65535   | intf MTU                 |
-| route     | *          | string         |         | ip route add args        |
+| route     | *          | strings 8      |         | ip route add args        |
 | nat       | *          | IP             |         | DNAT/SNAT to IP          |
-| netem     | *          | string         |         | tc qdisc NetEm options   |
+| netem     | *          | strings 8      |         | tc qdisc NetEm options   |
 | mode      | 5          | string         |         | virt intf mode           |
 | vlanid    | vlan       | number         |         | VLAN ID                  |
-| forward   | veth       | string array 6 |         | forward conlink ports 7  |
+| forward   | veth       | strings 6 8    |         | forward conlink ports 7  |
 
 - 1 - veth, dummy, vlan, ipvlan, macvlan, ipvtap, macvtap
 - 2 - defaults to outer compose service
@@ -117,6 +154,7 @@ The following table describes the link properties:
 - 5 - macvlan, macvtap, ipvlan, ipvtap
 - 6 - string syntax: `conlink_port:container_port/proto`
 - 7 - offset by scale/replica index
+- 8 - either a single string or an array of strings
 
 Each link has a 'type' key that defaults to "veth" and each link
 definition must also have either a `service` key or a `container` key.
@@ -531,13 +569,15 @@ validate connectivity using ping:
 ```
 export BRIDGE_MODE="linux"  # "ovs", "patch", "auto"
 docker-compose -f examples/test9-compose.yaml up --build --force-recreate
-docker-compose -f examples/test9-compose.yaml exec node ping 10.0.1.2
+docker-compose -f examples/test9-compose.yaml exec node1 ping 10.0.1.2
 ```
 
-### test10: port forwarding
+### test10: port forwarding and routing
 
 This example demonstrates port forwarding from the conlink container
-to two containers running simple web servers.
+to two containers running simple web servers. It also demonstrates the
+use of a router container and multiple route rules in the other
+containers.
 
 Start the test10 compose configuration:
 
@@ -581,6 +621,154 @@ curl 0.0.0.0:80
 curl 0.0.0.0:81
 ```
 
+Start a two tcpdump processes in the conlink container to watch
+routed ICMP traffic and then ping between containers across the router
+container:
+
+```
+docker compose -f examples/test10-compose.yaml exec network tcpdump -nli router_1-es1 icmp
+docker compose -f examples/test10-compose.yaml exec network tcpdump -nli router_1-es2 icmp
+```
+
+```
+docker-compose -f examples/test10-compose.yaml exec node1 ping 10.2.0.1
+docker-compose -f examples/test10-compose.yaml exec node1 ping 10.2.0.2
+docker-compose -f examples/test10-compose.yaml exec node2 ping 10.1.0.1
+
+```
+
+## Compose scripts: mdc, wait.sh, and copy.sh
+
+### mdc
+
+The `mdc` command adds flexibility and power to the builtin overlay
+capability of docker compose. Docker compose can specify multiple
+compose files that will be combined into a single configuration.
+Compose files that are specified later will overlay or override
+earlier compose files. For example, if compose files A and B are
+loaded by docker compose, then the `image` property of a service in
+file B will take precedence (or override) the `image` property for the
+same service in file A. Some properties such as `volumes` and
+`environment` will have the sub-properties merged or appended to.
+
+There are several ways that `mdc` adds to the composition capabilities
+of docker compose:
+1. **Mode/module dependency resolution**. The modes or modules that
+   are combined by `mdc` are defined as directories that contain
+   mode/module specific content. A `deps` file in a mode/module
+   directory is used to specify dependencies on other modes/modules.
+   The syntax and resolution algorithm is defined by the
+   [resolve-deps](https://github.com/Viasat/resolve-deps) project.
+2. **Environment variable file combining/overlaying**. Each `.env`
+   file that appears in a mode/module directory will be appended into
+   a single `.env` file at the top-level where the `mdc` command is
+   invoked. Later environment variables will override earlier ones
+   with the same name. Variable interpolation and some shell-style
+   variable expansion can be used to combine/append environment
+   variables. For example if FOO and BAR are defined in an earlier
+   mode/module, then BAZ could be defined like this:
+   `BAZ="${FOO:-${BAR}-SUFF"` which will set BAZ to FOO if FOO is set,
+   otherwise, it will set BAZ to BAR with a "-SUFF" suffix.
+3. **Directory hierarchy combining/overlaying**. If the mode/module
+   directory has subdirectories that themselves contain a "files/"
+   sub-directory, then the mode subdirectories will be recursively
+   copied into the top-level ".files/" directory. For example,
+   consider if the following files exists under the modes "foo" and
+   "bar" (with a dependency of "bar" on "foo"):
+   `foo/svc1/files/etc/conf1`, `foo/svc2/files/etc/conf2`, and
+   `bar/svc1/files/etc/conf1`. When `mdc` is run this will result in
+   the following two files: `.files/svc1/etc/conf1` and
+   `.files/svc2/etc/conf2`. The content of `conf1` will come from the
+   "bar" mode because it is resolved second. The use of the `copy.sh`
+   script (described below) simplifies recursive file copying and also
+   provides variable templating of copied files.
+4. **Set environment variables based on the selected modes/modules**.
+   When `mdc` is run it will set the following special environment
+   variables in the top-level `.env` file:
+   * `COMPOSE_FILE`: A colon separated and dependency ordered list of
+     compose file paths from each resolved mode/module directory.
+   * `COMPOSE_DIR`: The directory where the top-level `.env` is
+     created.
+   * `COMPOSE_PRPOFILES`: A comma separated list of each resolved
+     mode/module with a `MODE_` prefix on the name. These are docker
+     compose profiles that can be used to enable services in one
+     mode/module compose file when a different mode/module is
+     selected/resolved by `mdc`. For example, if a compose file in
+     "bar" has a service that should only be enabled when the "foo"
+     mode/module is also requested/resolved, then the service can be
+     tagged with the `MODE_foo` profile.
+   * `MDC_MODE_DIRS`: A comma separated list of mode/module
+     directories. This can be used by other external tools that have
+     specific mode/module behavior.
+
+Conlink network configuration can be specified in `x-network`
+properties within compose files. This can be a problem with the
+builtin overlay functionality of docker compose because `x-` prefixed
+properties are simply overriden as a whole without any special merging
+behavior. To work around this limitation, conlink has the ability to
+directly merge `x-network` configuration from multiple compose files
+by passing the `COMPOSE_FILE` variable to the conlink `--compose-file`
+parameter (which supports a colon sperated list of compose files).
+
+### wait.sh
+
+The dynamic event driven nature of conlink mean that interfaces may
+appear after the container service code starts running (unlike plain
+docker container networking). For this reason, the `wait.sh` script is
+provided to simplify waiting for interfaces to appear (and other
+network conditions). Here is a compose file snippit that will wait for
+`eth0` to appear and for `eni1` to both appear and have an IP address
+assigned before running the startup command (after the `--`):
+
+```
+services:
+  svc1:
+    volumes:
+      - ./conlink/scripts:/scripts:ro
+    command: /scripts/wait.sh -i eth0 -I eni1 -- /start-cmd.sh arg1 arg2
+```
+
+In addition to waiting for interfaces and address assignment,
+`wait.sh` can also wait for a file to appear (`-f FILE`), a remote TCP
+port to become accessible (`-t HOST:PORT`), or run a command until it
+completes successfully (`-c COMMAND`).
+
+
+### copy.sh
+
+One of the features of the `mdc` command is to collect directory
+hierarchies from mode/module directories into a single `.files/`
+directory at the top-level. The intended use of the merged directory
+hierarchy is to be merged into file-systems of running containers.
+However, simple volume mounts will replace entire directory
+hierarchies (and hide all prior files under the mount point). The
+`copy.sh` script is provided for easily merging/overlaying one
+directory hierarchy onto another one. In addition, the `-T` option
+will also replace special `{{VAR}}` tokens in the files being copied
+with the value of the matching environment variable.
+
+Here is a compose file snippit that shows the use of `copy.sh` to
+recursively copy/overlay the directory tree in `./.files/svc2` onto
+the container root file-system. In addition, due to the use of the
+`-T` option, the script will replace any occurence of the string
+`{{FOO}}` with the value of the `FOO` environment variable within any
+of the files that are copied:
+
+```
+services:
+  svc2:
+    environment:
+      - FOO=123
+    volumes:
+      - ./.files/svc2:/files:ro
+    command: /scripts/copy.sh -T /files / -- /start-cmd.sh arg1 arg2
+```
+
+Note that instances of `copy.sh` and `wait.sh` can be easily chained
+together like this:
+```
+/scripts/copy.sh -T /files / -- /scripts/wait.sh -i eth0 -- cmd args
+```
 
 ## GraphViz network configuration rendering
 

package/examples/test10-compose.yaml

@@ -7,7 +7,8 @@ services:
     command: "python3 -m http.server -d /var 80"
     x-network:
       links:
-        - {bridge: s2, ip: "10.0.1.1/24", route: "default",
+        - {bridge: s1, ip: "10.1.0.1/24",
+           route: ["10.0.0.0/8 via 10.1.0.100"],
            forward: ["1080:80/tcp", "1180:80/tcp"]}
 
   node2:
@@ -17,8 +18,18 @@ services:
     command: "python3 -m http.server -d /usr 80"
     x-network:
       links:
-        - {bridge: s1, ip: "10.0.2.1/24", route: "default",
-           forward: ["80:80/tcp"]}
+        - {bridge: s2, ip: "10.2.0.1/24",
+           route: "10.0.0.0/8 via 10.2.0.100",
+           forward: "80:80/tcp"}
+
+  router:
+    image: python:3-alpine
+    network_mode: none
+    command: sleep Infinity
+    x-network:
+      links:
+        - {bridge: s1, ip: "10.1.0.100/24", dev: es1}
+        - {bridge: s2, ip: "10.2.0.100/24", dev: es2}
 
   network:
     build: {context: ../}

package/examples/test7-compose.yaml

@@ -28,7 +28,12 @@ services:
           ip: 10.0.1.1/16
           mac: 00:0a:0b:0c:0d:01
           mtu: 4111
-          netem: "delay 40ms rate 10mbit"
+          netem: "rate 10mbit delay 40ms"
         - bridge: s2
           ip: 100.0.1.1/16
           dev: eth1
+
+x-network:
+  links:
+    # The delay setting is overridden by the one in the service
+    - {service: node, bridge: s1, netem: "delay 200ms"}

package/examples/test9-compose.yaml

@@ -19,14 +19,19 @@ services:
       - BRIDGE_MODE
     command: /app/build/conlink.js --default-bridge-mode linux --compose-file /test/test9-compose.yaml
 
-  node:
+  node1:
+    image: alpine
+    network_mode: none
+    command: sleep Infinity
+
+  node2:
     image: alpine
     network_mode: none
-    scale: 2
     command: sleep Infinity
 
 x-network:
   links:
-    - {bridge: s1, service: node, ip: 10.0.1.1/24}
+    - {bridge: s1, service: node1, ip: 10.0.1.1/24}
+    - {bridge: s1, service: node2, ip: 10.0.1.2/24}
   bridges:
     - {bridge: s1, mode: "${BRIDGE_MODE:-auto}"}

package/link-add.sh

@@ -32,9 +32,8 @@ usage () {
   echo >&2 "  --mac  MAC0              - MAC address for INTF0"
   echo >&2 "  --mac0 MAC0              - MAC address for INTF0"
   echo >&2 "  --mac1 MAC1              - MAC address for INTF1"
-  echo >&2 "  --route  'ROUTE'         - route to add to INTF0"
-  echo >&2 "  --route|--route0 'ROUTE' - route to add to INTF0"
-  echo >&2 "  --route1 'ROUTE'         - route to add to INTF1"
+  echo >&2 "  --route|--route0 'ROUTE' - route to add to INTF0 (can repeat)"
+  echo >&2 "  --route1 'ROUTE'         - route to add to INTF1 (can repeat)"
   echo >&2 "  --mtu MTU                - MTU for both interfaces"
   echo >&2 ""
   echo >&2 "  --mode MODE              - Mode settings for *vlan TYPEs"
@@ -43,7 +42,7 @@ usage () {
   echo >&2 "  --remote REMOTE          - Remote address for geneve/vxlan types"
   echo >&2 "  --vni VNI                - Virtual Network Identifier for geneve/vxlan types"
   echo >&2 ""
-  echo >&2 "  --netem NETEM            - tc qdisc netem OPTIONS (man 8 netem)"
+  echo >&2 "  --netem NETEM            - tc qdisc netem OPTIONS (man 8 netem) (can repeat)"
   echo >&2 "  --nat TARGET             - Stateless NAT traffic to/from TARGET"
   echo >&2 "                             (in primary/PID0 netns)"
   echo >&2 ""
@@ -54,17 +53,21 @@ info() { echo "link-add [${LOG_ID}] ${*}"; }
 warn() { >&2 echo "link-add [${LOG_ID}] ${*}"; }
 die()  { warn "ERROR: ${*}"; exit 1; }
 
-# Set MAC, IP, ROUTE, MTU, and up state for interface in netns
+# Set MAC, IP, ROUTES, MTU, and up state for interface in netns
 setup_if() {
-  local IF=$1 NS=$2 MAC=$3 IP=$4 ROUTE=$5 MTU=$6
+  local IF=$1 NS=$2 MAC=$3 IP=$4 MTU=$5 ROUTES=$6 routes=
+  echo >&2 "ROUTES: ${ROUTES}"
+  while read rt; do
+      [ "${rt}" ] && routes="${routes}\nroute add ${rt} dev ${IF}"
+  done < <(echo -e "${ROUTES}")
 
-  info "Setting ${IP:+IP ${IP}, }${MAC:+MAC ${MAC}, }${MTU:+MTU ${MTU}, }${ROUTE:+ROUTE '${ROUTE}', }up state"
+  info "Setting ${IP:+IP ${IP}, }${MAC:+MAC ${MAC}, }${MTU:+MTU ${MTU}, }${ROUTES:+ROUTES '${ROUTES//$'\n'/,}', }up state"
   ip -netns ${NS} --force -b - <<EOF
       ${IP:+addr add ${IP} dev ${IF}}
       ${MAC:+link set dev ${IF} address ${MAC}}
       ${MTU:+link set dev ${IF} mtu ${MTU}}
       link set dev ${IF} up
-      ${ROUTE:+route add ${ROUTE} dev ${IF}}
+      $(echo -e "${routes}")
 EOF
 }
 
@@ -78,7 +81,7 @@ IPTABLES() {
 # Parse arguments
 VERBOSE=${VERBOSE:-}
 PID1=${PID1:-<SELF>} IF1=${IF1:-eth0}
-IP0= IP1= MAC0= MAC1= ROUTE0= ROUTE1= MTU=
+IP0= IP1= MAC0= MAC1= ROUTES0= ROUTES1= MTU=
 MODE= VLANID= REMOTE= VNI= NETEM= NAT=
 positional=
 while [ "${*}" ]; do
@@ -91,8 +94,8 @@ while [ "${*}" ]; do
   --ip1)            IP1="${OPTARG}"; shift ;;
   --mac|--mac0)     MAC0="${OPTARG}"; shift ;;
   --mac1)           MAC1="${OPTARG}"; shift ;;
-  --route|--route0) ROUTE0="${OPTARG}"; shift ;;
-  --route1)         ROUTE1="${OPTARG}"; shift ;;
+  --route|--route0) ROUTES0="${ROUTES0}\n${OPTARG}"; shift ;;
+  --route1)         ROUTES1="${ROUTES1}\n${OPTARG}"; shift ;;
   --mtu)            MTU="${OPTARG}"; shift ;;
 
   --mode)           MODE="${OPTARG}"; shift ;;
@@ -101,13 +104,15 @@ while [ "${*}" ]; do
   --remote)         REMOTE="${OPTARG}"; shift ;;
   --vni)            VNI="${OPTARG}"; shift ;;
 
-  --netem)          NETEM="${OPTARG}"; shift ;;
+  --netem)          NETEM="${NETEM} ${OPTARG}"; shift ;;
   --nat)            NAT="${OPTARG}"; shift ;;
   -h|--help)        usage ;;
   *)                positional="${positional} $1" ;;
   esac
   shift
 done
+ROUTES0="${ROUTES0#\\n}"
+ROUTES1="${ROUTES1#\\n}"
 set -- ${positional}
 TYPE=$1 PID0=$2 IF0=$3
 
@@ -181,9 +186,9 @@ geneve|vxlan)
   ;;
 esac
 
-setup_if ${IF0} ${NS0} "${MAC0}" "${IP0}" "${ROUTE0}" "${MTU}"
+setup_if ${IF0} ${NS0} "${MAC0}" "${IP0}" "${MTU}" "${ROUTES0}"
 [ "${TYPE}" = "veth" ] && \
-  setup_if ${IF1} ${NS1} "${MAC1}" "${IP1}" "${ROUTE1}" "${MTU}"
+  setup_if ${IF1} ${NS1} "${MAC1}" "${IP1}" "${MTU}" "${ROUTES1}"
 
 if [ "${NETEM}" ]; then
   info "Setting tc qdisc netem: ${NETEM}"

package/link-forward.sh

@@ -37,7 +37,8 @@ LOG_ID="${spec}"
 info "${action^} forwarding ${intf_a} -> ${intf_b}"
 
 IPTABLES PREROUTING -t nat -i ${intf_a} -p ${proto} --dport ${port_a} -j DNAT --to-destination ${ip}:${port_b}
-IPTABLES POSTROUTING -t nat -o ${intf_b} -j MASQUERADE
+IPTABLES PREROUTING -t nat -i ${intf_a} -p ${proto} --dport ${port_a} -j MARK --set-mark 1
+IPTABLES POSTROUTING -t nat -o ${intf_b} -m mark --mark 1 -j MASQUERADE
 
 case "${action}" in
   add) ip route replace ${ip} dev ${intf_b} ;;

package/link-mirred.sh

@@ -34,41 +34,20 @@ die()  { warn "ERROR: ${*}"; exit 1; }
 add_ingress() {
   local IF=$1 res=
 
-  res=$(tc qdisc show dev ${IF} 2>&1)
-  case "${res}" in
-    *"qdisc ingress ffff:"*)
-      info "${IF0} already has ingress qdisc"
-      ;;
-    ""|*"qdisc noqueue"*)
-      info "Adding ingress qdisc to ${IF}"
-      tc qdisc add dev "${IF}" ingress \
-        || die "Could not add ingress qdisc to ${IF}"
-      ;;
-    *)
-      die "${IF} has invalid ingress qdisc or could not be queried"
-      ;;
-  esac
+  info "Adding ingress qdisc to ${IF}"
+  tc qdisc replace dev "${IF}" ingress \
+    || die "Could not add ingress qdisc to ${IF}"
 }
 
 # Idempotently add mirred filter redirect rule to an interface
 add_mirred() {
   local IF0=$1 IF1=$2 res=
 
-  res=$(tc filter show dev ${IF0} parent ffff: 2>&1)
-  case "${res}" in
-    *"action order 1: mirred (Egress Redirect to device ${IF1}"*)
-      info "${IF0} already has filter redirect action"
-      ;;
-    "")
-      info "Adding filter redirect action from ${IF0} to ${IF1}"
-      tc filter add dev ${IF0} parent ffff: protocol all u32 match u8 0 0 action \
-        mirred egress redirect dev ${IF1} \
-        || die "Could not add filter redirect action from ${IF0} to ${IF1}"
-      ;;
-    *)
-      die "${IF0} has invalid filter redirect action or could not be queried"
-      ;;
-  esac
+  info "Adding filter redirect action from ${IF0} to ${IF1}"
+  tc filter del dev ${IF0} root
+  tc filter replace dev ${IF0} parent ffff: protocol all u32 match u8 0 0 action \
+    mirred egress redirect dev ${IF1} \
+    || die "Could not add filter redirect action from ${IF0} to ${IF1}"
 }
 
 # Parse arguments

package/mdc

@@ -17,21 +17,18 @@ LS=$(which ls)
 RESOLVE_DEPS="${RESOLVE_DEPS-./node_modules/@lonocloud/resolve-deps/resolve-deps.py}"
 DOCKER_COMPOSE="${DOCKER_COMPOSE:-docker-compose}"
 
-[ -f "${RESOLVE_DEPS}" ] || die "Missing ${RESOLVE_DEPS}. Perhaps 'npm install'?"
+which ${RESOLVE_DEPS} >/dev/null 2>/dev/null \
+  || die "Missing ${RESOLVE_DEPS}. Perhaps 'npm install'?"
+
+# Resolve mode directory paths
+
 MODE_SPEC="${1}"; shift
-if [ "${RESOLVE_DEPS}" ]; then
-    MODES="$(${RESOLVE_DEPS} "${MODES_DIR}" ${MODE_SPEC})"
-else
-    MODES="${MODE_SPEC//,/ }"
-fi
+RESOLVED_MODES="$(${RESOLVE_DEPS} --path "${MODES_DIR}" --format=paths ${MODE_SPEC})"
 
-echo >&2 "MODES:    ${MODES}"
-vecho "ENV_FILE: ${ENV_FILE}"
+# Create base (empty) compose file to anchor mode relative paths
+# to the same root directory.  Create files dir.
 
-declare -A FINISHED
 COMPOSE_FILE=./.compose-empty.yaml
-COMPOSE_PROFILES=
-MDC_MODE_DIRS=
 cat /dev/null > ${ENV_FILE}-mdc-tmp
 echo -e "version: '2.4'\nservices: {}" > ./.compose-empty.yaml
 
@@ -42,19 +39,25 @@ esac
 [ -d "${MDC_FILES_DIR}" ] && rm -r${VERBOSE:+v} ${MDC_FILES_DIR}/
 mkdir -p "${MDC_FILES_DIR}"
 
-for mode in ${MODES}; do
+# Incorporate modes' compose config, env, and files
+
+declare -A FINISHED
+COMPOSE_PROFILES=
+MDC_MODE_NAMES=
+MDC_MODE_DIRS=
+for resolved in ${RESOLVED_MODES}; do
+    mode=${resolved%=*}
+    path=${resolved#*=}
+
     # Only process each mode once
     [ "${FINISHED[${mode}]}" ] && continue
     FINISHED["${mode}"]=1
 
-    # mode dir must exist
-    [ -d "${MODES_DIR}/${mode}" ] || \
-        die "No mode dir found for ${mode}"
-
-    MDC_MODE_DIRS="${MDC_MODE_DIRS},${MODES_DIR}/${mode}"
+    MDC_MODE_NAMES="${MDC_MODE_NAMES} ${mode}"
+    MDC_MODE_DIRS="${MDC_MODE_DIRS},${path}"
 
     # mode can refer to a compose file in multiple ways
-    cfiles="${MODES_DIR}/${mode}/compose.yaml ${MODES_DIR}/${mode}/compose.yml ${MODES_DIR}/${mode}/docker-compose.yaml ${MODES_DIR}/${mode}/docker-compose.yml"
+    cfiles="${path}/compose.yaml ${path}/compose.yml ${path}/docker-compose.yaml ${path}/docker-compose.yml"
     for cfile in ${cfiles}; do
         if [ -e "${cfile}" ]; then
             COMPOSE_FILE="${COMPOSE_FILE}:${cfile}"
@@ -66,27 +69,35 @@ for mode in ${MODES}; do
     COMPOSE_PROFILES="${COMPOSE_PROFILES},MODE_${mode}"
 
     # if there is a mode specific env file then include it
-    efiles="${MODES_DIR}/${mode}/env ${MODES_DIR}/${mode}/.env"
+    efiles="${path}/env ${path}/.env"
     for efile in ${efiles}; do
         if [ -e "${efile}" ]; then
-            echo "### mdc begin mode ${mode}" >> ${ENV_FILE}-mdc-tmp
+            echo "### mdc begin mode ${mode} (${efile})" >> ${ENV_FILE}-mdc-tmp
             vecho "cat ${efile} >> ${ENV_FILE}-mdc-tmp"
             cat ${efile} >> ${ENV_FILE}-mdc-tmp
-            echo "### mdc end mode ${mode}" >> ${ENV_FILE}-mdc-tmp
+            echo >> ${ENV_FILE}-mdc-tmp
+            echo "### mdc end mode ${mode} (${efile})" >> ${ENV_FILE}-mdc-tmp
         fi
     done
 
     # if there are mode specific files then copy them to MDC_FILES_DIR
-    if [ -d "${MODES_DIR}/${mode}" ]; then
-        for vfd in $(cd ${MODES_DIR}/${mode} && $LS -d */files 2>/dev/null || true); do
+    if [ -d "${path}" ]; then
+        for vfd in $(cd ${path} && $LS -d */files 2>/dev/null || true); do
             dest=${MDC_FILES_DIR}/${vfd%/files}
             mkdir -p ${dest}
-            vecho cp -a ${MODES_DIR}/${mode}/${vfd}/* ${dest}
-            cp -a ${MODES_DIR}/${mode}/${vfd}/* ${dest}
+            vecho cp -a ${path}/${vfd}/* ${dest}
+            cp -a ${path}/${vfd}/* ${dest}
         done
     fi
 done
 
+vecho
+
+# Summarize and set env
+
+echo >&2 "MODES: ${MDC_MODE_NAMES}"
+vecho "ENV_FILE: ${ENV_FILE}"
+
 COMPOSE_FILE="${COMPOSE_FILE#:}"
 vecho "COMPOSE_FILE: ${COMPOSE_FILE}"
 echo "COMPOSE_FILE=${COMPOSE_FILE}" >> ${ENV_FILE}-mdc-tmp
@@ -100,6 +111,8 @@ MDC_MODE_DIRS="${MDC_MODE_DIRS#,}"
 vecho "MDC_MODE_DIRS: ${MDC_MODE_DIRS}"
 echo "MDC_MODE_DIRS=\"${MDC_MODE_DIRS}\"" >> ${ENV_FILE}-mdc-tmp
 
+vecho
+
 vecho mv ${ENV_FILE}-mdc-tmp ${ENV_FILE}
 mv ${ENV_FILE}-mdc-tmp ${ENV_FILE}
 

package/net2dot.cljs

@@ -29,7 +29,7 @@
       (S/replace #"[*]" "_STAR_")
       (S/replace #"[$]" "_DOLLAR_")
       (S/replace #"[{]" "_LCURLY_")
-      (S/replace #"[}]" "_LCURLY_")
+      (S/replace #"[}]" "_RCURLY_")
       (S/replace #"[ ]" "_SPACE_")))
 
 (defn node-props [label props]
@@ -61,7 +61,7 @@
                   #(->> (subgraph conlink (str "cluster_bridge_" %2)
                                   %2 BRIDGE-PROPS)
                         (assoc %1 %2))
-                  {} (keep :bridge (:links network-config)))
+                  {} (map :bridge (keep :bridge (:links network-config))))
         services (reduce
                    #(->> (subgraph host (str "cluster_service_" (dot-id %2))
                                    (str "service '" (name %2) "'") SVC-PROPS)
@@ -85,7 +85,7 @@
                                    "-" (name dev)))
                 out-id (str "out-" outer-dev)
                 out-parent (condp = (keyword base)
-                             :conlink (get bridges bridge)
+                             :conlink (get bridges (:bridge bridge))
                              :host host)
                 {:keys [type vlanid]} link
                 [elabel iprops] (if (= "host" base)