conlink 2.5.2 → 2.5.4

package/docs/_sidebar.md

@@ -0,0 +1,9 @@
+- **Home**
+  - [Quickstart](/)
+  - [Usage Notes](/usage-notes.md)
+- **Reference**
+  - [Network Configuration Syntax](/reference/network-configuration-syntax.md)
+- **Guides**
+  - [Compose Scripts](/guides/compose-scripts.md)
+  - [Graphviz Rendering](/guides/graphviz-rendering.md)
+  - [Examples](/guides/examples.md)

package/docs/guides/compose-scripts.md

@@ -0,0 +1,143 @@
+# Compose scripts
+
+Conlink also includes scripts that make docker compose a much more
+powerful development and testing environment:
+
+* `mdc` - modular management of multiple compose configurations
+* `wait.sh` - wait for network and file conditions before continuing
+* `copy.sh` - recursively copy files with variable templating
+
+## mdc
+
+The `mdc` command adds flexibility and power to the builtin overlay
+capability of docker compose. Docker compose can specify multiple
+compose files that will be combined into a single configuration.
+Compose files that are specified later will overlay or override
+earlier compose files. For example, if compose files A and B are
+loaded by docker compose, then the `image` property of a service in
+file B will take precedence (or override) the `image` property for the
+same service in file A. Some properties such as `volumes` and
+`environment` will have the sub-properties merged or appended to.
+
+There are several ways that `mdc` adds to the composition capabilities
+of docker compose:
+1. **Mode/module dependency resolution**. The modes or modules that
+   are combined by `mdc` are defined as directories that contain
+   mode/module specific content. A `deps` file in a mode/module
+   directory is used to specify dependencies on other modes/modules.
+   The syntax and resolution algorithm is defined by the
+   [resolve-deps](https://github.com/Viasat/resolve-deps) project.
+2. **Environment variable file combining/overlaying**. Each `.env`
+   file that appears in a mode/module directory will be appended into
+   a single `.env` file at the top-level where the `mdc` command is
+   invoked. Later environment variables will override earlier ones
+   with the same name. Variable interpolation and some shell-style
+   variable expansion can be used to combine/append environment
+   variables. For example if FOO and BAR are defined in an earlier
+   mode/module, then BAZ could be defined like this:
+   `BAZ="${FOO:-${BAR}-SUFF"` which will set BAZ to FOO if FOO is set,
+   otherwise, it will set BAZ to BAR with a "-SUFF" suffix.
+3. **Directory hierarchy combining/overlaying**. If the mode/module
+   directory has subdirectories that themselves contain a "files/"
+   sub-directory, then the mode subdirectories will be recursively
+   copied into the top-level ".files/" directory. For example,
+   consider if the following files exists under the modes "foo" and
+   "bar" (with a dependency of "bar" on "foo"):
+   `foo/svc1/files/etc/conf1`, `foo/svc2/files/etc/conf2`, and
+   `bar/svc1/files/etc/conf1`. When `mdc` is run this will result in
+   the following two files: `.files/svc1/etc/conf1` and
+   `.files/svc2/etc/conf2`. The content of `conf1` will come from the
+   "bar" mode because it is resolved second. The use of the `copy.sh`
+   script (described below) simplifies recursive file copying and also
+   provides variable templating of copied files.
+4. **Set environment variables based on the selected modes/modules**.
+   When `mdc` is run it will set the following special environment
+   variables in the top-level `.env` file:
+   * `COMPOSE_FILE`: A colon separated and dependency ordered list of
+     compose file paths from each resolved mode/module directory.
+   * `COMPOSE_DIR`: The directory where the top-level `.env` is
+     created.
+   * `COMPOSE_PROFILES`: A comma separated list of each resolved
+     mode/module with a `MODE_` prefix on the name. These are docker
+     compose profiles that can be used to enable services in one
+     mode/module compose file when a different mode/module is
+     selected/resolved by `mdc`. For example, if a compose file in
+     "bar" has a service that should only be enabled when the "foo"
+     mode/module is also requested/resolved, then the service can be
+     tagged with the `MODE_foo` profile.
+   * `MDC_MODE_[mode]`: Each active mode will have an environment
+     variable set to 'enabled'. Important Note: the mode name will
+     have all non-alphanumeric characters changed to an underscore so
+     that it is usable as the environment variable suffix.
+   * `MDC_MODE_DIRS`: A comma separated list of mode/module
+     directories. This can be used by other external tools that have
+     specific mode/module behavior.
+
+Conlink network configuration can be specified in `x-network`
+properties within compose files. This can be a problem with the
+builtin overlay functionality of docker compose because `x-` prefixed
+properties are simply overriden as a whole without any special merging
+behavior. To work around this limitation, conlink has the ability to
+directly merge `x-network` configuration from multiple compose files
+by passing the `COMPOSE_FILE` variable to the conlink `--compose-file`
+parameter (which supports a colon sperated list of compose files).
+
+## wait.sh
+
+The dynamic event driven nature of conlink mean that interfaces may
+appear after the container service code starts running (unlike plain
+docker container networking). For this reason, the `wait.sh` script is
+provided to simplify waiting for interfaces to appear (and other
+network conditions). Here is a compose file snippit that will wait for
+`eth0` to appear and for `eni1` to both appear and have an IP address
+assigned before running the startup command (after the `--`):
+
+```
+services:
+  svc1:
+    volumes:
+      - ./conlink/scripts:/scripts:ro
+    command: /scripts/wait.sh -i eth0 -I eni1 -- /start-cmd.sh arg1 arg2
+```
+
+In addition to waiting for interfaces and address assignment,
+`wait.sh` can also wait for a file to appear (`-f FILE`), a remote TCP
+port to become accessible (`-t HOST:PORT`), or run a command until it
+completes successfully (`-c COMMAND`).
+
+
+## copy.sh
+
+One of the features of the `mdc` command is to collect directory
+hierarchies from mode/module directories into a single `.files/`
+directory at the top-level. The intended use of the merged directory
+hierarchy is to be merged into file-systems of running containers.
+However, simple volume mounts will replace entire directory
+hierarchies (and hide all prior files under the mount point). The
+`copy.sh` script is provided for easily merging/overlaying one
+directory hierarchy onto another one. In addition, the `-T` option
+will also replace special `{{VAR}}` tokens in the files being copied
+with the value of the matching environment variable.
+
+Here is a compose file snippit that shows the use of `copy.sh` to
+recursively copy/overlay the directory tree in `./.files/svc2` onto
+the container root file-system. In addition, due to the use of the
+`-T` option, the script will replace any occurence of the string
+`{{FOO}}` with the value of the `FOO` environment variable within any
+of the files that are copied:
+
+```
+services:
+  svc2:
+    environment:
+      - FOO=123
+    volumes:
+      - ./.files/svc2:/files:ro
+    command: /scripts/copy.sh -T /files / -- /start-cmd.sh arg1 arg2
+```
+
+Note that instances of `copy.sh` and `wait.sh` can be easily chained
+together like this:
+```
+/scripts/copy.sh -T /files / -- /scripts/wait.sh -i eth0 -- cmd args
+```

package/docs/guides/examples.md

@@ -0,0 +1,395 @@
+# Examples
+
+The [examples](https://github.com/LonoCloud/conlink/tree/master/examples)
+directory contains the necessary files to follow along below.
+
+The examples also require a conlink docker image. Build the image for both
+docker and podman like this:
+
+```
+docker build -t conlink .
+podman build -t conlink .
+```
+
+## test1: compose file with embedded network config
+
+Start the test1 compose configuration:
+
+```
+docker-compose -f examples/test1-compose.yaml up --build --force-recreate
+```
+
+From h1 ping the address of h3 (routed via the r0 container):
+
+```
+docker-compose -f examples/test1-compose.yaml exec h1 ping 10.0.0.100
+```
+
+
+## test2: compose file with separate network config and live scaling
+
+Start the test2 compose configuration:
+
+```
+docker-compose -f examples/test2-compose.yaml up -d --build --force-recreate
+```
+
+From the first node ping the second:
+
+```
+docker-compose -f examples/test2-compose.yaml exec --index 1 node ping 10.0.1.2
+```
+
+From the second node ping an address in the internet service:
+
+```
+docker-compose -f examples/test2-compose.yaml exec --index 2 node ping 8.8.8.8
+```
+
+Scale the nodes from 2 to 5 and then ping the fifth node from the second:
+
+```
+docker-compose -f examples/test2-compose.yaml up -d --scale node=5
+docker-compose -f examples/test2-compose.yaml exec --index 2 node ping 10.0.1.5
+```
+
+
+## test3: network config file only (no compose) and variable templating
+
+### test3 with docker
+
+Start two containers named `ZZZ_node_1` and `ZZZ_node_2`.
+
+```
+docker run --name=ZZZ_node_1 --rm -d --network none alpine sleep 864000
+docker run --name=ZZZ_node_2 --rm -d --network none alpine sleep 864000
+```
+
+Start the conlink container `ZZZ_network` that will setup a network
+configuration that is connected to the other containers:
+
+```
+./conlink-start.sh -v --mode docker --host-mode docker --network-file examples/test3-network.yaml -- --name ZZZ_network --rm -e NODE_NAME_1=ZZZ_node_1 -e NODE_NAME_2=ZZZ_node_2
+```
+
+In a separate terminal, ping the node 2 from node 1.
+
+```
+docker exec -it ZZZ_node_1 ping 10.0.1.2
+```
+
+### test3 with rootless podman
+
+Same as test3 but using rootless podman instead
+
+Start two containers named `ZZZ_node_1` and `ZZZ_node_2`.
+
+```
+podman run --name=ZZZ_node_1 --rm -d --network none alpine sleep 864000
+podman run --name=ZZZ_node_2 --rm -d --network none alpine sleep 864000
+```
+
+Start the conlink container `ZZZ_network` that will setup a network
+configuration that is connected to the other containers:
+
+```
+./conlink-start.sh -v --mode podman --host-mode podman --network-file examples/test3-network.yaml -- --name ZZZ_network --rm -e NODE_NAME_1=ZZZ_node_1 -e NODE_NAME_2=ZZZ_node_2
+```
+
+In a separate terminal, ping the node 2 from node 1.
+
+```
+podman exec -it ZZZ_node_1 ping 10.0.1.2
+```
+
+## test4: multiple compose files and container commands
+
+Docker-compose has the ability to specify multiple compose files that
+are merged together into a single runtime configuration. This test
+has conlink configuration spread across multiple compose files and
+a separate network config file. The network configuration appears at the
+top-level of the compose files and also within multiple compose
+service definitions.
+
+Run docker-compose using two compose files. The first defines the
+conlink/network container and a basic network configuration that
+includes a router and switch (`s0`). The second defines a single
+container (`node1`) and switch (`s1`) that is connected to the router
+defined in the first compose file.
+
+```
+MODES_DIR=./examples/test4-multiple/modes ./mdc node1 up --build --force-recreate
+```
+
+Ping the router host from `node1`:
+
+```
+docker-compose exec node1 ping 10.0.0.100
+```
+
+Restart the compose instance and add another compose file that defines
+two node2 replicas and a switch (`s2`) that is connected to the
+router.
+
+```
+MODES_DIR=./examples/test4-multiple/modes ./mdc node1,nodes2 up --build --force-recreate
+```
+
+From both `node2` replicas, ping `node1` across the switches and `r0` router:
+
+```
+docker-compose exec --index 1 node2 ping 10.1.0.1
+docker-compose exec --index 2 node2 ping 10.1.0.1
+```
+
+From `node1`, ping both `node2` replicas across the switches and `r0` router:
+
+```
+docker-compose exec node1 ping 10.2.0.1
+docker-compose exec node1 ping 10.2.0.2
+```
+
+
+Restart the compose instance and add another compose file that starts
+conlink using an addition network file `web-network.yaml`. The network
+file starts up a simple web server on the router.
+
+```
+MODES_DIR=./examples/test4-multiple/modes ./mdc node1,nodes2,web up --build --force-recreate
+```
+
+From the second `node2`, perform a download from the web server running on the
+router host:
+
+```
+docker-compose exec --index 2 node2 wget -O- 10.0.0.100
+```
+
+We can simplify the above launch by using the `all` mode, which contains
+depends on `node1`, `nodes2`, and `web`.  Each of those modes depends on
+`base`, so there's no need to specify that again (transitive deps).
+
+```
+MODES_DIR=./examples/test4-multiple/modes ./mdc all up --build --force-recreate
+```
+
+Remove the `.env` file as a final cleanup step:
+
+```
+rm .env
+```
+
+
+## test5: conlink on two hosts with overlay connectivity via geneve
+
+Launch a compose instance on host 1 and point it at host 2:
+
+```
+echo "REMOTE=ADDRESS_OF_HOST_2" > .env
+echo "NODE_IP=192.168.100.1" > .env \
+docker-compose --env-file .env -f examples/test5-geneve-compose.yaml up
+```
+
+Launch another compose instance on host 2 and point it at host 1:
+On host 2 run conlink like this:
+
+```
+echo "REMOTE=ADDRESS_OF_HOST_1" > .env
+echo "NODE_IP=192.168.100.2" >> .env \
+docker-compose --env-file .env -f examples/test5-geneve-compose.yaml up
+```
+
+On host 1, start a tcpdump on the main interface capturing Geneve
+(encapsulated) traffic:
+
+```
+sudo tcpdump -nli eth0 port 6081
+```
+
+On host 2, start a ping within the "node1" network namespace created
+by conlink:
+
+```
+docker-compose -f examples/test5-geneve-compose.yaml exec node ping 192.168.100.1
+```
+
+On host 1 you should see bi-directional encapsulated ping traffic on the host.
+
+
+## test6: conlink on two hosts deployed with CloudFormation
+
+This test uses AWS CloudFormation to deploy two AWS EC2 instances that
+automatically install, configure, and start conlink (and dependencies)
+using the `test5-geneve-compose.yaml` compose file.
+
+Authenticate with AWS and set the `MY_KEY`, `MY_VPC`, and `MY_SUBNET`
+variables to refer to a preexisting key pair name, VPC ID, and Subnet
+ID respectively. Then use the AWS CLI to deploy the stack:
+
+```
+export MY_KEY=... MY_VPC=... MY_SUBNET=...
+aws --region us-west-2 cloudformation deploy --stack-name ${USER}-conlink-test6 --template-file examples/test6-cfn.yaml --parameter-overrides KeyPairName=${MY_KEY} VpcId=${MY_VPC} SubnetId=${MY_SUBNET}
+```
+
+The stack will take about 8 minutes to finish deploying. You can
+reduce the time to under a minute if you create your own AMI with the
+pre-signal steps in `BaseUserData` baked in and modify the template to
+use that instead.
+
+Once the stack is finish deploying, show the outputs of the stack
+(including instance IP addresses) like this:
+
+```
+aws --region us-west-2 cloudformation describe-stacks --stack-name ${USER}-conlink-test6 | jq '.Stacks[0].Outputs'
+```
+
+Use ssh to connect to instance 1 and 2 (as the "ubuntu" user), then
+sudo to root and cd into `/root/conlink`. You can now run the tcpdump
+and ping test described for test5.
+
+
+## test7: MAC, MTU, and NetEm settings
+
+This example demonstrates using interface MAC, MTU, and NetEm (tc
+qdisc) settings.
+
+Start the test7 compose configuration:
+
+```
+docker-compose -f examples/test7-compose.yaml up --build --force-recreate
+```
+
+Show the links in both node containers to see that on the eth0
+interfaces the MAC addresses are `00:0a:0b:0c:0d:0*` and the MTUs are
+set to `4111`. The eth1 interfaces should have the command line set
+default MTU of `5111`.
+
+```
+docker-compose -f examples/test7-compose.yaml exec --index 1 node ip link
+docker-compose -f examples/test7-compose.yaml exec --index 2 node ip link
+```
+
+Ping the second node from the first to show the the NetEm setting is
+adding 40ms delay in both directions (80ms roundtrip).
+
+```
+docker-compose -f examples/test7-compose.yaml exec --index 1 node ping 10.0.1.2
+```
+
+## test8: Connections to macvlan/vlan host interfaces
+
+This example has two nodes with web servers bound to local addresses.
+The first node is connected to a macvlan sub-interfaces of a host
+physical interface. The second node is connected to a VLAN
+sub-interface of the same host (using VLAN ID/tag 5). Static NAT
+(SNAT+DNAT) is setup inside each container to map the external
+address/interface to the internal address/interface (dummy) where the
+web server is running.
+
+Create an environment file with the name of the parent host interface
+and the external IP addresses to assign to each container:
+
+```
+cat << EOF > .env
+HOST_INTERFACE=enp6s0
+NODE1_HOST_ADDRESS=192.168.0.32/24
+NODE2_HOST_ADDRESS=192.168.0.33/24
+EOF
+```
+
+Start the test8 compose configuration using the environment file:
+
+```
+docker-compose --env-file .env -f examples/test8-compose.yaml up --build --force-recreate
+```
+
+Connect to the macvlan node (NODE1_HOST_ADDRESS) from an external host
+on your network (traffic to macvlan interfaces on the same host is
+prevented):
+
+```
+ping -c1 192.168.0.32
+curl 192.168.0.32
+```
+
+Note: to connect to the vlan node (NODE2_HOST_ADDRESS) you will need
+to configure your physical switch/router with routing/connectivity to
+VLAN 5 on the same physical link to your host.
+
+## test9: bridge modes
+
+This example demonstrates the supported bridge modes.
+
+Start the test9 compose configuration using different bridge modes and
+validate connectivity using ping:
+
+```
+export BRIDGE_MODE="linux"  # "ovs", "patch", "auto"
+docker-compose -f examples/test9-compose.yaml up --build --force-recreate
+docker-compose -f examples/test9-compose.yaml exec node1 ping 10.0.1.2
+```
+
+## test10: port forwarding and routing
+
+This example demonstrates port forwarding from the conlink container
+to two containers running simple web servers. It also demonstrates the
+use of a router container and multiple route rules in the other
+containers.
+
+Start the test10 compose configuration:
+
+```
+docker-compose -f examples/test10-compose.yaml up --build --force-recreate
+```
+
+Ports 3080 and 8080 are both published on the host by the conlink
+container using standard Docker port mapping. The internal mapping of
+those ports (1080 and 1180 respectively) are both are forwarded to
+port 80 in the node1 container using conlink's port forwarding
+mechanism. The two paths look like this:
+
+```
+host:3080 --> 1080 (in conlink) --> node1:80
+host:8080 --> 1180 (in conlink) --> node1:80
+```
+
+Use curl on the host to query both of these paths to node1:
+
+```
+curl 0.0.0.0:3080
+curl 0.0.0.0:8080
+```
+
+Ports 80 and 81 are published on the host by the conlink container
+using standard Docker port mapping. Then conlink forwards from ports
+80 and 81 to the first and second replica (respectively) of node2,
+each of which listen internally on port 80. The two paths look like
+this:
+
+```
+host:80 -> 80 (in conlink) -> node2_1:80
+host:81 -> 81 (in conlink) -> node2_2:80
+```
+
+Use curl on the host to query both replicas of node2:
+
+```
+curl 0.0.0.0:80
+curl 0.0.0.0:81
+```
+
+Start a two tcpdump processes in the conlink container to watch
+routed ICMP traffic and then ping between containers across the router
+container:
+
+```
+docker compose -f examples/test10-compose.yaml exec network tcpdump -nli router_1-es1 icmp
+docker compose -f examples/test10-compose.yaml exec network tcpdump -nli router_1-es2 icmp
+```
+
+```
+docker-compose -f examples/test10-compose.yaml exec node1 ping 10.2.0.1
+docker-compose -f examples/test10-compose.yaml exec node1 ping 10.2.0.2
+docker-compose -f examples/test10-compose.yaml exec node2 ping 10.1.0.1
+
+```

package/docs/guides/graphviz-rendering.md

@@ -0,0 +1,32 @@
+# GraphViz network configuration rendering
+
+You can use d3 and GraphViz to create a visual graph rendering of
+a network configuration. First start a simple web server in the
+examples directory:
+
+```
+cd examples
+python3 -m http.server 8080
+```
+
+Use the `net2dot` script to transform a network
+configuration into a GraphViz data file (dot language). To render the
+network configuration for example test1, run the following in another
+window:
+
+```
+./conlink --show-config --compose-file examples/test1-compose.yaml | ./net2dot > examples/test1.dot
+```
+
+Then load `http://localhost:8080?data=test1.dot` in your browser to see the rendered
+image.
+
+The file `examples/net2dot.yaml` contains a configuration that
+combines many different configuration elements (veth links, dummy
+interfaces, vlan type links, tunnels, etc).
+
+```
+./conlink --network-file examples/net2dot.yaml --show-config | ./net2dot > examples/net2dot.dot
+```
+
+Then load `http://localhost:8080?data=net2dot.dot` in your browser.

package/docs/index.html

@@ -0,0 +1,33 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <title>conlink - Declarative Low-Level Networking for Containers</title>
+  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
+  <meta name="description" content="Create (layer 2 and layer 3) networking between containers using a declarative configuration.">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
+  <!-- using docsify-themeable Theme: Simple -->
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsify-themeable@0/dist/css/theme-simple.css">
+</head>
+<body>
+  <div id="app"></div>
+  <script>
+    window.$docsify = {
+      name: 'conlink',
+      repo: 'LonoCloud/conlink',
+      homepage: 'https://raw.githubusercontent.com/LonoCloud/conlink/master/README.md',
+      // Uncomment to follow a symlink to root README.md and test rendering locally
+      //homepage: '_render-local-README.md',
+
+      auto2top: true,
+      loadSidebar: true,
+      subMaxLevel: 2
+    }
+  </script>
+  <!-- Docsify v4 -->
+  <script src="//cdn.jsdelivr.net/npm/docsify@4"></script>
+  <!-- code highlighting -->
+  <script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-bash.min.js"></script>
+  <script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-yaml.min.js"></script>
+</body>
+</html>