inmanta-module-openstack 5.0.0__tar.gz

inmanta_module_openstack-5.0.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include inmanta_plugins/openstack/setup.cfg
+include inmanta_plugins/openstack/py.typed
+recursive-include inmanta_plugins/openstack/model *.cf
+graft inmanta_plugins/openstack/files
+graft inmanta_plugins/openstack/templates

inmanta_module_openstack-5.0.0/PKG-INFO

@@ -0,0 +1,8 @@
+Metadata-Version: 2.4
+Name: inmanta-module-openstack
+Version: 5.0.0
+License: Apache 2.0
+Requires-Dist: inmanta-module-ssh
+Requires-Dist: inmanta-module-std
+Requires-Dist: python-dateutil<3,>=2.8
+Requires-Dist: openstacksdk>=1.0

inmanta_module_openstack-5.0.0/README.md

@@ -0,0 +1,261 @@
+# openstack adapter
+
+The openstack module provides support for managing various resources on OpenStack, including virtual
+machines, networks, routers, ...
+
+## How to use it
+
+This guide explains how to start virtual machines on OpenStack.More examples can be found in the `examples` folder of this repo.
+
+### Prerequisites
+
+This tutorial requires you to have an account on an OpenStack. The example below loads the required
+credentials from environment variables, just like the OpenStack command line tools. Additionally,
+the following parameters are also required:
+
+- **ssh_public_key** : Your public ssh key (the key itself, not the name of the file it is in)
+- **network_name**   : The name of the Openstack network to connect the VM to
+- **subnet_name**    : The name of the Openstack subnet to connect the VM to
+- **network_address**: The network address of the subnet above
+- **flavor_name**    : The name of the Openstack flavor to create the VM from
+- **image_id**       : The ID of the Openstack image to boot the VM from
+- **os**             : The OS of the image
+
+### Creating machines
+
+The following model creates a new virtual machine. The parameters in the list above are exposed as variables at the start of the code snippet.
+
+```inmanta
+import openstack
+import ssh
+
+## Edit this parameters
+image_id = ""
+network_name = ""
+subnet_name = ""
+network_address = ""
+
+flavor_name = ""
+ssh_public_key=""
+
+# change OS parameter to match the actual image. If an OS is not modelled in an existing module,
+# std::linux can be used for example. However, other modules might not have support for a
+# generic os definition such as std::linux
+os = std::linux
+## End edit
+
+# register ssh key
+ssh_key = ssh::Key(name="mykey", public_key=ssh_public_key)
+
+# Define the OpenStack provider to use
+provider = openstack::Provider(name="iaas_openstack", connection_url=std::get_env("OS_AUTH_URL"),
+                               username=std::get_env("OS_USERNAME"),
+                               password=std::get_env("OS_PASSWORD"),
+                               project_name=std::get_env("OS_PROJECT_NAME"),
+                               user_domain_name=std::gen_env("OS_USER_DOMAIN_NAME"),
+                               project_domain_name=std::gen_env("OS_PROJECT_DOMAIN_NAME"))
+
+# Define the project to boot the VM in, but do not let inmanta manage it
+project = openstack::Project(provider=provider, name=provider.project_name, description="", enabled=true,
+                             managed=false)
+
+# Define the network objects to connect the virtual machine to but again, do not manage them
+net = openstack::Network(provider=provider, project=project, name=network_name, managed=false)
+subnet = openstack::Subnet(provider=provider, project=project, network=net, dhcp=true, managed=false,
+                           name=subnet_name, network_address=network_address)
+
+# Define the virtual machine
+vm = openstack::Host(provider=provider, project=project, key_pair=ssh_key, name="testhost",
+                     image=image_id, os=os, flavor=flavor_name, user_data="", subnet=subnet)
+```
+
+### Getting the agent on the machine
+
+The user_data attribute of the :inmanta:entity:`openstack::VirtualMachine` entity can inject a shell script that is executed
+at first boot of the virtual machine (through cloud-init). Below is an example script to install
+the inmanta agent (from RPM) and let it connect back to the management server.
+
+```bash
+#!/bin/bash
+
+hostname {{ name }}
+
+curl -1sLf \
+  'https://packages.inmanta.com/public/oss-stable/setup.rpm.sh' \
+  | sudo -E bash
+
+dnf install -y inmanta-oss-agent
+
+cat > /etc/inmanta/agent.cfg <<EOF
+[config]
+heartbeat-interval = 60
+fact-expire = 60
+state-dir=/var/lib/inmanta
+environment={{ env_id }}
+agent-names=\$node-name
+[agent_rest_transport]
+port={{port}}
+host={{env_server}}
+EOF
+
+systemctl start inmanta-agent
+systemctl enable inmanta-agent
+
+```
+
+### Pushing resources to the machine
+
+You can use adapters from std or other modules to manage resources on the machine. For example, we can create a config file in /tmp by adding the following lines to the model that created the virtual machine:
+
+```inmanta
+#put a file on the machine
+std::ConfigFile(host = host1, path="/tmp/test", content="I did it!")
+```
+
+### Actual usage
+
+Creating instances of `openstack::Host`, as shown above requires many parameters and relations,
+creating a model that is hard to read. Often, these parameters are all the same within a single
+model. This means that Inmanta can encapsulate this complexity.
+
+In a larger model, a new `Host` type can encapsulate all settings that are the same for all hosts.
+Additionally, an entity that represents the `infrastructure` can hold shared configuration such as
+the provider, monitoring, shared networks, global parameters,...)
+
+For example (`full source here <https://github.com/inmanta/openstack/tree/master/examples/openstackclean>`_)
+
+Applied to the example above the main file is reduced to:
+
+```inmanta
+import mymodule
+import ssh
+
+## Edit this parameters
+image_id = ""
+network_name = ""
+subnet_name = ""
+network_address = ""
+
+flavor_name = ""
+ssh_public_key=""
+
+# change OS parameter to match the actual image. If an OS is not modelled in an existing module,
+# std::linux can be used for example. However, other modules might not have support for a
+# generic os definition such as std::linux
+os = std::linux
+## End edit
+
+# register ssh key
+ssh_key = ssh::Key(name="mykey", public_key=ssh_public_key)
+
+# create the cluster
+cluster = mymodule::MyCluster(network_name=network_name, subnet_name=subnet_name,
+                              image_id=image_id, flavor=flavor_name, key=ssh_key,
+                              network_address=network_address, os=os)
+
+# make a vm!
+host1 = mymodule::MyHost(name="testhost", cluster=cluster)
+```
+
+With the following module:
+
+```inmanta
+import openstack
+import ssh
+
+
+entity MyCluster:
+    """
+        A cluster object that represents all shared config and infrastructure,
+        including connecting to OpenStack.
+    """
+    string network_name
+    string subnet_name
+    string network_address
+    string image_id
+    string flavor
+end
+
+#input: the ssh key for all VMs
+MyCluster.key [1] -- ssh::Key
+
+#input: the OS for all VMs
+MyCluster.os [1] -- std::OS
+
+#internal: objects needed to construct hosts
+MyCluster.provider [1] -- openstack::Provider
+MyCluster.project [1] -- openstack::Project
+MyCluster.net [1] -- openstack::Network
+MyCluster.subnet [1] -- openstack::Subnet
+
+implementation connection for MyCluster:
+    # Define the OpenStack provider to use
+    self.provider = openstack::Provider(name="iaas_openstack",
+                                        connection_url=std::get_env("OS_AUTH_URL"),
+                                        username=std::get_env("OS_USERNAME"),
+                                        password=std::get_env("OS_PASSWORD"),
+                                        project_name=std::get_env("OS_PROJECT_NAME"),
+                                        user_domain_name=std::gen_env("OS_USER_DOMAIN_NAME"),
+                                        project_domain_name=std::gen_env("OS_PROJECT_DOMAIN_NAME")
+                                        )
+
+    # Define the project to boot the VM in, but do not let inmanta manage it
+    self.project = openstack::Project(provider=self.provider, name=self.provider.project_name,
+                                      description="", enabled=true, managed=false)
+
+    # Define the network objects to connect the virtual machine to but again, do not manage them
+    self.net = openstack::Network(provider=self.provider, project=self.project,
+                                  name=self.network_name, managed=false)
+    self.subnet = openstack::Subnet(provider=self.provider, project=self.project,
+                                    network=self.net, dhcp=true, name=self.subnet_name,
+                                    network_address=self.network_address, managed=false)
+end
+
+implement MyCluster using connection
+
+#define our own host type
+entity MyHost extends openstack::Host:
+end
+
+#input: the cluster object
+MyCluster.hosts [0:] -- MyHost.cluster [1]
+
+implementation myhost for MyHost:
+    #wire up all config for agent injection
+    env_name = std::environment_name()
+    env_id = std::environment()
+    env_server = std::environment_server()
+    port = std::server_port()
+
+    #wire up all config for vm creation
+    self.provider = cluster.provider
+    self.project = cluster.project
+    self.image = cluster.image_id
+    self.subnet = cluster.subnet
+    self.user_data = std::template("mymodule/user_data.tmpl")
+    self.key_pair = cluster.key
+    self.os = cluster.os
+    self.flavor = cluster.flavor
+end
+
+# use our implemenation
+# and also the catchall std::hostDefaults
+# and the openstackVM implementation that sets the ip and create the eth0 port
+implement MyHost using myhost, std::hostDefaults, openstack::openstackVM, openstack::eth0Port
+```
+
+If this were not an example, we would make the following changes:
+
+- hardcode the ``image_id`` and ``os`` (and perhaps ``flavor``) into the defintion of ``myhost``.
+- the parameters on top would be moved to either an lsm service or filled in directly into the constructor.
+- use ``std::password`` to store passwords, to prevent accidential check-ins with passwords in the source
+
+
+
+
+
+```{toctree}
+:maxdepth: 1
+autodoc.rst
+CHANGELOG.md
+```

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/PKG-INFO

@@ -0,0 +1,8 @@
+Metadata-Version: 2.4
+Name: inmanta-module-openstack
+Version: 5.0.0
+License: Apache 2.0
+Requires-Dist: inmanta-module-ssh
+Requires-Dist: inmanta-module-std
+Requires-Dist: python-dateutil<3,>=2.8
+Requires-Dist: openstacksdk>=1.0

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/SOURCES.txt

@@ -0,0 +1,48 @@
+MANIFEST.in
+README.md
+pyproject.toml
+setup.cfg
+inmanta_module_openstack.egg-info/PKG-INFO
+inmanta_module_openstack.egg-info/SOURCES.txt
+inmanta_module_openstack.egg-info/dependency_links.txt
+inmanta_module_openstack.egg-info/not-zip-safe
+inmanta_module_openstack.egg-info/requires.txt
+inmanta_module_openstack.egg-info/top_level.txt
+inmanta_plugins/openstack/__init__.py
+inmanta_plugins/openstack/setup.cfg
+inmanta_plugins/openstack/common/__init__.py
+inmanta_plugins/openstack/common/base.py
+inmanta_plugins/openstack/common/deps.py
+inmanta_plugins/openstack/compute/__init__.py
+inmanta_plugins/openstack/compute/flavor.py
+inmanta_plugins/openstack/compute/host_port.py
+inmanta_plugins/openstack/compute/virtual_machine.py
+inmanta_plugins/openstack/identity/__init__.py
+inmanta_plugins/openstack/identity/group.py
+inmanta_plugins/openstack/identity/group_role.py
+inmanta_plugins/openstack/identity/keystone_base.py
+inmanta_plugins/openstack/identity/project.py
+inmanta_plugins/openstack/identity/role.py
+inmanta_plugins/openstack/identity/user.py
+inmanta_plugins/openstack/image/__init__.py
+inmanta_plugins/openstack/image/image.py
+inmanta_plugins/openstack/model/_init.cf
+inmanta_plugins/openstack/network/__init__.py
+inmanta_plugins/openstack/network/floating_ip.py
+inmanta_plugins/openstack/network/network.py
+inmanta_plugins/openstack/network/qos_policy.py
+inmanta_plugins/openstack/network/router.py
+inmanta_plugins/openstack/network/router_port.py
+inmanta_plugins/openstack/network/security_group.py
+inmanta_plugins/openstack/network/subnet.py
+inmanta_plugins/openstack/network/subnet_v6.py
+inmanta_plugins/openstack/quota/__init__.py
+inmanta_plugins/openstack/quota/quota.py
+tests/test_dependency_handling.py
+tests/test_examples.py
+tests/test_flavor.py
+tests/test_image.py
+tests/test_keystone.py
+tests/test_neutron.py
+tests/test_nova.py
+tests/test_quota.py

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/not-zip-safe

@@ -0,0 +1 @@
+

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/requires.txt

@@ -0,0 +1,4 @@
+inmanta-module-ssh
+inmanta-module-std
+python-dateutil<3,>=2.8
+openstacksdk>=1.0

inmanta_module_openstack-5.0.0/inmanta_module_openstack.egg-info/top_level.txt

@@ -0,0 +1 @@
+inmanta_plugins

inmanta_module_openstack-5.0.0/inmanta_plugins/openstack/__init__.py

@@ -0,0 +1,186 @@
+"""
+Copyright 2022 Inmanta
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+Contact: code@inmanta.com
+"""
+
+import datetime
+import logging
+import math
+from typing import Optional
+
+import keystoneauth1.exceptions
+import openstack.connection
+import openstack.exceptions
+from inmanta.execute.util import Unknown
+from inmanta.plugins import PluginException, plugin
+
+IMAGES = {}
+FIND_IMAGE_RESULT = {}
+
+LOGGER = logging.getLogger(__name__)
+
+
+@plugin
+def find_image(
+    provider: "openstack::Provider", os: "std::OS", name: "string" = None
+) -> "string":
+    """
+    Search for an image that matches the given operating system. This plugin uses
+    the os_distro and os_version tags of an image and the name and version attributes of
+    the OS parameter.
+
+    If multiple images match, the most recent image is returned.
+
+    :param provider: The provider to query for an image
+    :param os: The operating system and version (using os_distro and os_version metadata)
+    :param name: An optional string that the image name should contain
+    """
+    ident = (provider.name, os.name.lower(), str(os.version).lower(), name)
+    if ident in FIND_IMAGE_RESULT:
+        return FIND_IMAGE_RESULT[ident]
+
+    conn = None
+    try:
+        if provider.name not in IMAGES:
+            conn = openstack.connection.Connection(
+                auth_url=provider.connection_url,
+                username=provider.username,
+                password=provider.password,
+                project_name=provider.project_name,
+                user_domain_name=provider.user_domain_name,
+                project_domain_name=provider.project_domain_name,
+                verify_cert=provider.verify_cert,
+            )
+            IMAGES[provider.name] = list(conn.image.images())
+
+        target_distro = os.name.lower()
+        target_version = str(os.version).lower()
+
+        newest_time = datetime.datetime(1900, 1, 1)
+        newest_image = None
+
+        for image in IMAGES[provider.name]:
+            if image.visibility != "public":
+                continue
+            if not image.name or (name and name not in image.name):
+                continue
+
+            os_distro = getattr(image, "os_distro", None)
+            os_version = getattr(image, "os_version", None)
+            if not os_distro or not os_version:
+                continue
+            if (
+                os_distro.lower() != target_distro
+                or os_version.lower() != target_version
+            ):
+                continue
+
+            try:
+                t = datetime.datetime.strptime(image.updated_at, "%Y-%m-%dT%H:%M:%SZ")
+            except Exception:
+                continue
+
+            if t > newest_time:
+                newest_time = t
+                newest_image = image
+
+        if newest_image is None:
+            raise Exception(f"No image found for os {os.name} and version {os.version}")
+
+        result = newest_image.id
+        FIND_IMAGE_RESULT[ident] = result
+        return result
+    except (
+        openstack.exceptions.SDKException,
+        keystoneauth1.exceptions.ClientException,
+    ) as exc:
+        LOGGER.warning("Failed to fetch images for provider %s: %s", provider.name, exc)
+        return Unknown(None)
+    finally:
+        if conn is not None:
+            conn.close()
+
+
+FLAVORS = {}
+FIND_FLAVOR_RESULT = {}
+
+
+@plugin
+def find_flavor(
+    provider: "openstack::Provider",
+    vcpus: "number",
+    ram: "number",
+    pinned: "bool" = False,
+) -> "string":
+    """
+    Find the flavor that matches the closest to the resources requested.
+
+    :param vcpus: The number of virtual cpus in the flavor
+    :param ram: The amount of ram in gigabyte
+    :param pinned: Wether the CPUs need to be pinned (#vcpu == #pcpu)
+    """
+    ident = (provider.name, vcpus, ram, pinned)
+    if ident in FIND_FLAVOR_RESULT:
+        return FIND_FLAVOR_RESULT[ident]
+
+    conn = None
+    try:
+        if provider.name not in FLAVORS:
+            conn = openstack.connection.Connection(
+                auth_url=provider.connection_url,
+                username=provider.username,
+                password=provider.password,
+                project_name=provider.project_name,
+                user_domain_name=provider.user_domain_name,
+                project_domain_name=provider.project_domain_name,
+                verify_cert=provider.verify_cert,
+            )
+            FLAVORS[provider.name] = list(conn.compute.flavors(get_extra_specs=True))
+
+        selected: tuple[float, Optional[openstack.compute.v2.flavor.Flavor]] = (
+            1000000,
+            None,
+        )
+        for flavor in FLAVORS[provider.name]:
+            is_pinned = flavor.extra_specs.get("hw:cpu_policy") == "dedicated"
+            if is_pinned ^ pinned:
+                continue
+
+            d_cpu = flavor.vcpus - vcpus
+            d_ram = (flavor.ram / 1024) - ram
+            distance = math.sqrt(math.pow(d_cpu, 2) + math.pow(d_ram, 2))
+            if d_cpu >= 0 and d_ram >= 0 and distance < selected[0]:
+                selected = (distance, flavor)
+
+        if selected[1] is None:
+            raise PluginException(
+                "Couldn't find a flavor with at least %s %s CPUs and %s Gigabytes of RAM."
+                % (vcpus, "pinned" if pinned else "unpinned", ram)
+            )
+
+        FIND_FLAVOR_RESULT[ident] = selected[1].name
+        return selected[1].name
+    except (
+        openstack.exceptions.SDKException,
+        keystoneauth1.exceptions.ClientException,
+    ) as exc:
+        LOGGER.warning(
+            "Failed to fetch flavors for provider %s: %s", provider.name, exc
+        )
+        return Unknown(None)
+    finally:
+        if conn is not None:
+            conn.close()

inmanta_module_openstack-5.0.0/inmanta_plugins/openstack/common/__init__.py

@@ -0,0 +1,17 @@
+"""
+Copyright 2022 Inmanta
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+Contact: code@inmanta.com
+"""