annet 0.0__tar.gz

annet-0.0/AUTHORS

@@ -0,0 +1,21 @@
+The following authors have created the source code of "annet" published and distributed by YANDEX LLC as the owner:
+
+Aleksandr Balezin <gescheit@yandex-team.ru>
+Aleksandr Klimenko <v0lk@yandex-team.ru>
+Aleksei Lymar <alymar@nebius.com>
+Alexey Andriyanov <alan@yandex-team.ru>
+Alexey Esin <adess@yandex-team.ru>
+Anton Egorov <eg0rov@yandex-team.ru>
+Artem Denisov <artemdenisov@yandex-team.ru>
+Azat Kurbanov <azatkurbanov@yandex-team.ru>
+Devaev Maxim <mdevaev@yandex-team.ru>
+Fedor Zhukov <azryve@nebius.com>
+Grigorii Ozhegov <ozhegov@yandex-team.ru>
+Grigorii Solovev <gslv@yandex-team.ru>
+Konstantin Sazonov <moonug@yandex-team.ru>
+Roman Glebov <kitaro@yandex-team.ru>
+Roman Karaulanov <anteron92@yandex-team.ru>
+Sergey Mishchenko <smishche@yandex-team.ru>
+Vlad Starostin <vladstar@yandex-team.ru>
+Vladimir Sukhonosov <xornet@yandex-team.ru>
+Vladislav Daniliuk <jayt@nebius.com>

annet-0.0/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) [2023] YANDEX LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

annet-0.0/MANIFEST.in

@@ -0,0 +1 @@
+include requirements.txt

annet-0.0/PKG-INFO

@@ -0,0 +1,25 @@
+Metadata-Version: 2.1
+Name: annet
+Version: 0.0
+Summary: annet
+Home-page: https://github.com/annetutil/annet
+License: MIT
+Requires-Python: >=3.8
+License-File: LICENSE
+License-File: AUTHORS
+Requires-Dist: colorama>=0.4.6
+Requires-Dist: tabulate>=0.9.0
+Requires-Dist: jsonpatch>=1.33
+Requires-Dist: jsonpointer>=2.4
+Requires-Dist: PyYAML>=6.0.1
+Requires-Dist: Pygments>=2.14.0
+Requires-Dist: Mako>=1.2.4
+Requires-Dist: Jinja2>=3.1.2
+Requires-Dist: psutil>=5.8.0
+Requires-Dist: packaging>=23.2
+Requires-Dist: contextlog>=1.1
+Requires-Dist: valkit>=0.1.4
+Requires-Dist: aiohttp>=3.8.4
+Requires-Dist: yarl>=1.8.2
+Requires-Dist: adaptix==3.0.0b2
+Requires-Dist: dataclass-rest==0.4

annet-0.0/README.md

@@ -0,0 +1,111 @@
+# Annet - network configuration utility
+
+The system contains network appliance config generators written in Python with optional use of text preprocessors (Jinja2, Mako).
+Huawei, Cisco IOS, Cisco NX-OS, Cisco IOS-XR, Juniper, as well as devices configured via separate config files (Linux, FreeBSD, Cumulus) are supported.
+
+annet has a number of modes (subcommands):
+
+- ```annet gen``` - generates the entire config for the specified devices or specified parts of it
+- ```annet diff``` - first does gen and then builds diff with current config version
+- ```annet patch``` - first does diff and then generates a list of commands to apply diff on the device
+
+Usage help can be obtained by calling ```annet -h``` or for a specific command, such as ```annet gen -h```.
+
+## Overview
+
+## Configuration
+
+Provide `NETBOX_URL` and `NETBOX_TOKEN` environment variable to setup data source.
+
+```shell
+export NETBOX_URL="https://demo.netbox.dev"
+export NETBOX_TOKEN="1234567890abcdef01234567890abcdef0123456"
+```
+
+### annet gen
+
+The annet_generators directory contains many files called generators.
+A generator takes information about the switch as input and returns the configuration.
+The part of the config that the generator is responsible for is specified in the generator's acl function. If a generator returns a configuration that does not fall under acl, an exception will be thrown.
+
+Example generator:
+
+```python
+from annet.generators import PartialGenerator
+
+class Mtu(PartialGenerator):
+    TAGS = ["mtu"]
+    def acl_cisco(self, _):
+        return "system mtu jumbo"
+
+    def run_cisco(self, device):
+        yield "system mtu jumbo %d" % 9000
+```
+
+
+And an example of calling annet:
+```bash
+annet gen -g mtu sw6-i1
+# -------------------- sw6-i1.cfg --------------------
+system mtu jumbo 9000
+```
+
+Method `acl_cisco` defines scope of the generator, which commands and block it controls.
+The option `-g mtu` means that only generators with the mtu element in the TAGS variable should be called. If no tag is specified, all generators will be executed.
+
+
+### annet diff
+
+If we were configuring the switch from scratch, these options would be enough, but in our reality we need to be able not only to generate the desired configuration, but also to be able to bring the current configuration to the desired one.
+To do this, you need to be able to delete an outdated configuration and correctly add a new one. The **diff** module, which implements some tricky logic, is responsible for this work.
+This logic is defined in the rulebook/texts/VENDOR folder.
+
+Example diff:
+```diff
+# -------------------- sw1-i38.cfg --------------------
+  acl number 2610
+- rule 40 permit source 10.11.170.150 0
++ rule 12 permit source 10.11.133.81 0
+```
+
+### annet patch
+
+Next, you need to create a list of commands from the resulting diff. The **patch** module is responsible for this. It receives the diff, runs the logic specified in rulebook/texts/VENDOR and returns the list of commands.
+Let's take the above diff. It says to remove the command ``rule 40 permit source 10.11.170.150 0`` and add ``rule 12 permit source 10.11.133.81 0``.
+Basic command delete logic for huawei is adding undo to the command. So the undo command will look like this: ``undo rule 40 permit source 10.11.170.150 0```, but this is an invalid command. In case of canceling acl rules, you need to execute ``undo rule N```.
+So you need to write the undo logic for the ```rule ```` command in the ``acl ``` block.
+Here is the part of rulebook/texts/huawei.rul responsible for this:
+```
+acl name *
+	rule * %logic=huawei.misc.undo_redo
+```
+The asterisk here means that the key argument of the undo_redo function will contain the first word after rule, namely the rule number.
+
+Here, the undo_redo function from the file in rulebook/huawei/misc.py is used to generate the command to remove rules in acl.
+```python
+def undo_redo(rule, key, diff, **_):
+    ...
+```
+Now calling ```annet patch -g snmp sw1-i38```` returns the correct set of commands.
+```
+acl number 2610
+  undo rule 40
+  rule 12 permit source 10.11.133.81 0
+  quit
+```
+
+
+<!-- ### annet deploy
+
+To apply these commands on a switch there is a **deploy** module.
+annet can apply changes (roll out) to multiple devices at the same time.
+
+By default, the edits that annet proposes to roll out will be shown before the rollout.
+The user must confirm that they agree to roll out the proposed diff to a given list of devices.
+During the rollout, annet will display the overall progress of the task and the log of one of the devices.
+
+Normal layout. The screen with patches will be shown and the process of laying out will be displayed.
+```bash
+annet deploy -g snmp $HOST
+```
+Credentials will be used from the current user (username, ssh key, ssh agent, encrypted password in $HOME). -->

annet-0.0/annet/__init__.py

@@ -0,0 +1,61 @@
+import logging
+import logging.config
+import os
+import pkgutil
+import sys
+from argparse import SUPPRESS, Namespace
+
+import colorama
+import yaml
+from annet.annlib.errors import (  # pylint: disable=wrong-import-position
+    DeployCancelled,
+    ExecError,
+)
+from contextlog import patch_logging, patch_threading
+from valkit.python import valid_logging_level
+
+import annet.argparse
+
+
+__all__ = ("DeployCancelled", "ExecError")
+
+DEBUG2_LEVELV_NUM = 9
+
+
+def fill_base_args(parser: annet.argparse.ArgParser, pkg_name: str, logging_config: str):
+    parser.add_argument("--log-level", default="WARN", type=valid_logging_level,
+                        help="Уровень детализации логов (DEBUG, DEBUG2 (with comocutor debug), INFO, WARN, CRITICAL)")
+    parser.add_argument("--pkg_name", default=pkg_name, help=SUPPRESS)
+    parser.add_argument("--logging_config", default=logging_config, help=SUPPRESS)
+
+
+def init_logging(options: Namespace):
+    patch_logging()
+    patch_threading()
+    logging.captureWarnings(True)
+    logging_config = yaml.safe_load(pkgutil.get_data(options.pkg_name, options.logging_config))
+    if options.log_level is not None:
+        logging_config.setdefault("root", {})
+        logging_config["root"]["level"] = options.log_level
+    logging.addLevelName(DEBUG2_LEVELV_NUM, "DEBUG2")
+    logging.config.dictConfig(logging_config)
+
+
+def init(options: Namespace):
+    init_logging(options)
+
+    # Отключить colorama.init, если стоит env-переменная. Нужно в тестах
+    if os.environ.get("ANN_FORCE_COLOR", None) not in [None, "", "0", "no"]:
+        colorama.init = lambda *_, **__: None
+    colorama.init()
+
+    # Workaround for Python 3.8.0: https://bugs.python.org/issue38529
+    import asyncio.streams
+    if hasattr(asyncio.streams.StreamReaderProtocol, "_on_reader_gc"):
+        asyncio.streams.StreamReaderProtocol._on_reader_gc = lambda *args, **kwargs: None  # pylint: disable=protected-access
+
+
+def assert_python_version():
+    if sys.version_info < (3, 8, 0):
+        sys.stderr.write("Error: you need python 3.8.0 or higher\n")
+        sys.exit(1)

annet-0.0/annet/adapters/netbox/common/client.py

@@ -0,0 +1,87 @@
+from dataclasses import dataclass
+from functools import wraps
+from typing import Generic, Optional, List, TypeVar, Callable
+
+from dataclass_rest.http.requests import RequestsClient
+from requests import Session
+
+Model = TypeVar("Model")
+
+
+@dataclass
+class PagingResponse(Generic[Model]):
+    next: Optional[str]
+    previous: Optional[str]
+    count: int
+    results: List[Model]
+
+
+Func = TypeVar("Func", bound=Callable)
+
+
+def _collect_by_pages(func: Func) -> Func:
+    """Collect all results using only pagination."""
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        kwargs.setdefault("offset", 0)
+        limit = kwargs.setdefault("limit", 100)
+        results = []
+        method = func.__get__(self, self.__class__)
+        has_next = True
+        while has_next:
+            page = method(*args, **kwargs)
+            kwargs["offset"] += limit
+            results.extend(page.results)
+            has_next = bool(page.next)
+        return PagingResponse(
+            previous=None,
+            next=None,
+            count=len(results),
+            results=results,
+        )
+
+    return wrapper
+
+
+# default batch size 100 is calculated to fit list of UUIDs in 4k URL length
+def collect(func: Func, field: str = "", batch_size: int = 100) -> Func:
+    """
+    Collect data from method iterating over pages and filter batches.
+
+    :param func: Method to call
+    :param field: Field which defines a filter split into batches
+    :param batch_size: Limit of values in `field` filter requested at a time
+    """
+    func = _collect_by_pages(func)
+    if not field:
+        return func
+
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        value = kwargs.get(field)
+        if not value:
+            return func(*args, **kwargs)
+
+        method = func.__get__(self, self.__class__)
+        results = []
+        for offset in range(0, len(value), batch_size):
+            kwargs[field] = value[offset:offset + batch_size]
+            page = method(*args, **kwargs)
+            results.extend(page.results)
+        return PagingResponse(
+            previous=None,
+            next=None,
+            count=len(results),
+            results=results,
+        )
+
+    return wrapper
+
+
+class BaseNetboxClient(RequestsClient):
+    def __init__(self, url: str, token: str):
+        url = url.rstrip("/") + "/api/"
+        session = Session()
+        if token:
+            session.headers["Authorization"] = f"Token {token}"
+        super().__init__(url, session)

annet-0.0/annet/adapters/netbox/common/manufacturer.py

@@ -0,0 +1,62 @@
+from logging import getLogger
+
+from annet.annlib.netdev.views.hardware import HardwareView
+
+logger = getLogger(__name__)
+
+_VENDORS = {
+    "cisco": "Cisco",
+    "catalyst": "Cisco Catalyst",
+    "nexus": "Cisco Nexus",
+    "huawei": "Huawei",
+    "juniper": "Juniper",
+    "arista": "Arista",
+    "pc": "PC",
+    "nokia": "Nokia",
+    "aruba": "Aruba",
+    "routeros": "RouterOS",
+    "ribbon": "Ribbon",
+}
+
+
+def _vendor_to_hw(vendor):
+    return HardwareView(_VENDORS.get(vendor.lower(), vendor), None)
+
+
+def get_hw(manufacturer: str, model: str):
+    # by some reason Netbox calls Mellanox SN as MSN, so we fix them here
+    if manufacturer == "Mellanox" and model.startswith("MSN"):
+        model = model.replace("MSN", "SN", 1)
+    hw = _vendor_to_hw(manufacturer + " " + model)
+    if not hw:
+        raise ValueError(f"unsupported manufacturer {manufacturer}")
+    return hw
+
+
+def get_breed(manufacturer: str, model: str):
+    if manufacturer == "Huawei" and model.startswith("CE"):
+        return "vrp85"
+    elif manufacturer == "Huawei" and model.startswith("NE"):
+        return "vrp85"
+    elif manufacturer == "Huawei":
+        return "vrp55"
+    elif manufacturer == "Mellanox":
+        return "cuml2"
+    elif manufacturer == "Juniper":
+        return "jun10"
+    elif manufacturer == "Cisco":
+        return "ios12"
+    elif manufacturer == "Adva":
+        return "adva8"
+    elif manufacturer == "Arista":
+        return "eos4"
+    raise ValueError(f"unsupported manufacturer {manufacturer}")
+
+
+def is_supported(manufacturer: str) -> bool:
+    if manufacturer not in (
+            "Huawei", "Mellanox", "Juniper", "Cisco", "Adva", "Arista",
+    ):
+        logger.warning("Unsupported manufacturer `%s`", manufacturer)
+        return False
+    return True

annet-0.0/annet/adapters/netbox/common/models.py

@@ -0,0 +1,105 @@
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import List, Optional, Any, Dict
+
+from annet.annlib.netdev.views.hardware import HardwareView
+from annet.storage import Storage
+
+
+@dataclass
+class Entity:
+    id: int
+    name: str
+
+
+@dataclass
+class Label:
+    value: str
+    label: str
+
+
+@dataclass
+class IpFamily:
+    value: int
+    label: str
+
+
+@dataclass
+class DeviceType:
+    id: int
+    manufacturer: Entity
+    model: str
+
+
+@dataclass
+class DeviceIp:
+    id: int
+    display: str
+    address: str
+    family: int
+
+
+@dataclass
+class IpAddress:
+    id: int
+    assigned_object_id: int
+    display: str
+    family: IpFamily
+    address: str
+    status: Label
+    tags: List[Entity]
+    created: datetime
+    last_updated: datetime
+
+
+@dataclass
+class Interface(Entity):
+    device: Entity
+    enabled: bool
+    display: str = ""
+    ip_addresses: List[IpAddress] = field(default_factory=list)
+
+
+@dataclass
+class NetboxDevice(Entity):
+    url: str
+    storage: Storage
+    neighbours_ids: List[int]
+
+    display: str
+    device_type: DeviceType
+    device_role: Entity
+    tenant: Optional[Entity]
+    platform: Optional[Entity]
+    serial: str
+    asset_tag: Optional[str]
+    site: Entity
+    rack: Optional[Entity]
+    position: Optional[float]
+    face: Optional[Label]
+    status: Label
+    primary_ip: Optional[DeviceIp]
+    primary_ip4: Optional[DeviceIp]
+    primary_ip6: Optional[DeviceIp]
+    tags: List[Entity]
+    custom_fields: Dict[str, Any]
+    created: datetime
+    last_updated: datetime
+
+    fqdn: str
+    hostname: str
+    hw: Optional[HardwareView]
+    breed: str
+
+    interfaces: List[Interface]
+
+    # compat
+
+    def __hash__(self):
+        return hash((self.id, type(self)))
+
+    def __eq__(self, other):
+        return type(self) is type(other) and self.url == other.url
+
+    def is_pc(self):
+        return self.device_type.manufacturer.name == "Mellanox"

annet-0.0/annet/adapters/netbox/common/query.py

@@ -0,0 +1,23 @@
+from dataclasses import dataclass
+from typing import List, Union, Iterable, Optional
+
+from annet.storage import Query
+
+
+@dataclass
+class NetboxQuery(Query):
+    query: List[str]
+
+    @classmethod
+    def new(
+            cls, query: Union[str, Iterable[str]],
+            hosts_range: Optional[slice] = None,
+    ) -> "NetboxQuery":
+        if hosts_range is not None:
+            raise ValueError("host_range is not supported")
+        return cls(query=list(query))
+
+    @property
+    def globs(self):
+        # We process every query host as a glob
+        return self.query

annet-0.0/annet/adapters/netbox/common/status_client.py

@@ -0,0 +1,25 @@
+from dataclasses import dataclass
+from typing import Dict
+
+from adaptix import Retort, name_mapping, NameStyle
+from dataclass_rest import get
+from dataclass_rest.client_protocol import FactoryProtocol
+
+from .client import BaseNetboxClient
+
+
+@dataclass
+class Status:
+    netbox_version: str
+    plugins: Dict[str, str]
+
+
+class NetboxStatusClient(BaseNetboxClient):
+    def _init_response_body_factory(self) -> FactoryProtocol:
+        return Retort(recipe=[
+            name_mapping(name_style=NameStyle.LOWER_KEBAB)
+        ])
+
+    @get("status")
+    def status(self) -> Status:
+        ...

annet-0.0/annet/adapters/netbox/common/storage_opts.py

@@ -0,0 +1,14 @@
+import os
+
+
+class NetboxStorageOpts:
+    def __init__(self, url: str, token: str):
+        self.url = url
+        self.token = token
+
+    @classmethod
+    def from_cli_opts(cls, cli_opts):
+        return cls(
+            url=os.getenv("NETBOX_URL", "http://localhost"),
+            token=os.getenv("NETBOX_TOKEN", "").strip(),
+        )

annet-0.0/annet/adapters/netbox/provider.py

@@ -0,0 +1,34 @@
+from dataclass_rest.exceptions import ClientError
+
+from annet.storage import StorageProvider, Storage
+from .common.status_client import NetboxStatusClient
+from .common.storage_opts import NetboxStorageOpts
+from .common.query import NetboxQuery
+from .v24.storage import NetboxStorageV24
+from .v37.storage import NetboxStorageV37
+
+
+def storage_factory(opts: NetboxStorageOpts) -> Storage:
+    client = NetboxStatusClient(opts.url, opts.token)
+    try:
+        status = client.status()
+    except ClientError as e:
+        if e.status_code == 404:
+            # old version do not support status reqeust
+            return NetboxStorageV24(opts)
+        raise
+    if status.netbox_version.startswith("3."):
+        return NetboxStorageV37(opts)
+    else:
+        raise ValueError(f"Unsupported version: {status.netbox_version}")
+
+
+class NetboxProvider(StorageProvider):
+    def storage(self):
+        return storage_factory
+
+    def opts(self):
+        return NetboxStorageOpts
+
+    def query(self):
+        return NetboxQuery

annet-0.0/annet/adapters/netbox/v24/api_models.py

@@ -0,0 +1,73 @@
+from dataclasses import dataclass
+from datetime import datetime
+from typing import List, Optional, Any, Dict
+
+from annet.adapters.netbox.common.models import Entity, DeviceType
+
+
+@dataclass
+class Label:
+    value: int
+    label: str
+
+
+@dataclass
+class DeviceIp:
+    id: int
+    address: str
+    family: int
+
+
+@dataclass
+class Device(Entity):
+    url: str
+    display_name: str
+    device_type: DeviceType
+    device_role: Entity
+    tenant: Optional[Entity]
+    platform: Optional[Entity]
+    serial: str
+    asset_tag: Optional[str]
+    site: Entity
+    rack: Optional[Entity]
+    position: Optional[float]
+    face: Optional[Label]
+    status: Label
+    primary_ip: Optional[DeviceIp]
+    primary_ip4: Optional[DeviceIp]
+    primary_ip6: Optional[DeviceIp]
+    tags: List[str]
+    custom_fields: Dict[str, Any]
+    created: datetime
+    last_updated: datetime
+
+
+@dataclass
+class Interface(Entity):
+    device: Entity
+    enabled: bool
+
+
+@dataclass
+class Vrf(Entity):
+    rd: str
+
+
+@dataclass
+class IpAddress:
+    id: int
+    family: int
+    address: str
+    vrf: Optional[Vrf]
+    tenant: Any  # ???
+    status: Label
+    description: Optional[str]
+    custom_fields: Dict[str, Any]
+    tags: List[str]
+    created: datetime
+    last_updated: datetime
+
+    interface: Entity
+
+    nat_inside: Any  # ???
+    nat_outside: Any  # ???