annet 0.16.8__tar.gz → 0.16.9__tar.gz

{annet-0.16.8/annet.egg-info → annet-0.16.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: annet
-Version: 0.16.8
+Version: 0.16.9
 Summary: annet
 Home-page: https://github.com/annetutil/annet
 License: MIT
@@ -23,3 +23,5 @@ Requires-Dist: aiohttp>=3.8.4
 Requires-Dist: yarl>=1.8.2
 Requires-Dist: adaptix==3.0.0b7
 Requires-Dist: dataclass-rest==0.4
+Requires-Dist: requests>=2.32.3
+Requires-Dist: annetbox>=0.1.8; python_version > "3.9"

{annet-0.16.8 → annet-0.16.9}/README.md

@@ -1,13 +1,15 @@
-# Annet - network configuration utility
+# Annet - configuration generation and deploying utility for network equipment
 
-The system contains network appliance config generators written in Python with optional use of text preprocessors (Jinja2, Mako).
-Huawei, Cisco IOS, Cisco NX-OS, Cisco IOS-XR, Juniper, as well as devices configured via separate config files (Linux, FreeBSD, Cumulus) are supported.
+Annet is a configuration generator that can translate differences between old and new configurations into sequnce of commands. This feature is vital for CLI-based devices, such as Huawei, Cisco IOS, Cisco NX-OS, Juniper. Devices configured via separate config files, Linux, FreeBSD and Cumulus are also supported.
 
-annet has a number of modes (subcommands):
+It works this way. Annet `gen`erates configuration for a device by running Python code, which usually goes to the Network Source of Truth, like NetBox. Annet then gets the `diff`erence by getting the configuration from the device and comparing it. Finally, Annet translates the difference into a sequence of commands, called a `patch`. After `deploy`ing these commands, the diff will be empty.
+
+Annet has a number of modes (subcommands):
 
 - ```annet gen``` - generates the entire config for the specified devices or specified parts of it
 - ```annet diff``` - first does gen and then builds diff with current config version
 - ```annet patch``` - first does diff and then generates a list of commands to apply diff on the device
+- ```annet deploy``` - first does patch and then deploys it to the device
 
 Usage help can be obtained by calling ```annet -h``` or for a specific command, such as ```annet gen -h```.
 
@@ -68,7 +70,7 @@ So you need to write the undo logic for the ```rule ```` command in the ``acl ``
 Here is the part of rulebook/texts/huawei.rul responsible for this:
 ```
 acl name *
-	rule * %logic=huawei.misc.undo_redo
+  rule * %logic=huawei.misc.undo_redo
 ```
 The asterisk here means that the key argument of the undo_redo function will contain the first word after rule, namely the rule number.
 
@@ -77,7 +79,7 @@ Here, the undo_redo function from the file in rulebook/huawei/misc.py is used to
 def undo_redo(rule, key, diff, **_):
     ...
 ```
-Now calling ```annet patch -g snmp sw1-i38```` returns the correct set of commands.
+Now calling `annet patch -g snmp sw1-i38` returns the correct set of commands.
 ```
 acl number 2610
   undo rule 40
@@ -85,21 +87,62 @@ acl number 2610
   quit
 ```
 
-
-<!-- ### annet deploy
+### annet deploy
 
 To apply these commands on a switch there is a **deploy** module.
-annet can apply changes (roll out) to multiple devices at the same time.
+Annet can apply changes (roll out) to multiple devices at the same time.
 
 By default, the edits that annet proposes to roll out will be shown before the rollout.
 The user must confirm that they agree to roll out the proposed diff to a given list of devices.
 During the rollout, annet will display the overall progress of the task and the log of one of the devices.
 
 Normal layout. The screen with patches will be shown and the process of laying out will be displayed.
+
 ```bash
 annet deploy -g snmp $HOST
 ```
-Credentials will be used from the current user (username, ssh key, ssh agent, encrypted password in $HOME). -->
+
+Credentials will be used from the current user (username, ssh key, ssh agent).
+
+## Installation
+
+```shell
+mkdir myproject
+cd myproject
+python3 -m venv venv
+source venv/bin/activate
+pip install annet gnetcli_adapter
+
+cat > ~/.annet/context.yml_tmp<<EOF
+fetcher:
+  default:
+    adapter: gnetcli
+deployer:
+  default:
+    adapter: gnetcli
+generators:
+  default:
+    - my_generators
+storage:
+  netbox:
+    adapter: netbox
+    params:
+      url: http://127.0.0.1:8000
+      token: 1234567890abcdef01234567890abcdef0123456
+context:
+  default:
+    fetcher: default
+    deployer: default
+    connection: default
+    generators: default
+    storage: default
+selected_context: default
+EOF
+
+cp -r my_generators
+
+annet deploy mydevice
+```
 
 ## Configuration
 
@@ -174,3 +217,20 @@ devices:
 
 Annet uses [Entry Points](https://setuptools.pypa.io/en/latest/userguide/entry_point.html) mechanism for customization.
 For example, you can implement the Storage interface on top of your favorite inventory system.
+
+
+## Building doc
+
+1. Install dependencies:
+
+```shell
+pip install -r requirements-doc.txt
+```
+
+2. Build
+
+```shell
+sphinx-build -M html docs docs-build
+```
+
+3. Open rendered html in browser [docs-build/html/index.html](docs-build/html/index.html)

annet-0.16.9/annet/adapters/fetchers/stub/fetcher.py

@@ -0,0 +1,19 @@
+from annet.deploy import Fetcher
+from annet.connectors import AdapterWithConfig
+from typing import Dict, List, Any
+from annet.storage import Device
+
+
+class StubFetcher(Fetcher, AdapterWithConfig):
+    @classmethod
+    def with_config(cls, **kwargs: Dict[str, Any]) -> Fetcher:
+        return cls(**kwargs)
+
+    def fetch_packages(self, devices: List[Device],
+                       processes: int = 1, max_slots: int = 0):
+        raise NotImplementedError()
+
+    def fetch(self, devices: List[Device],
+              files_to_download: Dict[str, List[str]] = None,
+              processes: int = 1, max_slots: int = 0):
+        raise NotImplementedError()

annet-0.16.9/annet/adapters/file/provider.py

@@ -0,0 +1,226 @@
+from annet.annlib.netdev.views.dump import DumpableView
+from annet.storage import Query
+from dataclasses import dataclass, fields
+from typing import List, Iterable, Optional, Any
+from annet.storage import StorageProvider, Storage
+from annet.connectors import AdapterWithName
+from annet.storage import Device as DeviceCls
+from annet.annlib.netdev.views.hardware import vendor_to_hw, HardwareView
+import yaml
+
+
+@dataclass
+class Interface(DumpableView):
+    name: str
+    description: str
+    enabled: bool = True
+
+    @property
+    def _dump__list_key(self):
+        return self.name
+
+
+@dataclass
+class DeviceStorage:
+    fqdn: str
+    vendor: str
+    hostname: Optional[str] = None
+    serial: Optional[str] = None
+    id: Optional[str] = None
+    interfaces: Optional[list[Interface]] = None
+    storage: Optional[Storage] = None
+
+    def __post_init__(self):
+        if not self.id:
+            self.id = self.fqdn
+        if not self.hostname:
+            self.hostname = self.fqdn.split(".")[0]
+        hw = vendor_to_hw(self.vendor)
+        if not hw:
+            raise Exception("unknown vendor")
+        self.hw = hw
+        if isinstance(self.interfaces, list):
+            interfaces = []
+            for iface in self.interfaces:
+                try:
+                    interfaces.append(Interface(**iface))
+                except Exception as e:
+                    raise Exception("unable to parse %s as Interface %s" % (iface, e))
+            self.interfaces = interfaces
+
+    def set_storage(self, storage: Storage):
+        self.storage = storage
+
+
+@dataclass
+class Device(DeviceCls, DumpableView):
+    dev: DeviceStorage
+
+    @property
+    def hostname(self) -> str:
+        return self.dev.hostname
+
+    @property
+    def fqdn(self) -> str:
+        return self.dev.fqdn
+
+    @property
+    def id(self):
+        return self.dev.id
+
+    def __hash__(self):
+        return hash((self.id, type(self)))
+
+    def __eq__(self, other):
+        return type(self) is type(other) and self.fqdn == other.fqdn and self.vendor == other.vendor
+
+    def is_pc(self) -> bool:
+        return False
+
+    @property
+    def storage(self) -> Storage:
+        return self
+
+    @property
+    def hw(self) -> HardwareView:
+        return self.dev.hw
+
+    @property
+    def breed(self) -> str:
+        return self.dev.hw.vendor
+
+    @property
+    def neighbours_ids(self):
+        pass
+
+
+@dataclass
+class Devices:
+    devices: list[Device]
+
+    def __post_init__(self):
+        if isinstance(self.devices, list):
+            devices = []
+            for dev in self.devices:
+                try:
+                    devices.append(Device(dev=DeviceStorage(**dev)))
+                except Exception as e:
+                    raise Exception("unable to parse %s as Device %s" % (dev, e))
+            self.devices = devices
+
+
+class Provider(StorageProvider, AdapterWithName):
+    def storage(self):
+        return storage_factory
+
+    def opts(self):
+        return StorageOpts
+
+    def query(self):
+        return Query
+
+    @classmethod
+    def name(cls) -> str:
+        return "file"
+
+
+@dataclass
+class Query(Query):
+    query: List[str]
+
+    @classmethod
+    def new(cls, query: str | Iterable[str], hosts_range: Optional[slice] = None) -> "Query":
+        if hosts_range is not None:
+            raise ValueError("host_range is not supported")
+        return cls(query=list(query))
+
+    @property
+    def globs(self):
+        return self.query
+
+    def is_empty(self) -> bool:
+        return len(self.query) == 0
+
+
+class StorageOpts:
+    def __init__(self, path: str):
+        self.path = path
+
+    @classmethod
+    def parse_params(cls, conf_params: Optional[dict[str, str]], cli_opts: Any):
+        path = conf_params.get("path")
+        if not path:
+            raise Exception("empty path")
+        return cls(path=path)
+
+
+def storage_factory(opts: StorageOpts) -> Storage:
+    return FS(opts)
+
+
+class FS(Storage):
+    def __init__(self, opts: StorageOpts):
+        self.opts = opts
+        self.inventory: Devices = read_inventory(opts.path, self)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, _, __, ___):
+        pass
+
+    def resolve_object_ids_by_query(self, query: Query) -> list[str]:
+        result = filter_query(self.inventory.devices, query)
+        return [dev.fqdn for dev in result]
+
+    def resolve_fdnds_by_query(self, query: Query) -> list[str]:
+        result = filter_query(self.inventory.devices, query)
+        return [dev.fqdn for dev in result]
+
+    def make_devices(
+            self,
+            query: Query | list,
+            preload_neighbors=False,
+            use_mesh=None,
+            preload_extra_fields=False,
+            **kwargs,
+    ) -> list[Device]:
+        if isinstance(query, list):
+            query = Query.new(query)
+        result = filter_query(self.inventory.devices, query)
+        return result
+
+    def get_device(self, obj_id: str, preload_neighbors=False, use_mesh=None, **kwargs) -> Device:
+        result = filter_query(self.inventory.devices, Query.new(obj_id))
+        if not result:
+            raise Exception("not found")
+        return result[0]
+
+    def flush_perf(self):
+        pass
+
+
+def filter_query(devices: list[Device], query: Query) -> list[Device]:
+    result: list[Device] = []
+    for dev in devices:
+        if dev.fqdn in query.query:
+            result.append(dev)
+    return result
+
+
+def read_inventory(path: str, storage: Storage) -> Devices:
+    with open(path, "r") as f:
+        data = f.read()
+    file_data = yaml.load(data, Loader=yaml.BaseLoader)
+    res = dataclass_from_dict(Devices, file_data)
+    for dev in res.devices:
+        dev.dev.set_storage(storage)
+    return res
+
+
+def dataclass_from_dict(klass: type, d: dict[str, Any]):
+    try:
+        fieldtypes = {f.name: f.type for f in fields(klass)}
+    except TypeError:
+        return d
+    return klass(**{f: dataclass_from_dict(fieldtypes[f], d[f]) for f in d})

annet-0.16.9/annet/adapters/netbox/common/models.py

@@ -0,0 +1,276 @@
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from ipaddress import ip_interface, IPv6Interface
+from typing import List, Optional, Any, Dict, Sequence, Callable
+
+from annet.annlib.netdev.views.dump import DumpableView
+from annet.annlib.netdev.views.hardware import HardwareView, lag_name, svi_name
+from annet.storage import Storage
+
+
+@dataclass
+class Entity(DumpableView):
+    id: int
+    name: str
+
+    @property
+    def _dump__list_key(self):
+        return self.name
+
+
+@dataclass
+class Label:
+    value: str
+    label: str
+
+
+@dataclass
+class IpFamily:
+    value: int
+    label: str
+
+
+@dataclass
+class DeviceType:
+    id: int
+    manufacturer: Entity
+    model: str
+
+
+@dataclass
+class DeviceIp(DumpableView):
+    id: int
+    display: str
+    address: str
+    family: int
+
+    @property
+    def _dump__list_key(self):
+        return self.address
+
+
+@dataclass
+class Prefix(DumpableView):
+    id: int
+    prefix: str
+    site: Optional[Entity]
+    vrf: Optional[Entity]
+    tenant: Optional[Entity]
+    vlan: Optional[Entity]
+    role: Optional[Entity]
+    status: Label
+    is_pool: bool
+    custom_fields: dict[str, Any]
+    created: datetime
+    last_updated: datetime
+    description: Optional[str] = ""
+
+    @property
+    def _dump__list_key(self):
+        return self.prefix
+
+
+@dataclass
+class IpAddress(DumpableView):
+    id: int
+    assigned_object_id: int
+    display: str
+    family: IpFamily
+    address: str
+    status: Label
+    tags: List[Entity]
+    created: datetime
+    last_updated: datetime
+    prefix: Optional[Prefix] = None
+    vrf: Optional[Entity] = None
+
+    @property
+    def _dump__list_key(self):
+        return self.address
+
+
+@dataclass
+class InterfaceConnectedEndpoint(Entity):
+    device: Entity
+
+
+@dataclass
+class InterfaceType:
+    value: str
+    label: str
+
+
+@dataclass
+class InterfaceMode:
+    value: str
+    label: str
+
+
+@dataclass
+class InterfaceVlan(Entity):
+    vid: int
+
+
+@dataclass
+class Interface(Entity):
+    device: Entity
+    enabled: bool
+    description: str
+    type: InterfaceType
+    connected_endpoints: Optional[list[InterfaceConnectedEndpoint]]
+    mode: Optional[InterfaceMode]
+    untagged_vlan: Optional[InterfaceVlan]
+    tagged_vlans: Optional[List[InterfaceVlan]]
+    display: str = ""
+    ip_addresses: List[IpAddress] = field(default_factory=list)
+    vrf: Optional[Entity] = None
+    mtu: int | None = None
+    lag: Entity | None = None
+    lag_min_links: int | None = None
+
+    def add_addr(self, address_mask: str, vrf: str | None) -> None:
+        addr = ip_interface(address_mask)
+        if vrf is None:
+            vrf_obj = None
+        else:
+            vrf_obj = Entity(id=0, name=vrf)
+
+        if isinstance(addr, IPv6Interface):
+            family = IpFamily(value=6, label="IPv6")
+        else:
+            family = IpFamily(value=4, label="IPv4")
+        self.ip_addresses.append(IpAddress(
+            id=0,
+            display=address_mask,
+            address=address_mask,
+            vrf=vrf_obj,
+            prefix=None,
+            family=family,
+            created=datetime.now(timezone.utc),
+            last_updated=datetime.now(timezone.utc),
+            tags=[],
+            status=Label(value="active", label="Active"),
+            assigned_object_id=self.id,
+        ))
+
+
+@dataclass
+class NetboxDevice(Entity):
+    url: str
+    storage: Storage
+
+    display: str
+    device_type: DeviceType
+    device_role: Entity
+    tenant: Optional[Entity]
+    platform: Optional[Entity]
+    serial: str
+    asset_tag: Optional[str]
+    site: Entity
+    rack: Optional[Entity]
+    position: Optional[float]
+    face: Optional[Label]
+    status: Label
+    primary_ip: Optional[DeviceIp]
+    primary_ip4: Optional[DeviceIp]
+    primary_ip6: Optional[DeviceIp]
+    tags: List[Entity]
+    custom_fields: Dict[str, Any]
+    created: datetime
+    last_updated: datetime
+
+    fqdn: str
+    hostname: str
+    hw: Optional[HardwareView]
+    breed: str
+
+    interfaces: List[Interface]
+    neighbours: Optional[List["NetboxDevice"]]
+
+    # compat
+    @property
+    def neighbours_fqdns(self) -> list[str]:
+        if not self.neighbours:
+            return []
+        return [dev.fqdn for dev in self.neighbours]
+
+    @property
+    def neighbours_ids(self):
+        if not self.neighbours:
+            return []
+        return [dev.id for dev in self.neighbours]
+
+    def __hash__(self):
+        return hash((self.id, type(self)))
+
+    def __eq__(self, other):
+        return type(self) is type(other) and self.url == other.url
+
+    def is_pc(self) -> bool:
+        return self.device_type.manufacturer.name == "Mellanox"
+
+    def _make_interface(self, name: str, type: InterfaceType) -> Interface:
+        return Interface(
+            name=name,
+            device=self,
+            enabled=True,
+            description="",
+            type=type,
+            id=0,
+            vrf=None,
+            display=name,
+            untagged_vlan=None,
+            tagged_vlans=[],
+            ip_addresses=[],
+            connected_endpoints=[],
+            mode=None,
+        )
+
+    def _lag_name(self, lag: int) -> str:
+        return lag_name(self.hw, lag)
+
+    def make_lag(self, lag: int, ports: Sequence[str], lag_min_links: int | None) -> Interface:
+        new_name = self._lag_name(lag)
+        for target_interface in self.interfaces:
+            if target_interface.name == new_name:
+                return target_interface
+        lag_interface = self._make_interface(
+            name=new_name,
+            type=InterfaceType(value="lag", label="Link Aggregation Group (LAG)"),
+        )
+        lag_interface.lag_min_links = lag_min_links
+        for interface in self.interfaces:
+            if interface.name in ports:
+                interface.lag = lag_interface
+        self.interfaces.append(lag_interface)
+        return lag_interface
+
+    def _svi_name(self, svi: int) -> str:
+        return svi_name(self.hw, svi)
+
+    def add_svi(self, svi: int) -> Interface:
+        name = self._svi_name(svi)
+        for interface in self.interfaces:
+            if interface.name == name:
+                return interface
+        interface = self._make_interface(
+            name=name,
+            type=InterfaceType("virtual", "Virtual")
+        )
+        self.interfaces.append(interface)
+        return interface
+
+    def _subif_name(self, interface: str, subif: int) -> str:
+        return f"{interface}.{subif}"
+
+    def add_subif(self, interface: str, subif: int) -> Interface:
+        name = self._subif_name(interface, subif)
+        for target_port in self.interfaces:
+            if target_port.name == name:
+                return target_port
+        target_port = self._make_interface(
+            name=name,
+            type=InterfaceType("virtual", "Virtual")
+        )
+        self.interfaces.append(target_port)
+        return target_port