nectar-conformance 0.1.0__tar.gz

nectar_conformance-0.1.0/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: nectar-conformance
+Version: 0.1.0
+Summary: Conformance checker for Nectar puppet-managed OpenStack cloud sites
+Author: ARDC Nectar Core Services
+License: Apache-2.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.24
+Requires-Dist: PyYAML>=6
+Requires-Dist: jsonschema>=4
+Requires-Dist: packaging>=21
+Requires-Dist: cliff>=4
+Requires-Dist: rich>=13
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Requires-Dist: pytest-cov>=4; extra == "test"
+Requires-Dist: respx>=0.20; extra == "test"
+Requires-Dist: fastapi>=0.110; extra == "test"
+Requires-Dist: uvicorn>=0.27; extra == "test"
+Provides-Extra: static
+Provides-Extra: web
+Requires-Dist: fastapi>=0.110; extra == "web"
+Requires-Dist: uvicorn[standard]>=0.27; extra == "web"
+
+# nectar-conformance
+
+A conformance checker for Nectar puppet-managed OpenStack cloud sites.
+
+Nectar Core Services manage every site through a central puppet server. Sites drift
+from the published specification over time. This tool tells a site operator how their
+site conforms to a versioned Nectar conformance specification, what is wrong or
+missing, and how to fix it.
+
+## How it works
+
+```
+DataSource -> normalised Model -> Engine (versioned rules) -> Report -> human / JSON
+```
+
+- A **site is a puppet environment**. The tool identifies a site by the node's
+  `environment` in PuppetDB.
+- The **PuppetDB** data source (primary) reads the compiled catalog (resources and
+  parameters, with hiera fully resolved) and facts for every node in the site's
+  environment. The **static repo** data source reads pre-compiled catalog JSON, or
+  compiles catalogs from a site repo, for pre-deployment and commissioning checks.
+- **Checks** are curated by Core Services as value-free *definitions* (the logic) plus
+  per-version *manifests* (which checks apply and their expected values). Conformance
+  is versioned; several conformance versions are active at once.
+
+## Usage
+
+The check definitions and conformance changelog live in the separate
+[`nectar-conformance-checks`](https://review.rc.nectar.org.au) repository, not in this
+tool. Every command that reads them needs a checks directory: pass `--checks-dir <path>`,
+set `NECTAR_CONFORMANCE_CHECKS_DIR`, or set `checks_dir` in the config file. The examples
+below omit it for brevity.
+
+```
+nectar-conformance check run --site ardctest --conformance-version 2025.1
+nectar-conformance check run --site ardctest --conformance-version 2025.1 --format json
+nectar-conformance check list --conformance-version 2025.1
+nectar-conformance check show glance.api.image_tag
+nectar-conformance version list
+nectar-conformance version diff 2024.1 2025.1
+```
+
+Static source (no live PuppetDB), either pre-compiled catalogs or compile-from-repo:
+
+```
+# pre-compiled catalog JSON, one file per node
+nectar-conformance check run --site ardctest --source static \
+  --catalog-dir ./catalogs --facts-dir ./facts
+
+# compile from the site repo (one node per facts file; compiler set via
+# static.compile_command in config, defaults to octocatalog-diff)
+nectar-conformance check run --site ardctest --source static \
+  --site-repo /path/to/site-repo --facts-dir ./facts
+```
+
+### Will a change fix conformance before it goes live?
+
+A site is a puppet environment, and r10k deploys each branch of the control repo as its
+own environment, so you can check a *proposed* change pre-merge and compare it to the
+live site. Capture the live baseline, capture the proposed environment, and diff them:
+
+```
+# baseline: the live site
+nectar-conformance check run --site ardctest --conformance-version 2025.1 \
+  --format json > before.json
+
+# the proposed change, deployed as a branch environment (--environment keeps the
+# 'ardctest' label but queries that environment)
+nectar-conformance check run --site ardctest --environment ardctest_fix_glance \
+  --conformance-version 2025.1 --format json > after.json
+
+# what does the change fix, and does it break anything? (exit 1 if it regresses)
+nectar-conformance report diff before.json after.json
+```
+
+For a truly offline check (no deploy, no node runs), produce `after.json` with the
+static source by compiling the proposed branch (`--source static --site-repo ...`); that
+needs the environment's modules assembled, which is what octocatalog-diff / r10k do.
+
+Exit codes: `0` conformant, `1` conformance failure at or above the severity
+threshold, `2` usage error, `3` operational error (PuppetDB unreachable, etc.).
+`report diff` exits `1` if the change introduces any new failure.
+
+## Development
+
+```
+tox            # run unit tests and lint
+tox -e pep8    # lint only
+```
+
+Releases use [reno](https://docs.openstack.org/reno/) for release notes. Contributions
+go through gerrit; use conventional commits and `git commit -s`.

nectar_conformance-0.1.0/README.md

@@ -0,0 +1,92 @@
+# nectar-conformance
+
+A conformance checker for Nectar puppet-managed OpenStack cloud sites.
+
+Nectar Core Services manage every site through a central puppet server. Sites drift
+from the published specification over time. This tool tells a site operator how their
+site conforms to a versioned Nectar conformance specification, what is wrong or
+missing, and how to fix it.
+
+## How it works
+
+```
+DataSource -> normalised Model -> Engine (versioned rules) -> Report -> human / JSON
+```
+
+- A **site is a puppet environment**. The tool identifies a site by the node's
+  `environment` in PuppetDB.
+- The **PuppetDB** data source (primary) reads the compiled catalog (resources and
+  parameters, with hiera fully resolved) and facts for every node in the site's
+  environment. The **static repo** data source reads pre-compiled catalog JSON, or
+  compiles catalogs from a site repo, for pre-deployment and commissioning checks.
+- **Checks** are curated by Core Services as value-free *definitions* (the logic) plus
+  per-version *manifests* (which checks apply and their expected values). Conformance
+  is versioned; several conformance versions are active at once.
+
+## Usage
+
+The check definitions and conformance changelog live in the separate
+[`nectar-conformance-checks`](https://review.rc.nectar.org.au) repository, not in this
+tool. Every command that reads them needs a checks directory: pass `--checks-dir <path>`,
+set `NECTAR_CONFORMANCE_CHECKS_DIR`, or set `checks_dir` in the config file. The examples
+below omit it for brevity.
+
+```
+nectar-conformance check run --site ardctest --conformance-version 2025.1
+nectar-conformance check run --site ardctest --conformance-version 2025.1 --format json
+nectar-conformance check list --conformance-version 2025.1
+nectar-conformance check show glance.api.image_tag
+nectar-conformance version list
+nectar-conformance version diff 2024.1 2025.1
+```
+
+Static source (no live PuppetDB), either pre-compiled catalogs or compile-from-repo:
+
+```
+# pre-compiled catalog JSON, one file per node
+nectar-conformance check run --site ardctest --source static \
+  --catalog-dir ./catalogs --facts-dir ./facts
+
+# compile from the site repo (one node per facts file; compiler set via
+# static.compile_command in config, defaults to octocatalog-diff)
+nectar-conformance check run --site ardctest --source static \
+  --site-repo /path/to/site-repo --facts-dir ./facts
+```
+
+### Will a change fix conformance before it goes live?
+
+A site is a puppet environment, and r10k deploys each branch of the control repo as its
+own environment, so you can check a *proposed* change pre-merge and compare it to the
+live site. Capture the live baseline, capture the proposed environment, and diff them:
+
+```
+# baseline: the live site
+nectar-conformance check run --site ardctest --conformance-version 2025.1 \
+  --format json > before.json
+
+# the proposed change, deployed as a branch environment (--environment keeps the
+# 'ardctest' label but queries that environment)
+nectar-conformance check run --site ardctest --environment ardctest_fix_glance \
+  --conformance-version 2025.1 --format json > after.json
+
+# what does the change fix, and does it break anything? (exit 1 if it regresses)
+nectar-conformance report diff before.json after.json
+```
+
+For a truly offline check (no deploy, no node runs), produce `after.json` with the
+static source by compiling the proposed branch (`--source static --site-repo ...`); that
+needs the environment's modules assembled, which is what octocatalog-diff / r10k do.
+
+Exit codes: `0` conformant, `1` conformance failure at or above the severity
+threshold, `2` usage error, `3` operational error (PuppetDB unreachable, etc.).
+`report diff` exits `1` if the change introduces any new failure.
+
+## Development
+
+```
+tox            # run unit tests and lint
+tox -e pep8    # lint only
+```
+
+Releases use [reno](https://docs.openstack.org/reno/) for release notes. Contributions
+go through gerrit; use conventional commits and `git commit -s`.

nectar_conformance-0.1.0/nectar_conformance/__init__.py

@@ -0,0 +1,3 @@
+"""Conformance checker for Nectar puppet-managed OpenStack cloud sites."""
+
+__version__ = "0.1.0"

nectar_conformance-0.1.0/nectar_conformance/cli/__init__.py

@@ -0,0 +1 @@
+"""Operator-facing command-line interface (cliff)."""

nectar_conformance-0.1.0/nectar_conformance/cli/commands/__init__.py

@@ -0,0 +1 @@
+"""cliff command implementations."""

nectar_conformance-0.1.0/nectar_conformance/cli/commands/changelog_cmd.py

@@ -0,0 +1,46 @@
+"""``changelog lint`` - structurally validate a checks directory.
+
+The checks data lives in its own repository (``nectar-conformance-checks``); this is the
+CI gate that repository runs to catch a malformed changelog or definition before merge.
+It loads the definitions and changelog, runs :func:`changelog_lint`, and reports any
+structural violations (unknown check ids, ``effective`` after ``due``, test due later
+than prod, colliding entries).
+"""
+
+from __future__ import annotations
+
+from cliff.command import Command
+
+from nectar_conformance import config as config_mod
+from nectar_conformance.errors import ConformanceError
+from nectar_conformance.service import lint_versions
+
+
+class ChangelogLint(Command):
+    """Validate the conformance changelog and definitions are structurally sound."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument("--config", help="path to a config file")
+        parser.add_argument("--checks-dir", help="checks dir to lint")
+        return parser
+
+    def take_action(self, parsed_args):
+        overrides = (
+            {"checks_dir": parsed_args.checks_dir}
+            if parsed_args.checks_dir
+            else None
+        )
+        cfg = config_mod.load(parsed_args.config, overrides)
+        try:
+            violations = lint_versions(cfg)
+        except ConformanceError as exc:
+            self.app.stderr.write(f"error: {exc}\n")
+            return 3
+        if violations:
+            self.app.stderr.write("changelog lint found problems:\n")
+            for v in violations:
+                self.app.stderr.write(f"  - {v}\n")
+            return 1
+        self.app.stdout.write("changelog lint: ok\n")
+        return 0

nectar_conformance-0.1.0/nectar_conformance/cli/commands/check_meta.py

@@ -0,0 +1,112 @@
+"""``check list`` and ``check show`` - inspect checks without touching PuppetDB."""
+
+from __future__ import annotations
+
+from cliff.command import Command
+
+from nectar_conformance import config as config_mod
+from nectar_conformance.errors import ConformanceError
+from nectar_conformance.service import get_check, list_checks
+
+
+def _resolve_version(parsed_args, cfg):
+    return parsed_args.conformance_version or cfg.default_conformance_version
+
+
+class CheckList(Command):
+    """List the checks that apply at a conformance version."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument(
+            "--conformance-version", help="conformance version, e.g. 2026.1"
+        )
+        parser.add_argument("--config", help="path to a config file")
+        parser.add_argument("--checks-dir", help="load checks from this dir")
+        parser.add_argument(
+            "--severity", choices=["info", "warning", "error"], help="filter"
+        )
+        return parser
+
+    def take_action(self, parsed_args):
+        overrides = (
+            {"checks_dir": parsed_args.checks_dir}
+            if parsed_args.checks_dir
+            else None
+        )
+        cfg = config_mod.load(parsed_args.config, overrides)
+        version = _resolve_version(parsed_args, cfg)
+        if not version:
+            self.app.stderr.write(
+                "error: no conformance version (use --conformance-version)\n"
+            )
+            return 2
+        try:
+            rules = list_checks(cfg, version)
+        except ConformanceError as exc:
+            self.app.stderr.write(f"error: {exc}\n")
+            return 3
+        out = self.app.stdout
+        out.write(f"Checks for conformance {version}:\n")
+        for rule in rules:
+            if parsed_args.severity and rule.severity != parsed_args.severity:
+                continue
+            expected = (
+                ""
+                if rule.expected is None
+                else f"  expected={rule.expected!r}"
+            )
+            section = rule.spec_section or "-"
+            out.write(
+                f"  [{rule.severity:7}] {rule.id:30} ({section}){expected}\n"
+            )
+        return 0
+
+
+class CheckShow(Command):
+    """Show the full definition of one check."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument(
+            "check_id", help="check id, e.g. glance.api.image_tag"
+        )
+        parser.add_argument("--config", help="path to a config file")
+        parser.add_argument("--checks-dir", help="load checks from this dir")
+        return parser
+
+    def take_action(self, parsed_args):
+        overrides = (
+            {"checks_dir": parsed_args.checks_dir}
+            if parsed_args.checks_dir
+            else None
+        )
+        cfg = config_mod.load(parsed_args.config, overrides)
+        check = get_check(cfg, parsed_args.check_id)
+        if check is None:
+            self.app.stderr.write(
+                f"error: no such check '{parsed_args.check_id}'\n"
+            )
+            return 3
+        out = self.app.stdout
+        out.write(f"{check.id}\n")
+        out.write(f"  title:        {check.title}\n")
+        out.write(f"  severity:     {check.severity}\n")
+        out.write(f"  spec_section: {check.spec_section}\n")
+        out.write(f"  kind:         {check.kind}\n")
+        out.write(
+            f"  selector:     {check.selector.type} {check.selector.params}\n"
+        )
+        if check.query is not None:
+            out.write(
+                f"  query:        {check.query.type} {check.query.params}\n"
+            )
+        if check.assertion_op is not None:
+            out.write(f"  assertion:    {check.assertion_op}\n")
+        if check.remediation is not None:
+            out.write(f"  remediation:  {check.remediation.guidance}\n")
+            if check.remediation.hiera_key:
+                out.write(
+                    f"                hiera_key={check.remediation.hiera_key}\n"
+                )
+        return 0

nectar_conformance-0.1.0/nectar_conformance/cli/commands/check_run.py

@@ -0,0 +1,136 @@
+"""``check run`` - evaluate a site against a conformance version."""
+
+from __future__ import annotations
+
+from cliff.command import Command
+
+from nectar_conformance import config as config_mod
+from nectar_conformance.errors import ConformanceError
+from nectar_conformance.report import human, json_report
+from nectar_conformance.results.model import Report, Severity
+from nectar_conformance.service import run_check
+
+_RANK = {Severity.INFO: 1, Severity.WARNING: 2, Severity.ERROR: 3}
+_THRESHOLD = {"info": 1, "warning": 2, "error": 3}
+
+# Process exit codes.
+EXIT_OK = 0
+EXIT_NONCONFORMANT = 1
+EXIT_OPERATIONAL = 3
+
+
+def exit_code(report: Report, threshold: str) -> int:
+    worst = report.worst_failing_severity()
+    if worst is None:
+        return EXIT_OK
+    return (
+        EXIT_NONCONFORMANT
+        if _RANK[worst] >= _THRESHOLD[threshold]
+        else EXIT_OK
+    )
+
+
+class CheckRun(Command):
+    """Run conformance checks for a site and report the results."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument(
+            "--site", required=True, help="site (puppet environment) to check"
+        )
+        parser.add_argument(
+            "--environment",
+            help="puppet environment to query instead of the site name; use to check a "
+            "proposed branch environment before it goes live (report is still labelled --site)",
+        )
+        parser.add_argument(
+            "--conformance-version",
+            help="conformance version tag to pin the evaluation date to (e.g. 2026.1); "
+            "omit for a live run against today",
+        )
+        parser.add_argument(
+            "--as-of",
+            help="evaluate as if today were this date (YYYY-MM-DD), for scheduled and "
+            "what-if runs; overrides the version tag's pinned date",
+        )
+        parser.add_argument(
+            "--site-tier",
+            choices=["test", "prod"],
+            help="treat the site as this tier for dated changes (overrides config; "
+            "default prod)",
+        )
+        parser.add_argument(
+            "--source", choices=["puppetdb", "static"], help="data source"
+        )
+        parser.add_argument(
+            "--format", choices=["human", "json"], default="human"
+        )
+        parser.add_argument(
+            "--severity-threshold",
+            choices=["info", "warning", "error"],
+            default="error",
+            help="lowest severity that makes the run exit non-zero (default: error)",
+        )
+        parser.add_argument("--config", help="path to a config file")
+        parser.add_argument(
+            "--puppetdb-url", help="override the PuppetDB base URL"
+        )
+        parser.add_argument(
+            "--checks-dir",
+            help="load check definitions/manifests from this dir",
+        )
+        parser.add_argument(
+            "--site-repo", help="path to the site puppet repo (static source)"
+        )
+        parser.add_argument(
+            "--catalog-dir",
+            help="dir of compiled catalog JSON (static source)",
+        )
+        parser.add_argument(
+            "--facts-dir", help="dir of per-node facts JSON (static source)"
+        )
+        return parser
+
+    def take_action(self, parsed_args):
+        overrides: dict = {}
+        if parsed_args.puppetdb_url:
+            overrides.setdefault("puppetdb", {})["base_url"] = (
+                parsed_args.puppetdb_url
+            )
+        if parsed_args.source:
+            overrides["source"] = parsed_args.source
+        if parsed_args.checks_dir:
+            overrides["checks_dir"] = parsed_args.checks_dir
+
+        try:
+            cfg = config_mod.load(parsed_args.config, overrides)
+            if parsed_args.environment:
+                # Query this environment but keep the site label/identity.
+                cfg.site_environment[parsed_args.site] = (
+                    parsed_args.environment
+                )
+            if parsed_args.site_tier:
+                # Override the site's tier for this run (mirrors --environment).
+                cfg.site_tier[parsed_args.site] = parsed_args.site_tier
+            source_kwargs = {
+                "site_repo": parsed_args.site_repo,
+                "catalog_dir": parsed_args.catalog_dir,
+                "facts_dir": parsed_args.facts_dir,
+            }
+            report = run_check(
+                cfg,
+                site=parsed_args.site,
+                version=parsed_args.conformance_version,
+                source=parsed_args.source,
+                source_kwargs=source_kwargs,
+                as_of=parsed_args.as_of,
+            )
+        except ConformanceError as exc:
+            self.app.stderr.write(f"error: {exc}\n")
+            return EXIT_OPERATIONAL
+
+        if parsed_args.format == "json":
+            json_report.render(report, self.app.stdout)
+        else:
+            human.render(report, self.app.stdout)
+        return exit_code(report, parsed_args.severity_threshold)

nectar_conformance-0.1.0/nectar_conformance/cli/commands/diff.py

@@ -0,0 +1,50 @@
+"""``version diff`` - show how the check set differs between two versions."""
+
+from __future__ import annotations
+
+from cliff.command import Command
+
+from nectar_conformance import config as config_mod
+from nectar_conformance.errors import ConformanceError
+from nectar_conformance.service import diff_versions
+
+
+class DiffVersions(Command):
+    """Show checks added and removed between two conformance versions."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument("version_a", help="the 'from' conformance version")
+        parser.add_argument("version_b", help="the 'to' conformance version")
+        parser.add_argument("--config", help="path to a config file")
+        parser.add_argument("--checks-dir", help="load checks from this dir")
+        return parser
+
+    def take_action(self, parsed_args):
+        overrides = (
+            {"checks_dir": parsed_args.checks_dir}
+            if parsed_args.checks_dir
+            else None
+        )
+        cfg = config_mod.load(parsed_args.config, overrides)
+        try:
+            diff = diff_versions(
+                cfg, parsed_args.version_a, parsed_args.version_b
+            )
+        except ConformanceError as exc:
+            self.app.stderr.write(f"error: {exc}\n")
+            return 3
+        out = self.app.stdout
+        out.write(f"{diff['from']} -> {diff['to']}\n")
+        out.write(f"  changed ({len(diff['changed'])}):\n")
+        for change in diff["changed"]:
+            out.write(
+                f"    ~ {change['check_id']}: {change['from']} -> {change['to']}\n"
+            )
+        out.write(f"  added ({len(diff['added'])}):\n")
+        for cid in diff["added"]:
+            out.write(f"    + {cid}\n")
+        out.write(f"  removed ({len(diff['removed'])}):\n")
+        for cid in diff["removed"]:
+            out.write(f"    - {cid}\n")
+        return 0

nectar_conformance-0.1.0/nectar_conformance/cli/commands/report_diff.py

@@ -0,0 +1,52 @@
+"""``report diff`` - compare two conformance report JSON files (before vs after)."""
+
+from __future__ import annotations
+
+import json
+
+from cliff.command import Command
+
+from nectar_conformance.results.compare import compare_reports
+
+
+class ReportDiff(Command):
+    """Diff two 'check run --format json' reports to see what a change fixes or breaks."""
+
+    def get_parser(self, prog_name):
+        parser = super().get_parser(prog_name)
+        parser.add_argument("old", help="baseline report JSON (current/live)")
+        parser.add_argument("new", help="proposed report JSON (the change)")
+        return parser
+
+    def take_action(self, parsed_args):
+        try:
+            with open(parsed_args.old) as fh:
+                old = json.load(fh)
+            with open(parsed_args.new) as fh:
+                new = json.load(fh)
+        except (OSError, json.JSONDecodeError) as exc:
+            self.app.stderr.write(f"error: could not read report: {exc}\n")
+            return 3
+
+        diff = compare_reports(old, new)
+        out = self.app.stdout
+
+        def section(label, rows):
+            out.write(f"{label} ({len(rows)}):\n")
+            for r in rows:
+                arrow = f"{r['old']} -> {r['new']}"
+                out.write(f"  {r['rule_id']} [{r['severity']}]  {arrow}\n")
+
+        section("Fixed", diff["fixed"])
+        section("Regressed", diff["regressed"])
+        section("Still failing", diff["still_failing"])
+        if diff["added"] or diff["removed"]:
+            section("Added checks", diff["added"])
+            section("Removed checks", diff["removed"])
+
+        out.write(
+            f"\nSummary: {len(diff['fixed'])} fixed, {len(diff['regressed'])} regressed, "
+            f"{len(diff['still_failing'])} still failing\n"
+        )
+        # Non-zero exit if the change introduces any new failure, so CI can gate on it.
+        return 1 if diff["regressed"] else 0