greengate 0.1.0__tar.gz

greengate-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tharusha Rasath Hemachandra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

greengate-0.1.0/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.4
+Name: greengate
+Version: 0.1.0
+Summary: Confidence-aware cascading for green AI inference: route easy queries to a small local model, escalate only when uncertain, and account for the carbon honestly
+Author: Tharusha Rasath Hemachandra
+License: MIT
+Project-URL: Homepage, https://github.com/Rasath16/GreenGate
+Project-URL: Repository, https://github.com/Rasath16/GreenGate
+Project-URL: Issues, https://github.com/Rasath16/GreenGate/issues
+Keywords: green-ai,llm,carbon,sustainability,inference,model-cascade,routing,energy-efficiency,uncertainty
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Requires-Dist: transformers>=4.40
+Provides-Extra: gpu
+Requires-Dist: pynvml>=11.5; extra == "gpu"
+Requires-Dist: bitsandbytes>=0.43; extra == "gpu"
+Provides-Extra: api
+Requires-Dist: ecologits; extra == "api"
+Requires-Dist: openai; extra == "api"
+Provides-Extra: vision
+Requires-Dist: torchvision; extra == "vision"
+Requires-Dist: pillow; extra == "vision"
+Provides-Extra: eval
+Requires-Dist: pandas; extra == "eval"
+Requires-Dist: numpy; extra == "eval"
+Requires-Dist: matplotlib; extra == "eval"
+Requires-Dist: datasets; extra == "eval"
+Dynamic: license-file
+
+# GreenGate
+
+**Cut the cost and carbon of LLM inference with one line of routing.** Easy queries are answered by a small local model; only genuinely hard ones escalate to a large model. Every query is carbon-accounted, honestly.
+
+```bash
+pip install greengate
+```
+
+```python
+import greengate
+
+gw = greengate.GreenGate(
+    small="Qwen/Qwen2.5-0.5B-Instruct",   # runs locally, exposes logits
+    large="gpt-4o-mini",                   # or any local model
+    budget_g=0.5,                          # optional carbon ceiling
+)
+
+r = gw.route("Summarise this document ...")
+print(r.response)     # answered by whichever tier was appropriate
+print(r.decision)     # "LOCAL" or "ESCALATE"
+print(r.carbon_g)     # gCO2 for this query, including any wasted small run
+
+gw.profile()          # session totals: carbon, escalation rate, latency
+```
+
+## Why this exists
+
+Today every query, easy or hard, is sent to the same large model. Most queries do not need it. GreenGate sits between your application and your models, measures how uncertain the small model is, and escalates only when that uncertainty is high.
+
+Three things it does that other cascading systems do not:
+
+1. **Measures real energy.** Local inference is metered with NVML at 100 ms across all GPUs, not estimated. API tiers use [EcoLogits](https://ecologits.ai/) and are labelled as estimates.
+2. **Full carbon accounting.** When a query escalates, it is charged for *both* the discarded small-model run and the large-model run. Most published cascade savings omit the first, which overstates them.
+3. **Routes vision queries too**, using the average token probability of the generated answer as the confidence signal.
+
+## What to expect
+
+From the evaluation in the accompanying study (ShareGPT, MMLU and VQAv2; measured on dual T4):
+
+| Workload | Carbon reduction | Quality retention |
+|---|---|---|
+| MMLU (structured) | 29.5% | 95.0% |
+| ShareGPT (open-ended) | 49.8% | 80.6% |
+| ShareGPT (quality-first) | 14.5% | 90.1% |
+| VQAv2 (vision) | escalates only 5% | exceeds both tiers |
+
+Savings depend on four measurable things: the energy ratio between your tiers, your escalation rate, the retention you accept, and your hardware. GreenGate reports all of them rather than assuming them. On an inefficient local GPU, escalating to an efficient API can genuinely be greener — the profiler tells you which case you are in.
+
+## Tiers
+
+- **Small tier** must be a local open-weight model. The routing signal is computed from token logits, which hosted APIs do not expose. This is also where the privacy and cost win comes from.
+- **Large tier** can be anything: another local model, or an OpenAI API model.
+
+## Calibration
+
+Language models are overconfident, so raw entropy thresholds are unreliable. GreenGate ships fitted temperature-scaling values for evaluated models and self-calibrates for anything else:
+
+```python
+gw.calibrate()   # ~15 min, fits T on held-out MMLU validation, saved to ~/.greengate/
+```
+
+## Modes and budgets
+
+```python
+gw.config(mode="green")      # escalate less; "balanced" and "quality" also available
+gw.config(threshold=2.9)     # or set the entropy threshold explicitly
+gw.config(budget_g=0.05)     # sliding-window carbon ceiling; escalation defers when exhausted
+```
+
+## Honest limitations
+
+- On open-ended generation with small models, token entropy is a weak signal (near chance in our evaluation). It is informative on structured tasks and vision. Where it is weak, savings come from the cascade structure rather than from selective routing.
+- Energy measurement requires an NVIDIA GPU (NVML). CPU runs fall back to a documented estimate.
+- API-tier carbon is an estimate, not a measurement, and is not directly comparable to metered local figures.
+
+## Install extras
+
+```bash
+pip install greengate[gpu]    # bitsandbytes + pynvml for quantised local models and metering
+pip install greengate[api]    # openai + ecologits for API large tiers
+pip install greengate[eval]   # pandas/matplotlib for the evaluation scripts
+```
+
+## Reproducing the evaluation
+
+The `RUNBOOK.md` in this repository reproduces every published number: calibration, three deployment configurations, four baselines, threshold sweeps, grid conditions, trace replay against real Azure arrival traces, and three ablation studies. Raw per-query records for all runs are in `experiments/`.
+
+## Citation
+
+If you use GreenGate in academic work, please cite the accompanying study:
+
+> T. R. Hemachandra, "GreenGate: A Confidence-Aware Cascading Framework for Optimizing Energy and Cost in Large Language and Multimodal Model Inference," BSc thesis, NSBM Green University, 2026.
+
+## License
+
+MIT

greengate-0.1.0/README.md

@@ -0,0 +1,96 @@
+# GreenGate
+
+**Cut the cost and carbon of LLM inference with one line of routing.** Easy queries are answered by a small local model; only genuinely hard ones escalate to a large model. Every query is carbon-accounted, honestly.
+
+```bash
+pip install greengate
+```
+
+```python
+import greengate
+
+gw = greengate.GreenGate(
+    small="Qwen/Qwen2.5-0.5B-Instruct",   # runs locally, exposes logits
+    large="gpt-4o-mini",                   # or any local model
+    budget_g=0.5,                          # optional carbon ceiling
+)
+
+r = gw.route("Summarise this document ...")
+print(r.response)     # answered by whichever tier was appropriate
+print(r.decision)     # "LOCAL" or "ESCALATE"
+print(r.carbon_g)     # gCO2 for this query, including any wasted small run
+
+gw.profile()          # session totals: carbon, escalation rate, latency
+```
+
+## Why this exists
+
+Today every query, easy or hard, is sent to the same large model. Most queries do not need it. GreenGate sits between your application and your models, measures how uncertain the small model is, and escalates only when that uncertainty is high.
+
+Three things it does that other cascading systems do not:
+
+1. **Measures real energy.** Local inference is metered with NVML at 100 ms across all GPUs, not estimated. API tiers use [EcoLogits](https://ecologits.ai/) and are labelled as estimates.
+2. **Full carbon accounting.** When a query escalates, it is charged for *both* the discarded small-model run and the large-model run. Most published cascade savings omit the first, which overstates them.
+3. **Routes vision queries too**, using the average token probability of the generated answer as the confidence signal.
+
+## What to expect
+
+From the evaluation in the accompanying study (ShareGPT, MMLU and VQAv2; measured on dual T4):
+
+| Workload | Carbon reduction | Quality retention |
+|---|---|---|
+| MMLU (structured) | 29.5% | 95.0% |
+| ShareGPT (open-ended) | 49.8% | 80.6% |
+| ShareGPT (quality-first) | 14.5% | 90.1% |
+| VQAv2 (vision) | escalates only 5% | exceeds both tiers |
+
+Savings depend on four measurable things: the energy ratio between your tiers, your escalation rate, the retention you accept, and your hardware. GreenGate reports all of them rather than assuming them. On an inefficient local GPU, escalating to an efficient API can genuinely be greener — the profiler tells you which case you are in.
+
+## Tiers
+
+- **Small tier** must be a local open-weight model. The routing signal is computed from token logits, which hosted APIs do not expose. This is also where the privacy and cost win comes from.
+- **Large tier** can be anything: another local model, or an OpenAI API model.
+
+## Calibration
+
+Language models are overconfident, so raw entropy thresholds are unreliable. GreenGate ships fitted temperature-scaling values for evaluated models and self-calibrates for anything else:
+
+```python
+gw.calibrate()   # ~15 min, fits T on held-out MMLU validation, saved to ~/.greengate/
+```
+
+## Modes and budgets
+
+```python
+gw.config(mode="green")      # escalate less; "balanced" and "quality" also available
+gw.config(threshold=2.9)     # or set the entropy threshold explicitly
+gw.config(budget_g=0.05)     # sliding-window carbon ceiling; escalation defers when exhausted
+```
+
+## Honest limitations
+
+- On open-ended generation with small models, token entropy is a weak signal (near chance in our evaluation). It is informative on structured tasks and vision. Where it is weak, savings come from the cascade structure rather than from selective routing.
+- Energy measurement requires an NVIDIA GPU (NVML). CPU runs fall back to a documented estimate.
+- API-tier carbon is an estimate, not a measurement, and is not directly comparable to metered local figures.
+
+## Install extras
+
+```bash
+pip install greengate[gpu]    # bitsandbytes + pynvml for quantised local models and metering
+pip install greengate[api]    # openai + ecologits for API large tiers
+pip install greengate[eval]   # pandas/matplotlib for the evaluation scripts
+```
+
+## Reproducing the evaluation
+
+The `RUNBOOK.md` in this repository reproduces every published number: calibration, three deployment configurations, four baselines, threshold sweeps, grid conditions, trace replay against real Azure arrival traces, and three ablation studies. Raw per-query records for all runs are in `experiments/`.
+
+## Citation
+
+If you use GreenGate in academic work, please cite the accompanying study:
+
+> T. R. Hemachandra, "GreenGate: A Confidence-Aware Cascading Framework for Optimizing Energy and Cost in Large Language and Multimodal Model Inference," BSc thesis, NSBM Green University, 2026.
+
+## License
+
+MIT

greengate-0.1.0/greengate/__init__.py

@@ -0,0 +1,12 @@
+from greengate.core import GreenGate, RouteResult
+
+__version__ = "0.1.0"
+__all__ = ["GreenGate", "RouteResult"]
+
+
+def __getattr__(name):
+    # legacy demo class, loaded lazily to keep `import greengate` light
+    if name == "GreenGateRouter":
+        from greengate.router import GreenGateRouter
+        return GreenGateRouter
+    raise AttributeError(name)

greengate-0.1.0/greengate/api_tier.py

@@ -0,0 +1,153 @@
+"""Large-tier inference via OpenAI API with EcoLogits carbon estimation.
+
+Dual-Mode Carbon Profiler, API side: local models are measured with
+pynvml (profiler.py); API models are estimated with EcoLogits
+(Rince et al., 2025, JOSS). If EcoLogits is unavailable or fails,
+falls back to a documented per-token estimate and flags it.
+"""
+
+import os
+import time
+from dataclasses import dataclass
+
+# Fallback constants (used ONLY if EcoLogits fails) — documented in thesis:
+# Jegham et al. (2025) measure ~0.42 Wh for a mean GPT-4o query (~150 output
+# tokens). GPT-4o-mini is a distilled/smaller deployment; we conservatively
+# assume 0.5x the 4o figure, scaled linearly by output tokens.
+FALLBACK_WH_PER_OUTPUT_TOKEN = 0.42 * 0.5 / 150  # = 0.0014 Wh/token
+WORLD_CARBON_INTENSITY = 475.0  # gCO2/kWh
+DATACENTER_PUE = 1.2
+
+
+@dataclass
+class APIResult:
+    response: str
+    model: str
+    input_tokens: int
+    output_tokens: int
+    latency_s: float
+    energy_wh: float
+    carbon_grams: float
+    carbon_source: str  # "ecologits" or "fallback_estimate"
+    # confidence signals from top-20 logprobs (None unless logprobs=True):
+    avg_logprob: float | None = None      # mean log P(chosen token)
+    trunc_entropy: float | None = None    # mean entropy over renormalised top-20 (bits)
+
+
+class APILargeTier:
+    def __init__(self, model: str = "gpt-4o-mini", max_tokens: int = 300,
+                 dry_run: bool = False, logprobs: bool = False):
+        self.model = model
+        self.max_tokens = max_tokens
+        self.dry_run = dry_run
+        self.logprobs = logprobs
+        self._ecologits_ok = False
+
+        if dry_run:
+            return
+
+        # EcoLogits must be initialised BEFORE the OpenAI client is created.
+        try:
+            from ecologits import EcoLogits
+            EcoLogits.init(providers=["openai"])
+            self._ecologits_ok = True
+        except Exception as e:
+            print(f"  [api_tier] EcoLogits unavailable ({e}) — using documented fallback estimate")
+
+        from openai import OpenAI
+        self.client = OpenAI()  # reads OPENAI_API_KEY env var
+
+    def query_vision(self, prompt: str, image) -> APIResult:
+        """Vision escalation: PIL image sent as a base64 data URL."""
+        if self.dry_run:
+            return self.query(prompt)
+        import base64
+        import io
+        buf = io.BytesIO()
+        image.save(buf, format="JPEG", quality=85)
+        b64 = base64.b64encode(buf.getvalue()).decode()
+        content = [
+            {"type": "text", "text": prompt},
+            {"type": "image_url",
+             "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
+        ]
+        return self._chat(content)
+
+    def query(self, prompt: str) -> APIResult:
+        if self.dry_run:
+            return APIResult(
+                response="[DRY RUN — no API call made]",
+                model=self.model, input_tokens=len(prompt) // 4,
+                output_tokens=50, latency_s=0.5,
+                energy_wh=50 * FALLBACK_WH_PER_OUTPUT_TOKEN,
+                carbon_grams=50 * FALLBACK_WH_PER_OUTPUT_TOKEN / 1000
+                * DATACENTER_PUE * WORLD_CARBON_INTENSITY,
+                carbon_source="dry_run",
+            )
+
+        return self._chat(prompt)
+
+    def _chat(self, content, retries: int = 6) -> APIResult:
+        kwargs = {}
+        if self.logprobs:
+            kwargs = {"logprobs": True, "top_logprobs": 20}
+        from openai import RateLimitError, APIError
+        t0 = time.perf_counter()
+        for attempt in range(retries):
+            try:
+                resp = self.client.chat.completions.create(
+                    model=self.model,
+                    messages=[{"role": "user", "content": content}],
+                    max_tokens=self.max_tokens,
+                    **kwargs,
+                )
+                break
+            except (RateLimitError, APIError):
+                if attempt == retries - 1:
+                    raise
+                time.sleep(min(2 ** attempt, 30))  # 1,2,4,8,16,30s backoff
+        latency = time.perf_counter() - t0
+
+        avg_lp = trunc_H = None
+        lp_content = getattr(resp.choices[0].logprobs, "content", None) \
+            if self.logprobs and resp.choices[0].logprobs else None
+        if lp_content:
+            import math
+            lps, ents = [], []
+            for tok in lp_content:
+                lps.append(tok.logprob)
+                if tok.top_logprobs:
+                    ps = [math.exp(t.logprob) for t in tok.top_logprobs]
+                    z = sum(ps) or 1.0
+                    ents.append(-sum(p / z * math.log2(p / z) for p in ps if p > 0))
+            avg_lp = sum(lps) / len(lps) if lps else None
+            trunc_H = sum(ents) / len(ents) if ents else None
+
+        text = resp.choices[0].message.content or ""
+        in_tok = resp.usage.prompt_tokens
+        out_tok = resp.usage.completion_tokens
+
+        energy_wh, carbon_g, source = None, None, "fallback_estimate"
+        if self._ecologits_ok and hasattr(resp, "impacts"):
+            try:
+                # EcoLogits attaches .impacts; energy in kWh (value or range)
+                e = resp.impacts.energy.value
+                energy_kwh = (e.min + e.max) / 2 if hasattr(e, "min") else float(e)
+                g = resp.impacts.gwp.value
+                gwp_kg = (g.min + g.max) / 2 if hasattr(g, "min") else float(g)
+                energy_wh = energy_kwh * 1000
+                carbon_g = gwp_kg * 1000
+                source = "ecologits"
+            except Exception:
+                pass
+        if energy_wh is None:
+            energy_wh = out_tok * FALLBACK_WH_PER_OUTPUT_TOKEN
+            carbon_g = energy_wh / 1000 * DATACENTER_PUE * WORLD_CARBON_INTENSITY
+
+        return APIResult(
+            response=text.strip(), model=self.model,
+            input_tokens=in_tok, output_tokens=out_tok,
+            latency_s=latency, energy_wh=energy_wh,
+            carbon_grams=carbon_g, carbon_source=source,
+            avg_logprob=avg_lp, trunc_entropy=trunc_H,
+        )

greengate-0.1.0/greengate/budget.py

@@ -0,0 +1,31 @@
+"""Sliding-window carbon budget (Chapter 3, Design Analysis 2.5.3).
+
+Maintains a window of the last `window_s` seconds of routing decisions.
+If the carbon spent inside the window would exceed budget K, escalations
+are temporarily blocked (the query falls back to the small answer) until
+older queries age out of the window.
+"""
+
+from collections import deque
+
+
+class SlidingWindowBudget:
+    def __init__(self, budget_g: float, window_s: float = 3600.0):
+        self.budget_g = budget_g
+        self.window_s = window_s
+        self._events: deque[tuple[float, float]] = deque()  # (timestamp, carbon_g)
+
+    def _expire(self, now: float):
+        while self._events and self._events[0][0] < now - self.window_s:
+            self._events.popleft()
+
+    def window_carbon(self, now: float) -> float:
+        self._expire(now)
+        return sum(c for _, c in self._events)
+
+    def allows(self, now: float, escalation_cost_g: float) -> bool:
+        """Would escalating now keep the window under budget?"""
+        return self.window_carbon(now) + escalation_cost_g <= self.budget_g
+
+    def record(self, now: float, carbon_g: float):
+        self._events.append((now, carbon_g))

greengate-0.1.0/greengate/calibration.py

@@ -0,0 +1,53 @@
+"""Temperature scaling calibration (Guo et al., 2017).
+
+Fits a single scalar T on a held-out validation split by minimising
+negative log-likelihood. T > 1 softens overconfident distributions.
+Reports Expected Calibration Error (ECE) before and after.
+"""
+
+import torch
+import torch.nn.functional as F
+
+
+def fit_temperature(logits: torch.Tensor, labels: torch.Tensor,
+                    t_min: float = 0.25, t_max: float = 8.0,
+                    n_grid: int = 400) -> float:
+    """Fit T by fine grid search over NLL (robust, dependency-free).
+
+    Args:
+        logits: (n, k) raw choice logits from the validation split
+        labels: (n,) correct choice indices
+    """
+    best_t, best_nll = 1.0, float("inf")
+    for i in range(n_grid):
+        t = t_min + (t_max - t_min) * i / (n_grid - 1)
+        nll = F.cross_entropy(logits / t, labels).item()
+        if nll < best_nll:
+            best_nll, best_t = nll, t
+    return best_t
+
+
+def ece(logits: torch.Tensor, labels: torch.Tensor,
+        temperature: float = 1.0, n_bins: int = 15) -> float:
+    """Expected Calibration Error over equal-width confidence bins."""
+    probs = F.softmax(logits / temperature, dim=-1)
+    conf, pred = probs.max(dim=-1)
+    correct = (pred == labels).float()
+
+    total = 0.0
+    n = len(labels)
+    for b in range(n_bins):
+        lo, hi = b / n_bins, (b + 1) / n_bins
+        mask = (conf > lo) & (conf <= hi)
+        if mask.sum() == 0:
+            continue
+        avg_conf = conf[mask].mean().item()
+        avg_acc = correct[mask].mean().item()
+        total += (mask.sum().item() / n) * abs(avg_conf - avg_acc)
+    return total
+
+
+def calibrated_entropy(logits: torch.Tensor, temperature: float) -> float:
+    """Shannon entropy (bits) of the temperature-scaled distribution."""
+    probs = F.softmax(logits / temperature, dim=-1).clamp(min=1e-9)
+    return -(probs * probs.log2()).sum(dim=-1).item()