winnex-ai-normalize 1.0.0__tar.gz

winnex_ai_normalize-1.0.0/LICENSE

@@ -0,0 +1,16 @@
+Business Source License 1.1
+
+License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
+"Business Source License" is a trademark of MariaDB Corporation Ab.
+
+Parameters
+
+Licensor:             Winnex AI
+Licensed Work:        Tracer-GOV
+Additional Use Grant: Free for Brazilian government agencies.
+                      Any use by a public sector entity in Brazil is permitted.
+
+Change Date:          2036-01-01
+Change License:       GNU General Public License v2.0 or later
+
+For inquiries: pay@winnex.ai

winnex_ai_normalize-1.0.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: winnex-ai-normalize
+Version: 1.0.0
+Summary: Input normalization plug for the Madhava engine — embed text via any provider (OpenAI, Qwen3, nano, xfactor), validate/normalize to float32/uint8, with provider failover and NO fake-vector fallback. Consumed by winnex-madhava, winnex-tracer, Liferay and the Maestro.
+Author-email: Winnex Brasil Soluções Empresariais LTDA-ME <pay@winnex.ai>
+License: Business Source License 1.1
+        
+        License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
+        "Business Source License" is a trademark of MariaDB Corporation Ab.
+        
+        Parameters
+        
+        Licensor:             Winnex AI
+        Licensed Work:        Tracer-GOV
+        Additional Use Grant: Free for Brazilian government agencies.
+                              Any use by a public sector entity in Brazil is permitted.
+        
+        Change Date:          2036-01-01
+        Change License:       GNU General Public License v2.0 or later
+        
+        For inquiries: pay@winnex.ai
+        
+Project-URL: Homepage, https://winnex.ai
+Project-URL: Source, https://github.com/winnex-ai/winnex-ai-normalize
+Keywords: embedding,normalize,madhava,vector,openai,qwen3,rag,semantic-search,normalizer,winnex,failover
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: httpx>=0.24
+Provides-Extra: api
+Requires-Dist: fastapi>=0.100; extra == "api"
+Requires-Dist: uvicorn>=0.20; extra == "api"
+Provides-Extra: madhava
+Requires-Dist: winnex-madhava>=1.9.2; extra == "madhava"
+Provides-Extra: all
+Requires-Dist: fastapi>=0.100; extra == "all"
+Requires-Dist: uvicorn>=0.20; extra == "all"
+Requires-Dist: winnex-madhava>=1.9.2; extra == "all"
+Dynamic: license-file
+
+# winnex-ai-normalize
+
+**The input-normalization plug for the Madhava engine.**
+
+The Madhava engine is **agnostic** — it consumes float32 vectors. This
+package normalizes **any** input (text via an embedding provider, or raw
+vectors) into a valid float32/uint8 corpus ready for
+`winnex_madhava.build_engine`, with **provider failover** and **no silent
+fallback to fake vectors**.
+
+```
+text / raw vectors
+      │
+      ▼
+winnex-ai-normalize
+  ├── providers:  OpenAI / Qwen3 / nano / xfactor / direct (failover order)
+  ├── validate:   dim + NaN + norm (fail loudly)
+  ├── normalize:  L2 unit-norm (cosine contract)
+  └── quantize:   float32 → uint8 (the engine's native corpus)
+      │
+      ▼
+winnex-madhava (agnostic) → top-K + Cauchy-Schwarz proof
+```
+
+## Install
+
+```bash
+pip install winnex-ai-normalize               # core (numpy + httpx)
+pip install winnex-ai-normalize[api]          # + OpenAI-compatible /v1/embeddings
+pip install winnex-ai-normalize[all]          # + winnex-madhava
+```
+
+## Quick start
+
+```python
+from winnex_ai_normalize import EmbeddingNormalizer
+
+norm = EmbeddingNormalizer()                  # reads env / JSON config
+
+# 1. Embed text via the configured provider (OpenAI / Qwen3 / ...)
+vecs = norm.vectorize_texts(["hypertension treatment", "diabetes care"])
+# vecs: (2, d) float32 L2-normalized
+
+# 2. Or normalize raw vectors (validated, no NaN, unit-norm)
+vecs = norm.normalize_vectors(my_embeddings, dim=1024)
+
+# 3. Feed the Madhava engine (agnostic)
+u8 = norm.to_corpus(vecs, dim=1024)           # uint8 corpus for build_engine
+import winnex_madhava as wm
+engine = wm.build_engine(u8, dim=1024, metric="cosine", k=10)
+```
+
+## OpenAI-compatible API
+
+```bash
+uvicorn winnex_ai_normalize.api.server:app --port 8102
+curl -X POST http://localhost:8102/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{"model": "Qwen3-Embedding-0.6B", "input": ["text one", "text two"]}'
+# → {"data": [{"embedding": [...], "index": 0}, ...], ...}
+```
+
+## Provider failover (no fake fallback)
+
+Providers are tried in `provider_order` (JSON/env config). If **all**
+fail, a `RuntimeError` is raised with the collected errors — a normalizer
+must never fabricate embeddings.
+
+```json
+{
+  "default_dim": 1024,
+  "default_provider": "qwen3",
+  "providers": [
+    {"name": "qwen3", "base_url": "http://winnex-embedding:8102",
+     "model": "/workspace/models/Qwen3-Embedding-0.6B", "priority": 1},
+    {"name": "openai", "base_url": "https://api.openai.com/v1",
+     "model": "text-embedding-3-small", "api_key_env": "OPENAI_API_KEY", "priority": 2}
+  ],
+  "provider_order": ["qwen3", "openai"]
+}
+```
+
+## Consumed by
+
+- **winnex-madhava** (direct — the engine stays agnostic)
+- **winnex-tracer** (audit + commitment)
+- **Liferay bridges** (tracer-gov-liferay, tracer-med-liferay)
+- **Maestro** (winnex-ai-server / winnex-ai-engine) and any external tool
+
+## License
+
+Business Source License 1.1 (BSL 1.1) | pay@winnex.ai |
+Winnex Brasil Soluções Empresariais LTDA (CNPJ 58.364.637/0001-47)

winnex_ai_normalize-1.0.0/README.md

@@ -0,0 +1,93 @@
+# winnex-ai-normalize
+
+**The input-normalization plug for the Madhava engine.**
+
+The Madhava engine is **agnostic** — it consumes float32 vectors. This
+package normalizes **any** input (text via an embedding provider, or raw
+vectors) into a valid float32/uint8 corpus ready for
+`winnex_madhava.build_engine`, with **provider failover** and **no silent
+fallback to fake vectors**.
+
+```
+text / raw vectors
+      │
+      ▼
+winnex-ai-normalize
+  ├── providers:  OpenAI / Qwen3 / nano / xfactor / direct (failover order)
+  ├── validate:   dim + NaN + norm (fail loudly)
+  ├── normalize:  L2 unit-norm (cosine contract)
+  └── quantize:   float32 → uint8 (the engine's native corpus)
+      │
+      ▼
+winnex-madhava (agnostic) → top-K + Cauchy-Schwarz proof
+```
+
+## Install
+
+```bash
+pip install winnex-ai-normalize               # core (numpy + httpx)
+pip install winnex-ai-normalize[api]          # + OpenAI-compatible /v1/embeddings
+pip install winnex-ai-normalize[all]          # + winnex-madhava
+```
+
+## Quick start
+
+```python
+from winnex_ai_normalize import EmbeddingNormalizer
+
+norm = EmbeddingNormalizer()                  # reads env / JSON config
+
+# 1. Embed text via the configured provider (OpenAI / Qwen3 / ...)
+vecs = norm.vectorize_texts(["hypertension treatment", "diabetes care"])
+# vecs: (2, d) float32 L2-normalized
+
+# 2. Or normalize raw vectors (validated, no NaN, unit-norm)
+vecs = norm.normalize_vectors(my_embeddings, dim=1024)
+
+# 3. Feed the Madhava engine (agnostic)
+u8 = norm.to_corpus(vecs, dim=1024)           # uint8 corpus for build_engine
+import winnex_madhava as wm
+engine = wm.build_engine(u8, dim=1024, metric="cosine", k=10)
+```
+
+## OpenAI-compatible API
+
+```bash
+uvicorn winnex_ai_normalize.api.server:app --port 8102
+curl -X POST http://localhost:8102/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{"model": "Qwen3-Embedding-0.6B", "input": ["text one", "text two"]}'
+# → {"data": [{"embedding": [...], "index": 0}, ...], ...}
+```
+
+## Provider failover (no fake fallback)
+
+Providers are tried in `provider_order` (JSON/env config). If **all**
+fail, a `RuntimeError` is raised with the collected errors — a normalizer
+must never fabricate embeddings.
+
+```json
+{
+  "default_dim": 1024,
+  "default_provider": "qwen3",
+  "providers": [
+    {"name": "qwen3", "base_url": "http://winnex-embedding:8102",
+     "model": "/workspace/models/Qwen3-Embedding-0.6B", "priority": 1},
+    {"name": "openai", "base_url": "https://api.openai.com/v1",
+     "model": "text-embedding-3-small", "api_key_env": "OPENAI_API_KEY", "priority": 2}
+  ],
+  "provider_order": ["qwen3", "openai"]
+}
+```
+
+## Consumed by
+
+- **winnex-madhava** (direct — the engine stays agnostic)
+- **winnex-tracer** (audit + commitment)
+- **Liferay bridges** (tracer-gov-liferay, tracer-med-liferay)
+- **Maestro** (winnex-ai-server / winnex-ai-engine) and any external tool
+
+## License
+
+Business Source License 1.1 (BSL 1.1) | pay@winnex.ai |
+Winnex Brasil Soluções Empresariais LTDA (CNPJ 58.364.637/0001-47)

winnex_ai_normalize-1.0.0/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "winnex-ai-normalize"
+version = "1.0.0"
+description = "Input normalization plug for the Madhava engine — embed text via any provider (OpenAI, Qwen3, nano, xfactor), validate/normalize to float32/uint8, with provider failover and NO fake-vector fallback. Consumed by winnex-madhava, winnex-tracer, Liferay and the Maestro."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { file = "LICENSE" }
+authors = [
+    { name = "Winnex Brasil Soluções Empresariais LTDA-ME", email = "pay@winnex.ai" },
+]
+keywords = [
+    "embedding", "normalize", "madhava", "vector", "openai", "qwen3",
+    "rag", "semantic-search", "normalizer", "winnex", "failover",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: Other/Proprietary License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "numpy>=1.24",
+    "httpx>=0.24",
+]
+
+[project.optional-dependencies]
+api = ["fastapi>=0.100", "uvicorn>=0.20"]
+madhava = ["winnex-madhava>=1.9.2"]
+all = [
+    "fastapi>=0.100",
+    "uvicorn>=0.20",
+    "winnex-madhava>=1.9.2",
+]
+
+[project.urls]
+Homepage = "https://winnex.ai"
+Source = "https://github.com/winnex-ai/winnex-ai-normalize"
+
+[tool.pytest.ini_options]
+testpaths = ["winnex_ai_normalize/tests"]
+
+[tool.setuptools.packages.find]
+include = ["winnex_ai_normalize*"]

winnex_ai_normalize-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

winnex_ai_normalize-1.0.0/winnex_ai_normalize/__init__.py

@@ -0,0 +1,42 @@
+"""winnex-ai-normalize — the input-normalization plug for the Madhava engine.
+
+The Madhava engine is AGNOSTIC: it consumes float32 vectors. This package
+normalizes ANY input (text via an embedding provider, or raw vectors) into a
+valid float32/uint8 corpus ready for `winnex_madhava.build_engine`, with
+provider failover and NO silent fallback to fake vectors.
+
+Components:
+    - core.config:      JSON-driven provider/config (model, dim, failover order)
+    - core.embedding:   EmbeddingService with provider failover + cache
+    - core.normalize:   validate / L2-normalize / quantize → Madhava corpus
+    - api.server:       OpenAI-compatible /v1/embeddings endpoint
+
+Consumed by: winnex-madhava (direct), winnex-tracer, the Liferay bridges,
+and the Maestro — any tool that needs to feed vectors to Madhava.
+
+Business Source License 1.1 (BSL 1.1) | pay@winnex.ai
+"""
+from .core.config import NormalizeConfig, ProviderConfig, load_config
+from .core.embedding import EmbeddingProvider, EmbeddingService, get_embedding_service
+from .core.normalize import (
+    EmbeddingNormalizer,
+    validate_embeddings,
+    normalize_l2,
+    quantize_corpus,
+)
+
+__version__ = "1.0.0"
+
+__all__ = [
+    "NormalizeConfig",
+    "ProviderConfig",
+    "load_config",
+    "EmbeddingProvider",
+    "EmbeddingService",
+    "get_embedding_service",
+    "EmbeddingNormalizer",
+    "validate_embeddings",
+    "normalize_l2",
+    "quantize_corpus",
+    "__version__",
+]

winnex_ai_normalize-1.0.0/winnex_ai_normalize/api/server.py

@@ -0,0 +1,78 @@
+"""
+winnex-ai-normalize — OpenAI-compatible embedding API.
+
+Exposes the normalization service behind the OpenAI `/v1/embeddings`
+contract, so any consumer (Liferay, Maestro, tracer, or a plain OpenAI
+client) can get Madhava-ready vectors:
+
+    POST /v1/embeddings  {"model": "...", "input": ["text", ...]}
+        → {"data": [{"embedding": [..], "index": 0}], "model": "...", "usage": {...}}
+
+Also exposes:
+    GET  /v1/health            — provider availability
+    GET  /v1/normalize/health  — alias (Liferay-friendly)
+
+License: Business Source License 1.1 (BSL 1.1)
+"""
+import os
+from typing import List, Optional
+
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+
+from winnex_ai_normalize.core.embedding import get_embedding_service
+from winnex_ai_normalize.core.config import load_config
+
+app = FastAPI(
+    title="winnex-ai-normalize",
+    version="1.0.0",
+    description="Input normalization for the Madhava engine (OpenAI-compatible embeddings).",
+)
+
+
+class EmbeddingsRequest(BaseModel):
+    model: str = ""
+    input: List[str]
+
+
+@app.post("/v1/embeddings")
+def embeddings(req: EmbeddingsRequest):
+    """OpenAI-compatible embeddings endpoint (Madhava-ready vectors)."""
+    service = get_embedding_service()
+    try:
+        vecs = service.embed_texts(req.input)
+    except RuntimeError as e:
+        raise HTTPException(503, str(e))
+    data = [
+        {"embedding": vecs[i].tolist(), "index": i}
+        for i in range(len(vecs))
+    ]
+    return {
+        "object": "list",
+        "data": data,
+        "model": req.model or "winnex-ai-normalize",
+        "usage": {"prompt_tokens": sum(len(t) // 4 for t in req.input),
+                  "total_tokens": sum(len(t) // 4 for t in req.input)},
+    }
+
+
+@app.get("/v1/health")
+@app.get("/v1/normalize/health")
+def health():
+    """Provider availability (fail loudly if none reachable)."""
+    cfg = load_config()
+    svc = get_embedding_service()
+    status = svc.check_available()
+    return {
+        "status": "ok" if status.get("available") else "degraded",
+        "service": "winnex-ai-normalize",
+        "default_dim": cfg.default_dim,
+        "default_provider": cfg.default_provider,
+        "provider_order": cfg.provider_order,
+        "providers": status,
+    }
+
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "8102")))

winnex_ai_normalize-1.0.0/winnex_ai_normalize/core/__init__.py

@@ -0,0 +1 @@
+"""winnex-ai-normalize — core (config, embedding providers, normalization)."""

winnex_ai_normalize-1.0.0/winnex_ai_normalize/core/config.py

@@ -0,0 +1,97 @@
+"""
+winnex-ai-normalize — configuration (JSON-driven).
+
+The single source of truth for the normalization providers, the embedding
+model and the provider priority/failover order. Mirrors the Maestro's
+`ai_models_config` (JSON-driven, no hardcoded values).
+
+Config sources, in order of precedence:
+  1. Constructor / env vars (WINNEX_AI_NORMALIZE_*)
+  2. A JSON file path (WINNEX_AI_NORMALIZE_CONFIG)
+  3. Sensible defaults.
+
+License: Business Source License 1.1 (BSL 1.1)
+"""
+import json
+import os
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+
+@dataclass
+class ProviderConfig:
+    """A normalization provider (embedding source)."""
+    name: str                      # "openai" | "qwen3" | "nano" | "xfactor" | "direct"
+    type: str = "openai_compat"    # how to call it: openai_compat | local | direct
+    model: str = ""
+    base_url: str = ""
+    api_key_env: str = ""          # env var holding the API key (never hardcoded)
+    dim: int = 0                   # 0 = auto-detect
+    timeout: float = 20.0
+    priority: int = 10             # lower = tried first (failover order)
+    enabled: bool = True
+
+
+@dataclass
+class NormalizeConfig:
+    """Top-level normalization configuration."""
+    default_dim: int = 1024             # Qwen3-Embedding-0.6B
+    default_provider: str = "qwen3"     # the first provider to try
+    normalize_l2: bool = True           # L2-normalize embeddings (cosine contract)
+    validate_nan: bool = True           # fail loudly on NaN/inf
+    validate_range: bool = False        # warn (not fail) on out-of-range
+    cache_max: int = 512                # bounded embedding cache
+    providers: List[ProviderConfig] = field(default_factory=list)
+    provider_order: List[str] = field(default_factory=list)  # failover order
+
+    @classmethod
+    def from_env(cls) -> "NormalizeConfig":
+        """Load from WINNEX_AI_NORMALIZE_* env vars / JSON file."""
+        cfg = cls()
+        cfg.default_dim = int(os.environ.get("WINNEX_AI_NORMALIZE_DIM", cfg.default_dim))
+        cfg.default_provider = os.environ.get(
+            "WINNEX_AI_NORMALIZE_PROVIDER", cfg.default_provider)
+        # A config JSON may define the providers + failover order.
+        json_path = os.environ.get("WINNEX_AI_NORMALIZE_CONFIG", "")
+        if json_path and os.path.exists(json_path):
+            with open(json_path) as f:
+                data = json.load(f)
+            cfg.default_dim = int(data.get("default_dim", cfg.default_dim))
+            cfg.default_provider = data.get("default_provider", cfg.default_provider)
+            cfg.normalize_l2 = bool(data.get("normalize_l2", cfg.normalize_l2))
+            cfg.cache_max = int(data.get("cache_max", cfg.cache_max))
+            for p in data.get("providers", []):
+                cfg.providers.append(ProviderConfig(**p))
+            cfg.provider_order = list(data.get("provider_order", []))
+        # Default providers if none configured.
+        if not cfg.providers:
+            cfg.providers = [
+                ProviderConfig(
+                    name="qwen3", type="openai_compat",
+                    model=os.environ.get("EMBEDDING_MODEL", "/workspace/models/Qwen3-Embedding-0.6B"),
+                    base_url=os.environ.get("EMBEDDING_URL", "http://winnex-embedding:8102"),
+                    dim=cfg.default_dim, priority=1),
+                ProviderConfig(
+                    name="openai", type="openai_compat",
+                    model=os.environ.get("OPENAI_EMBEDDING_MODEL", "text-embedding-3-small"),
+                    base_url=os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1"),
+                    api_key_env="OPENAI_API_KEY", priority=2),
+                ProviderConfig(
+                    name="direct", type="direct",
+                    dim=cfg.default_dim, priority=99),
+            ]
+        if not cfg.provider_order:
+            cfg.provider_order = [p.name for p in sorted(
+                cfg.providers, key=lambda p: p.priority)]
+        return cfg
+
+
+def load_config() -> NormalizeConfig:
+    """Convenience loader (cached)."""
+    global _CONFIG
+    if _CONFIG is None:
+        _CONFIG = NormalizeConfig.from_env()
+    return _CONFIG
+
+
+_CONFIG: Optional[NormalizeConfig] = None

winnex_ai_normalize-1.0.0/winnex_ai_normalize/core/embedding.py

@@ -0,0 +1,166 @@
+"""
+winnex-ai-normalize — embedding providers (with failover).
+
+Extracted from the Maestro's `EmbeddingService` (winnex-madhava-maestro):
+connects to an OpenAI-compatible `/v1/embeddings` endpoint (e.g.
+Qwen3-Embedding-0.6B), with a bounded LRU cache and L2 normalization.
+
+NO fallback to fake vectors: if the configured providers are unavailable,
+the error is explicit (a normalizer must not fabricate embeddings).
+
+Provider failover: the providers are tried in `config.provider_order`
+(lowest priority first); if one fails, the next is tried. If ALL fail,
+a RuntimeError is raised with the collected errors.
+"""
+import logging
+import os
+import threading
+from typing import List, Optional
+
+import numpy as np
+
+logger = logging.getLogger("winnex-ai-normalize.embedding")
+
+
+class EmbeddingProvider:
+    """A single embedding provider (OpenAI-compatible /v1/embeddings)."""
+
+    def __init__(self, config, http_client_factory=None):
+        self.name = config.name
+        self.model = config.model
+        self.base_url = config.base_url
+        self.timeout = config.timeout
+        self.api_key_env = config.api_key_env
+        self.dim = config.dim
+        self._http_factory = http_client_factory
+
+    def _post(self, path: str, payload: dict) -> dict:
+        import httpx
+        headers = {}
+        if self.api_key_env:
+            key = os.environ.get(self.api_key_env, "")
+            if key:
+                headers["Authorization"] = f"Bearer {key}"
+        with httpx.Client(timeout=self.timeout) as client:
+            resp = client.post(f"{self.base_url}{path}", json=payload, headers=headers)
+            resp.raise_for_status()
+            return resp.json()
+
+    def embed(self, texts: List[str]) -> np.ndarray:
+        """Embed a list of texts → (n, d) float32 L2-normalized."""
+        result = self._post("/embeddings", {"model": self.model, "input": texts})
+        data = sorted(result.get("data", []), key=lambda x: x.get("index", 0))
+        if not data:
+            raise RuntimeError(f"provider {self.name}: empty embeddings response")
+        vecs = np.array([d["embedding"] for d in data], dtype=np.float32)
+        norms = np.linalg.norm(vecs, axis=1, keepdims=True)
+        vecs = vecs / np.maximum(norms, 1e-12)   # L2-normalize (cosine contract)
+        if not np.isfinite(vecs).all():
+            raise RuntimeError(f"provider {self.name}: embeddings contain NaN/inf")
+        self.dim = vecs.shape[1]
+        return np.ascontiguousarray(vecs, dtype=np.float32)
+
+    def check_available(self) -> dict:
+        try:
+            r = self._post("/embeddings", {"model": self.model, "input": ["health-check"]})
+            return {"available": bool(r.get("data")), "provider": self.name,
+                    "model": self.model, "base_url": self.base_url}
+        except Exception as e:
+            return {"available": False, "provider": self.name, "error": str(e)}
+
+
+class EmbeddingService:
+    """Embedding service with provider failover + bounded cache.
+
+    Tries the providers in `config.provider_order`; the first that succeeds
+    serves the batch. If ALL fail, raises RuntimeError with the collected
+    errors (NO silent fallback to fake vectors).
+    """
+
+    def __init__(self, config=None, provider_factory=EmbeddingProvider):
+        from .config import load_config
+        self.config = config or load_config()
+        self._providers = {
+            p.name: provider_factory(p) for p in self.config.providers
+        }
+        self._cache: dict[str, np.ndarray] = {}
+        self._lock = threading.Lock()
+
+    def _cache_put(self, text: str, vec: np.ndarray) -> None:
+        with self._lock:
+            if len(self._cache) >= max(self.config.cache_max, 1):
+                for old in list(self._cache)[: max(0, len(self._cache) - self.config.cache_max + 1)]:
+                    self._cache.pop(old, None)
+            self._cache[text] = vec
+
+    def _embed_missing(self, texts: List[str]) -> np.ndarray:
+        """Embed via the first available provider (failover)."""
+        errors = []
+        for name in self.config.provider_order:
+            p = self._providers.get(name)
+            if not p or not p.__dict__.get("_available", True):
+                continue
+            try:
+                return p.embed(texts)
+            except Exception as e:
+                errors.append(f"{name}: {str(e)[:120]}")
+                logger.warning(f"provider {name} failed: {e}")
+        raise RuntimeError(
+            "EmbeddingService: ALL providers unavailable — refusing to "
+            f"fabricate embeddings. Errors: {'; '.join(errors) or 'none configured'}")
+
+    def embed_texts(self, texts: List[str], dim: Optional[int] = None) -> np.ndarray:
+        """Embed a list of texts → (n, d) float32, cache repeated texts.
+
+        Args:
+            texts: list of strings.
+            dim: expected dimension (validated).
+        Returns:
+            (n, d) float32 L2-normalized embeddings.
+        """
+        if not texts:
+            return np.zeros((0, dim or self.config.default_dim), dtype=np.float32)
+        with self._lock:
+            missing_idx = [i for i, t in enumerate(texts) if t not in self._cache]
+        if missing_idx:
+            missing_texts = [texts[i] for i in missing_idx]
+            result = self._embed_missing(missing_texts)
+            if dim and result.shape[1] != dim:
+                raise ValueError(
+                    f"provider returned dim {result.shape[1]}, expected {dim}")
+            for i, v in zip(missing_idx, result):
+                self._cache_put(texts[i], v)
+        with self._lock:
+            vecs = np.stack([self._cache[t] for t in texts]).astype(np.float32)
+        return np.ascontiguousarray(vecs, dtype=np.float32)
+
+    def embed_one(self, text: str) -> np.ndarray:
+        """Embed a single text → (d,) float32."""
+        return self.embed_texts([text])[0]
+
+    def clear_cache(self) -> None:
+        with self._lock:
+            self._cache.clear()
+
+    def check_available(self) -> dict:
+        for name in self.config.provider_order:
+            p = self._providers.get(name)
+            if p:
+                r = p.check_available()
+                if r.get("available"):
+                    return r
+                logger.warning(f"provider {name} unavailable: {r.get('error')}")
+        return {"available": False,
+                "error": "no provider available",
+                "tried": self.config.provider_order}
+
+
+# Singleton for reuse
+_service = None
+
+
+def get_embedding_service() -> EmbeddingService:
+    global _service
+    if _service is None:
+        _service = EmbeddingService()
+    return _service

winnex_ai_normalize-1.0.0/winnex_ai_normalize/core/normalize.py

@@ -0,0 +1,188 @@
+"""
+winnex-ai-normalize — embedding normalization (the plug that feeds Madhava).
+
+The Madhava engine is AGNOSTIC: it consumes float32 vectors. This module
+normalizes ANY input (text via a provider, or raw vectors) into a valid
+float32 corpus ready for `winnex_madhava.build_engine`:
+
+  - validates dimension, dtype, NaN/inf (fail loudly — no silent fallback),
+  - L2-normalizes when the cosine contract requires it,
+  - quantizes float32 → uint8 for the engine's native corpus
+    (the exact scale logic from the Maestro's `quantize_corpus`).
+
+License: Business Source License 1.1 (BSL 1.1)
+"""
+import logging
+from typing import Optional, Union
+
+import numpy as np
+
+logger = logging.getLogger("winnex-ai-normalize")
+
+
+# ---------------------------------------------------------------------------
+# Validation (fail loudly — no fake embeddings)
+# ---------------------------------------------------------------------------
+def validate_embeddings(
+    vectors: np.ndarray,
+    dim: Optional[int] = None,
+    require_unit_norm: bool = False,
+) -> np.ndarray:
+    """Validate a (n, d) embedding matrix. Raises on invalid input.
+
+    Args:
+        vectors: (n, d) float32 (or float64) embeddings.
+        dim: expected dimensionality (raises if mismatch).
+        require_unit_norm: if True, each row must be unit-norm (cosine).
+    Returns:
+        the array as float32, contiguous.
+    Raises:
+        ValueError: on wrong dim, NaN/inf, or empty.
+    """
+    arr = np.ascontiguousarray(vectors, dtype=np.float32)
+    if arr.ndim != 2:
+        raise ValueError(f"embeddings must be 2D (n, d), got shape {arr.shape}")
+    if arr.shape[0] == 0:
+        raise ValueError("embeddings must not be empty")
+    if dim is not None and arr.shape[1] != dim:
+        raise ValueError(
+            f"dimension mismatch: expected {dim}, got {arr.shape[1]}")
+    if not np.isfinite(arr).all():
+        raise ValueError("embeddings contain NaN or inf — refusing to proceed")
+    if require_unit_norm:
+        norms = np.linalg.norm(arr, axis=1)
+        bad = np.where((norms < 1e-6) | (np.abs(norms - 1.0) > 1e-2))[0]
+        if len(bad):
+            raise ValueError(
+                f"{len(bad)} rows are not unit-norm (cosine contract) — "
+                "call normalize_l2() first or fix the provider")
+    return arr
+
+
+# ---------------------------------------------------------------------------
+# L2 normalization (cosine contract)
+# ---------------------------------------------------------------------------
+def normalize_l2(vectors: np.ndarray) -> np.ndarray:
+    """L2-normalize each row to unit norm (the cosine contract).
+
+    Raises on a zero-norm row (cannot be normalized meaningfully).
+    """
+    arr = validate_embeddings(vectors)
+    norms = np.linalg.norm(arr, axis=1, keepdims=True)
+    if (norms < 1e-12).any():
+        raise ValueError("a row has zero norm — cannot L2-normalize")
+    return arr / np.maximum(norms, 1e-12)
+
+
+# ---------------------------------------------------------------------------
+# Quantization float32 → uint8 (the engine's native corpus)
+# ---------------------------------------------------------------------------
+def quantize_corpus(embeddings: np.ndarray) -> np.ndarray:
+    """Quantize float32 normalized embeddings to the engine's uint8 corpus.
+
+    The native C++ engine requires a uint8 corpus and interprets each byte as
+    an integer coordinate in [0, 255] (see `load_raw` in winnex-madhava: it
+    casts uint8 → float32 without re-scaling, then L2-normalizes). Passing raw
+    float32 normalized embeddings straight into `build_engine` silently
+    truncates them (`astype(np.uint8)` maps everything in [-1, 1] to 0) — a
+    zero corpus whose cosine scores are all 0.0.
+
+    This is the exact scale logic from the Maestro's `quantize_corpus`:
+      - values in [-1, 1] (embedding domain)   → map to [0, 255] via (x+1)*127.5
+      - values in [0, 1] (post-activation)     → scale to [0, 255] via x*255
+      - values already in [0, 255]             → left as-is (round+clip)
+
+    Args:
+        embeddings: (n, d) float32 normalized embeddings (cosine).
+    Returns:
+        (n, d) uint8 corpus ready for `build_engine`.
+    """
+    arr = validate_embeddings(embeddings)
+    if arr.size and arr.min() < 0.0:
+        # Embedding-domain values (normalized vectors): [-1, 1] -> [0, 255].
+        u8 = np.round((arr + 1.0) * 127.5).clip(0, 255).astype(np.uint8)
+    else:
+        # [0, 1] normalized domain (Qwen post-activation): scale to [0, 255].
+        u8 = np.round(arr * 255.0).clip(0, 255).astype(np.uint8)
+    return np.ascontiguousarray(u8, dtype=np.uint8)
+
+
+# ---------------------------------------------------------------------------
+# The normalizer facade (text/vectors → ready-for-madhava)
+# ---------------------------------------------------------------------------
+class EmbeddingNormalizer:
+    """Normalizes text or raw vectors into Madhava-ready float32/uint8."""
+
+    def __init__(self, config=None, embedding_service=None):
+        from .config import load_config
+        self.config = config or load_config()
+        self._embedding_service = embedding_service
+
+    @property
+    def embedding_service(self):
+        if self._embedding_service is None:
+            from .embedding import get_embedding_service
+            self._embedding_service = get_embedding_service()
+        return self._embedding_service
+
+    def vectorize_texts(self, texts, dim=None) -> np.ndarray:
+        """Embed texts via the configured provider, normalized to float32.
+
+        Raises if no provider is available (NO fallback to fake vectors).
+        """
+        dim = dim or self.config.default_dim
+        return self.embedding_service.embed_texts(texts, dim=dim)
+
+    def normalize_vectors(
+        self,
+        vectors: Union[np.ndarray, list],
+        dim: Optional[int] = None,
+        require_unit_norm: bool = False,
+    ) -> np.ndarray:
+        """Validate + normalize raw vectors to float32 (unit-norm optional)."""
+        arr = np.asarray(vectors, dtype=np.float32)
+        return validate_embeddings(arr, dim=dim, require_unit_norm=require_unit_norm)
+
+    def to_corpus(
+        self,
+        vectors: Union[np.ndarray, list],
+        dim: Optional[int] = None,
+    ) -> np.ndarray:
+        """Quantize float32 → uint8 (for BIGANN-style RAW BYTE corpora).
+
+        NOTE: this is NOT the cosine path. For float32 embeddings
+        (OpenAI/Qwen3) use `build_engine()` which goes through the engine's
+        float32 manifold (build_float32) and preserves cosine exactly.
+        """
+        arr = self.normalize_vectors(vectors, dim=dim)
+        return quantize_corpus(arr)
+
+    def build_engine(self, vectors, dim=None, k=10, **engine_kwargs):
+        """Build a Madhava engine over float32 embeddings (the CORRECT path).
+
+        Uses `winnex_madhava.build_engine` with the float32 corpus (the
+        engine's build_float32 manifold preserves cosine). The madhava
+        engine stays agnostic — it receives ready float32 vectors.
+
+        Args:
+            vectors: (n, d) float32 embeddings (or text via vectorize_texts).
+            dim: embedding dimension.
+            k: top-k.
+            **engine_kwargs: passed to winnex_madhava.build_engine.
+        Returns:
+            a winnex_madhava engine.
+        """
+        import winnex_madhava as wm
+        arr = self.normalize_vectors(vectors, dim=dim)
+        return wm.build_engine(
+            np.ascontiguousarray(arr, dtype=np.float32),
+            dim=dim or arr.shape[1],
+            metric="cosine",
+            k=k,
+            normalize_input=True,
+            **engine_kwargs,
+        )
+
+    def check_available(self) -> dict:
+        """Health check: is the embedding provider reachable?"""
+        return self.embedding_service.check_available()

winnex_ai_normalize-1.0.0/winnex_ai_normalize/tests/test_normalize.py

@@ -0,0 +1,98 @@
+"""
+winnex-ai-normalize — normalization tests (REAL data, no fallback).
+
+Validates:
+  - validate_embeddings rejects NaN/wrong-dim (fail loudly).
+  - normalize_l2 produces unit-norm rows.
+  - quantize_corpus maps float32 → uint8 with the correct scale.
+  - EmbeddingNormalizer end-to-end with a REAL dataset (arXiv d=1536).
+
+Run:  python -m pytest winnex_ai_normalize/tests/ -v
+"""
+import os
+
+import numpy as np
+import pytest
+
+from winnex_ai_normalize.core.normalize import (
+    EmbeddingNormalizer,
+    validate_embeddings,
+    normalize_l2,
+    quantize_corpus,
+)
+from winnex_ai_normalize.core.config import NormalizeConfig
+
+
+def test_validate_rejects_nan():
+    v = np.random.randn(10, 128).astype(np.float32)
+    v[3, 5] = np.nan
+    with pytest.raises(ValueError, match="NaN"):
+        validate_embeddings(v)
+
+
+def test_validate_rejects_wrong_dim():
+    v = np.random.randn(10, 128).astype(np.float32)
+    with pytest.raises(ValueError, match="dimension"):
+        validate_embeddings(v, dim=64)
+
+
+def test_validate_rejects_empty():
+    with pytest.raises(ValueError, match="empty"):
+        validate_embeddings(np.zeros((0, 128), dtype=np.float32))
+
+
+def test_normalize_l2_unit_norm():
+    v = np.random.randn(50, 128).astype(np.float32)
+    n = normalize_l2(v)
+    norms = np.linalg.norm(n, axis=1)
+    assert np.allclose(norms, 1.0, atol=1e-5)
+
+
+def test_quantize_corpus_scale():
+    """uint8 quantization maps to [0, 255] with the correct scale.
+
+    NOTE: uint8 quantization is for BIGANN-style RAW BYTE corpora. For
+    float32 embeddings (OpenAI/Qwen3) the correct path is build_float32
+    (the engine's float32 manifold preserves cosine exactly); quantizing
+    float32 → uint8 with a shift is lossy and NOT the cosine path.
+    """
+    v = np.random.randn(20, 128).astype(np.float32)
+    v /= np.linalg.norm(v, axis=1, keepdims=True)
+    u8 = quantize_corpus(v)
+    assert u8.dtype == np.uint8
+    assert u8.min() >= 0 and u8.max() <= 255
+
+
+def test_build_float32_path_preserves_cosine():
+    """The float32 path (build_float32) preserves cosine — the correct
+    path for real embeddings (no uint8 quantization)."""
+    v = np.random.randn(20, 128).astype(np.float32)
+    v /= np.linalg.norm(v, axis=1, keepdims=True)
+    # After L2-normalization the cosine of the float32 IS the inner product.
+    c = float(v[0] @ v[1])
+    assert -1.0 <= c <= 1.0
+    # The normalizer validates + keeps float32 (no lossy uint8 for cosine).
+    from winnex_ai_normalize import validate_embeddings
+    out = validate_embeddings(v, dim=128)
+    assert np.allclose(out, v)
+
+
+def test_normalizer_with_real_arxiv():
+    """End-to-end with the REAL arXiv OpenAI embeddings (d=1536)."""
+    path = "/home/wnnx_user/zenodo/arxiv_100k.npy"
+    if not os.path.exists(path):
+        pytest.skip("arxiv_100k.npy not present — skipping real-data test")
+    a = np.load(path, mmap_mode="r")
+    X = np.ascontiguousarray(a[:200])          # real embeddings d=1536
+    n = np.linalg.norm(X, axis=1, keepdims=True)
+    X = X / np.maximum(n, 1e-12)
+
+    cfg = NormalizeConfig(default_dim=1536)
+    norm = EmbeddingNormalizer(config=cfg)
+    # Vectors pass through unchanged (validated + float32).
+    out = norm.normalize_vectors(X, dim=1536)
+    assert out.shape == (200, 1536)
+    assert np.isfinite(out).all()
+    # Corpus quantization for the madhava engine.
+    u8 = norm.to_corpus(X, dim=1536)
+    assert u8.shape == (200, 1536) and u8.dtype == np.uint8

winnex_ai_normalize-1.0.0/winnex_ai_normalize.egg-info/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: winnex-ai-normalize
+Version: 1.0.0
+Summary: Input normalization plug for the Madhava engine — embed text via any provider (OpenAI, Qwen3, nano, xfactor), validate/normalize to float32/uint8, with provider failover and NO fake-vector fallback. Consumed by winnex-madhava, winnex-tracer, Liferay and the Maestro.
+Author-email: Winnex Brasil Soluções Empresariais LTDA-ME <pay@winnex.ai>
+License: Business Source License 1.1
+        
+        License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
+        "Business Source License" is a trademark of MariaDB Corporation Ab.
+        
+        Parameters
+        
+        Licensor:             Winnex AI
+        Licensed Work:        Tracer-GOV
+        Additional Use Grant: Free for Brazilian government agencies.
+                              Any use by a public sector entity in Brazil is permitted.
+        
+        Change Date:          2036-01-01
+        Change License:       GNU General Public License v2.0 or later
+        
+        For inquiries: pay@winnex.ai
+        
+Project-URL: Homepage, https://winnex.ai
+Project-URL: Source, https://github.com/winnex-ai/winnex-ai-normalize
+Keywords: embedding,normalize,madhava,vector,openai,qwen3,rag,semantic-search,normalizer,winnex,failover
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: httpx>=0.24
+Provides-Extra: api
+Requires-Dist: fastapi>=0.100; extra == "api"
+Requires-Dist: uvicorn>=0.20; extra == "api"
+Provides-Extra: madhava
+Requires-Dist: winnex-madhava>=1.9.2; extra == "madhava"
+Provides-Extra: all
+Requires-Dist: fastapi>=0.100; extra == "all"
+Requires-Dist: uvicorn>=0.20; extra == "all"
+Requires-Dist: winnex-madhava>=1.9.2; extra == "all"
+Dynamic: license-file
+
+# winnex-ai-normalize
+
+**The input-normalization plug for the Madhava engine.**
+
+The Madhava engine is **agnostic** — it consumes float32 vectors. This
+package normalizes **any** input (text via an embedding provider, or raw
+vectors) into a valid float32/uint8 corpus ready for
+`winnex_madhava.build_engine`, with **provider failover** and **no silent
+fallback to fake vectors**.
+
+```
+text / raw vectors
+      │
+      ▼
+winnex-ai-normalize
+  ├── providers:  OpenAI / Qwen3 / nano / xfactor / direct (failover order)
+  ├── validate:   dim + NaN + norm (fail loudly)
+  ├── normalize:  L2 unit-norm (cosine contract)
+  └── quantize:   float32 → uint8 (the engine's native corpus)
+      │
+      ▼
+winnex-madhava (agnostic) → top-K + Cauchy-Schwarz proof
+```
+
+## Install
+
+```bash
+pip install winnex-ai-normalize               # core (numpy + httpx)
+pip install winnex-ai-normalize[api]          # + OpenAI-compatible /v1/embeddings
+pip install winnex-ai-normalize[all]          # + winnex-madhava
+```
+
+## Quick start
+
+```python
+from winnex_ai_normalize import EmbeddingNormalizer
+
+norm = EmbeddingNormalizer()                  # reads env / JSON config
+
+# 1. Embed text via the configured provider (OpenAI / Qwen3 / ...)
+vecs = norm.vectorize_texts(["hypertension treatment", "diabetes care"])
+# vecs: (2, d) float32 L2-normalized
+
+# 2. Or normalize raw vectors (validated, no NaN, unit-norm)
+vecs = norm.normalize_vectors(my_embeddings, dim=1024)
+
+# 3. Feed the Madhava engine (agnostic)
+u8 = norm.to_corpus(vecs, dim=1024)           # uint8 corpus for build_engine
+import winnex_madhava as wm
+engine = wm.build_engine(u8, dim=1024, metric="cosine", k=10)
+```
+
+## OpenAI-compatible API
+
+```bash
+uvicorn winnex_ai_normalize.api.server:app --port 8102
+curl -X POST http://localhost:8102/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{"model": "Qwen3-Embedding-0.6B", "input": ["text one", "text two"]}'
+# → {"data": [{"embedding": [...], "index": 0}, ...], ...}
+```
+
+## Provider failover (no fake fallback)
+
+Providers are tried in `provider_order` (JSON/env config). If **all**
+fail, a `RuntimeError` is raised with the collected errors — a normalizer
+must never fabricate embeddings.
+
+```json
+{
+  "default_dim": 1024,
+  "default_provider": "qwen3",
+  "providers": [
+    {"name": "qwen3", "base_url": "http://winnex-embedding:8102",
+     "model": "/workspace/models/Qwen3-Embedding-0.6B", "priority": 1},
+    {"name": "openai", "base_url": "https://api.openai.com/v1",
+     "model": "text-embedding-3-small", "api_key_env": "OPENAI_API_KEY", "priority": 2}
+  ],
+  "provider_order": ["qwen3", "openai"]
+}
+```
+
+## Consumed by
+
+- **winnex-madhava** (direct — the engine stays agnostic)
+- **winnex-tracer** (audit + commitment)
+- **Liferay bridges** (tracer-gov-liferay, tracer-med-liferay)
+- **Maestro** (winnex-ai-server / winnex-ai-engine) and any external tool
+
+## License
+
+Business Source License 1.1 (BSL 1.1) | pay@winnex.ai |
+Winnex Brasil Soluções Empresariais LTDA (CNPJ 58.364.637/0001-47)

winnex_ai_normalize-1.0.0/winnex_ai_normalize.egg-info/SOURCES.txt

@@ -0,0 +1,17 @@
+LICENSE
+README.md
+pyproject.toml
+winnex_ai_normalize/__init__.py
+winnex_ai_normalize.egg-info/PKG-INFO
+winnex_ai_normalize.egg-info/SOURCES.txt
+winnex_ai_normalize.egg-info/dependency_links.txt
+winnex_ai_normalize.egg-info/requires.txt
+winnex_ai_normalize.egg-info/top_level.txt
+winnex_ai_normalize/api/__init__.py
+winnex_ai_normalize/api/server.py
+winnex_ai_normalize/core/__init__.py
+winnex_ai_normalize/core/config.py
+winnex_ai_normalize/core/embedding.py
+winnex_ai_normalize/core/normalize.py
+winnex_ai_normalize/tests/__init__.py
+winnex_ai_normalize/tests/test_normalize.py

winnex_ai_normalize-1.0.0/winnex_ai_normalize.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

winnex_ai_normalize-1.0.0/winnex_ai_normalize.egg-info/requires.txt

@@ -0,0 +1,14 @@
+numpy>=1.24
+httpx>=0.24
+
+[all]
+fastapi>=0.100
+uvicorn>=0.20
+winnex-madhava>=1.9.2
+
+[api]
+fastapi>=0.100
+uvicorn>=0.20
+
+[madhava]
+winnex-madhava>=1.9.2

winnex_ai_normalize-1.0.0/winnex_ai_normalize.egg-info/top_level.txt

@@ -0,0 +1 @@
+winnex_ai_normalize