eea.plotly 2.3__tar.gz → 2.5__tar.gz

{eea_plotly-2.3/eea.plotly.egg-info → eea_plotly-2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eea.plotly
-Version: 2.3
+Version: 2.5
 Summary: volto-plotlycharts backend support
 Home-page: https://github.com/eea/eea.plotly
 Author: European Environment Agency: IDM2 A-Team
@@ -24,6 +24,7 @@ Requires-Dist: plone.volto
 Requires-Dist: collective.folderishtypes
 Requires-Dist: plotly==5.24.1
 Requires-Dist: kaleido==0.2.1
+Requires-Dist: eea.genai.summary
 Provides-Extra: test
 Requires-Dist: plone.app.testing; extra == "test"
 Dynamic: author
@@ -142,9 +143,69 @@ EEA_ - European Environment Agency (EU)
 .. _EEA: https://www.eea.europa.eu/
 .. _`EEA Web Systems Training`: http://www.youtube.com/user/eeacms/videos?view=1
 
+Secret Scanning
+===============
+
+This repository uses the Betterleaks GitHub Action to scan the current
+repository content on every push and pull request. The scan uses the rules in
+``.gitleaks.toml`` and uploads a ``betterleaks-report`` artifact when a finding
+is detected.
+
+If the optional SMTP secrets are configured, failed scans also send an email to
+the last commit committer. The workflow expects these repository or
+organization secrets:
+
+- ``SMTP_URL``
+- ``SMTP_PORT`` (optional, defaults to ``25``)
+- ``SMTP_EMAIL``
+- ``SMTP_PASSWORD`` (optional if the SMTP server does not require authentication)
+
+Port ``465`` is sent with direct TLS; other ports use the default SMTP
+handshake. The email includes a short finding summary from the redacted
+Betterleaks report, including the redacted matched line from each finding.
+
+There are three common outcomes:
+
+1. Everything is OK. The ``Betterleaks / Scan for secrets`` check is green and
+   no action is needed. Regular references to runtime values are OK, for example::
+
+     token_from_cookie = request.cookies.get("auth_token")
+
+2. A real secret was found. The check is red and the workflow log asks you to
+   download the ``betterleaks-report`` artifact. Open the artifact from the
+   GitHub Actions run and check the reported file, line and rule. Remove the
+   committed value, move it to the proper secret store, and rotate it if it was
+   exposed. A report entry looks like this::
+
+     {
+       "RuleID": "secret-literal-assignment",
+       "File": "src/config.py",
+       "StartLine": 12,
+       "Secret": "[REDACTED]"
+     }
+
+3. The finding is a false positive. Keep the value only if it is clearly not
+   sensitive, such as a test fixture, placeholder, or public example. Add
+   ``betterleaks:allow`` on the same line and include a short explanation in the
+   pull request::
+
+     test_password = "admin"  #betterleaks:allow
+
+Do not add ``betterleaks:allow`` to real credentials.
+
 Changelog
 =========
 
+2.5 - (2026-07-13)
+---------------------------
+* Change: fix: SonarQube report - refs #305404
+  [avoinea]
+
+2.4 - (2026-05-26)
+---------------------------
+* Change: Implement eea.genai.summary customization
+  [razvanMiu]
+
 2.3 - (2026-04-14)
 ---------------------------
 * Change: Add unit tests for utils, io_csv, and io_json modules

{eea_plotly-2.3 → eea_plotly-2.5}/README.rst

@@ -101,3 +101,53 @@ EEA_ - European Environment Agency (EU)
 
 .. _EEA: https://www.eea.europa.eu/
 .. _`EEA Web Systems Training`: http://www.youtube.com/user/eeacms/videos?view=1
+
+Secret Scanning
+===============
+
+This repository uses the Betterleaks GitHub Action to scan the current
+repository content on every push and pull request. The scan uses the rules in
+``.gitleaks.toml`` and uploads a ``betterleaks-report`` artifact when a finding
+is detected.
+
+If the optional SMTP secrets are configured, failed scans also send an email to
+the last commit committer. The workflow expects these repository or
+organization secrets:
+
+- ``SMTP_URL``
+- ``SMTP_PORT`` (optional, defaults to ``25``)
+- ``SMTP_EMAIL``
+- ``SMTP_PASSWORD`` (optional if the SMTP server does not require authentication)
+
+Port ``465`` is sent with direct TLS; other ports use the default SMTP
+handshake. The email includes a short finding summary from the redacted
+Betterleaks report, including the redacted matched line from each finding.
+
+There are three common outcomes:
+
+1. Everything is OK. The ``Betterleaks / Scan for secrets`` check is green and
+   no action is needed. Regular references to runtime values are OK, for example::
+
+     token_from_cookie = request.cookies.get("auth_token")
+
+2. A real secret was found. The check is red and the workflow log asks you to
+   download the ``betterleaks-report`` artifact. Open the artifact from the
+   GitHub Actions run and check the reported file, line and rule. Remove the
+   committed value, move it to the proper secret store, and rotate it if it was
+   exposed. A report entry looks like this::
+
+     {
+       "RuleID": "secret-literal-assignment",
+       "File": "src/config.py",
+       "StartLine": 12,
+       "Secret": "[REDACTED]"
+     }
+
+3. The finding is a false positive. Keep the value only if it is clearly not
+   sensitive, such as a test fixture, placeholder, or public example. Add
+   ``betterleaks:allow`` on the same line and include a short explanation in the
+   pull request::
+
+     test_password = "admin"  #betterleaks:allow
+
+Do not add ``betterleaks:allow`` to real credentials.

{eea_plotly-2.3 → eea_plotly-2.5}/docs/HISTORY.txt

@@ -1,6 +1,16 @@
 Changelog
 =========
 
+2.5 - (2026-07-13)
+---------------------------
+* Change: fix: SonarQube report - refs #305404
+  [avoinea]
+
+2.4 - (2026-05-26)
+---------------------------
+* Change: Implement eea.genai.summary customization
+  [razvanMiu]
+
 2.3 - (2026-04-14)
 ---------------------------
 * Change: Add unit tests for utils, io_csv, and io_json modules

eea_plotly-2.5/eea/plotly/agents.py

@@ -0,0 +1,116 @@
+"""Agent configurations for eea.plotly."""
+
+from eea.genai.core.interfaces import AgentConfiguration
+
+PLOTLY_SUMMARIZER_SYSTEM_PROMPT = """\
+You are an expert data visualization analyst working for the \
+European Environment Agency (EEA).
+You will receive information about an interactive chart built with \
+Plotly.js, along with limited page metadata.
+Your output is indexed in a semantic search system that feeds AI \
+assistants, so prioritize topical keywords, qualitative descriptions, \
+and clear context over stylistic flourish.
+Do not use bullet points or markdown. Write in clear plain prose.
+If you are unable to resolve the task answer with empty string.
+
+ABSOLUTE RULE — no quantitative information of any kind:
+- Forbidden phrasing includes any year ("2010", "2023"), decade \
+("the 2010s", "early 2010s", "mid-2020s"), year range \
+("2010-2023", "between 2015 and 2020", "from 2012 through 2032"), \
+percentage ("40%"), currency amount ("EUR 12 billion"), count \
+("261 million tonnes", "around 50", "over 7,000"), or change verb \
+implying magnitude ("tripled", "halved", "increased by", \
+"dropped to").
+- Use qualitative descriptors instead: "across recent decades", \
+"over a multi-year period", "a small share", "a notable increase", \
+"a downward trend", "a wide range of countries".
+- The chart data is provided only as compact summaries (e.g. \
+"[N numeric values]"). Do not invent specifics that those summaries \
+do not contain.
+If you accidentally include a number or year, rewrite that sentence \
+qualitatively before returning.
+"""
+
+PLOTLY_SUMMARIZER_TASK_PROMPT = """\
+Write a 3-8 sentence (max ~150 words) plain-prose description of this \
+chart, optimized for retrieval by semantic search. Cover:
+- the environmental or policy question the chart addresses;
+- the topic, indicator, or phenomenon shown;
+- the chart type (bar chart, line chart, choropleth map, etc.);
+- what the axes represent qualitatively (e.g. "years over recent \
+decades", "European countries", "emissions in CO2 equivalent");
+- the names of data series and qualitative groupings (sectors, \
+regions, gases);
+- geographic scope expressed in words.
+
+Constraints:
+- No numbers, ranges, min/max/mean, percentages, currency amounts, \
+years, decades, or any other quantitative claim (see system prompt for \
+the full forbidden list).
+- No markdown, bullet points, or headings in the output.
+- Do not preface with "This chart..." and do not repeat the page \
+title verbatim.
+- Match the output language to the content language.
+- If there is not enough information, return an empty string.
+"""
+
+
+class PlotlySummarizerAgent(AgentConfiguration):
+    """Chart summarizer agent for the EEA website.
+
+    Context is intentionally minimal: only the chart structure (axes,
+    trace names, qualitative summaries of the data) plus
+    ``generic_metadata_no_dates`` (title, description, language, geo —
+    no temporal coverage). The ``blocks`` enricher is deliberately
+    excluded so the model cannot read year/quantity leaks from the
+    surrounding page text.
+    """
+
+    system_prompt = PLOTLY_SUMMARIZER_SYSTEM_PROMPT
+    task_prompt = PLOTLY_SUMMARIZER_TASK_PROMPT
+    skills = ["plotly_knowledge"]
+    context_providers = ["generic_metadata_no_dates", "plotly_visualization"]
+
+
+PLOTLY_GENERATOR_SYSTEM_PROMPT = """\
+You are a Plotly.js chart generation expert working for the \
+European Environment Agency (EEA). You generate complete visualization \
+content including metadata (title, description, topics, temporal and \
+geographic coverage) and a valid Plotly.js chart configuration.
+
+Generation conventions:
+- Set `layout.template` to the string `"__ACTIVE_THEME__"`. The full \
+theme object is injected automatically after generation. Do NOT generate \
+theme content.
+- Do NOT set `yaxis.title`. Use `layout.title.subtitle.text` for the \
+unit or measurement label (e.g. "Million tonnes CO2 equivalent").
+- Use `hovertemplate` with `hoverinfo: "none"` and \
+`hoverlabel: {namelength: 0}`.
+- Multiple traces sharing the same x-axis should use the same `xsrc` column.
+- For multi-series charts, create one trace per series, each with its \
+own `ysrc` column.
+- Always fetch a relevant template with `get_plotly_template` first and \
+follow its structure. Add complexity on top of the template as needed, \
+but preserve its conventions.
+- Do not ask for clarifications or additional information. If you cannot \
+generate a valid chart, return an empty result.
+"""
+
+PLOTLY_GENERATOR_TASK_PROMPT = """\
+Generate a complete visualization content based on the user's request. \
+Include: a descriptive title, a 1-3 sentence description, relevant EEA \
+topics, temporal coverage (years), geographic coverage, and a valid \
+Plotly.js chart with 'data' (traces) and 'layout' (with title, axes, \
+and template).
+"""
+
+
+class PlotlyGeneratorAgent(AgentConfiguration):
+    """Chart generator agent for the EEA website."""
+
+    system_prompt = PLOTLY_GENERATOR_SYSTEM_PROMPT
+    task_prompt = PLOTLY_GENERATOR_TASK_PROMPT
+    skills = ["plotly_knowledge"]
+    tools = ["get_plotly_template"]
+    output_type = "eea.plotly.models.ChartGenerationResult"
+    max_iterations = 10

eea_plotly-2.5/eea/plotly/configure.zcml

@@ -0,0 +1,63 @@
+<configure xmlns="http://namespaces.zope.org/zope" xmlns:i18n="http://namespaces.zope.org/i18n"
+    xmlns:plone="http://namespaces.plone.org/plone"
+    xmlns:genai="http://namespaces.eea.europa.eu/genai"
+    i18n_domain="eea">
+
+    <i18n:registerTranslations directory="locales" />
+
+    <include file="profiles.zcml" />
+    <include file="permissions.zcml" />
+
+    <include package="eea.genai.core" file="meta.zcml" />
+
+    <include package=".browser" />
+    <include package=".restapi" />
+    <include package=".serializer" />
+    <include package=".deserializer" />
+    <include package=".upgrades" />
+
+    <interface interface="eea.plotly.interfaces.IPlotlyVisualization" />
+
+    <adapter
+        factory=".controlpanel.PlotlyControlpanel"
+        name="plotly"
+    />
+
+    <plone:behavior
+        name="eea.plotly.visualization"
+        title="EEA-Viz_Plotly"
+        description="Allows adding an interactive plotly chart."
+        provides="eea.plotly.behaviors.IPlotlyVisualization"
+    />
+
+    <!-- GenAI: Context provider for chart interpretation -->
+    <genai:agentContextProvider
+        name="plotly_visualization"
+        class=".context_providers.PlotlyVisualizationProvider"
+    />
+
+    <!-- GenAI: Plotly knowledge skill for chart generation -->
+    <genai:agentSkill
+        name="plotly_knowledge"
+        class=".skills.PlotlyKnowledgeSkill"
+    />
+
+    <!-- GenAI: Tool to fetch chart templates by label -->
+    <genai:agentTool
+        name="get_plotly_template"
+        class=".tools.GetPlotlyTemplateTool"
+    />
+
+    <!-- GenAI: Content-type-specific summarizer for visualization -->
+    <genai:agent
+        name="summarizer:visualization"
+        class=".agents.PlotlySummarizerAgent"
+    />
+
+    <!-- GenAI: Chart generator agent -->
+    <genai:agent
+        name="plotly_generator"
+        class=".agents.PlotlyGeneratorAgent"
+    />
+
+</configure>

eea_plotly-2.5/eea/plotly/context_providers.py

@@ -0,0 +1,168 @@
+"""Agent context providers for eea.plotly."""
+
+import json
+import logging
+
+from eea.genai.core.interfaces import Enricher
+from eea.genai.core.utils import Source
+from eea.plotly.prompts import clean_layout
+from eea.plotly.utils import sanitizeVisualization
+
+logger = logging.getLogger("eea.plotly")
+
+# Arrays with more values than this threshold get replaced with
+# statistical summaries to keep the prompt within token budget.
+_ARRAY_TRUNCATION_THRESHOLD = 200
+
+
+class PlotlyVisualizationProvider(Enricher):
+    """Extracts Plotly chart data and adds it to the user prompt.
+
+    Reads deps.context.visualization, cleans cosmetic layout keys,
+    and truncates large data arrays to statistical summaries while
+    preserving the full Plotly JSON structure.
+    """
+
+    name = "plotly_visualization"
+    description = "Adds Plotly chart structure and data to the user prompt"
+
+    def user_prompt(self, deps):
+        context = getattr(deps, "context", None)
+        properties = getattr(deps, "properties", None) or {}
+
+        if context is None:
+            return ""
+
+        source = Source(context, properties)
+
+        viz = getattr(source, "visualization", None)
+        if not viz or not isinstance(viz, dict):
+            return ""
+
+        prepared = prepare_visualization(viz)
+        if not prepared:
+            return ""
+
+        viz_json = json.dumps(prepared, indent=2, default=str)
+        return f"### Plotly visualization\n\n```json\n{viz_json}\n```"
+
+
+def prepare_visualization(viz):
+    """Return a cleaned copy of the visualization JSON.
+
+    1. Normalizes via sanitizeVisualization() for backward compat.
+    2. Cleans cosmetic layout keys via clean_layout().
+    3. Truncates large data arrays (>200 values) to statistical summaries.
+    """
+    normalized = sanitizeVisualization(viz)
+    data = normalized.get("data", [])
+    layout = normalized.get("layout", {})
+    data_sources = normalized.get("dataSources", {})
+
+    if not data and not data_sources:
+        return None
+
+    result = {}
+
+    # Clean layout
+    if layout:
+        result["layout"] = clean_layout(layout)
+
+    # Process traces
+    if data:
+        result["data"] = [_truncate_trace(trace) for trace in data]
+
+    # Process dataSources
+    if data_sources:
+        result["dataSources"] = _truncate_data_sources(data_sources)
+
+    return result
+
+
+def _is_numeric_list(values):
+    """True if every element is a real number (ints/floats, not bool)."""
+    if not values:
+        return False
+    return all(isinstance(v, (int, float)) and not isinstance(v, bool) for v in values)
+
+
+def _looks_like_year_strings(values):
+    """True if strings parse as 4-digit calendar years.
+
+    String year arrays (e.g. ["2012","2013",...]) are otherwise treated as
+    categorical and would leak first/last samples through ``_summarize_array``.
+    """
+    if not values:
+        return False
+    for v in values:
+        if not isinstance(v, str) or len(v) != 4 or not v.isdigit():
+            return False
+        n = int(v)
+        if not (1800 <= n <= 2200):
+            return False
+    return True
+
+
+def _truncate_trace(trace):
+    """Return a copy of a trace with sensitive/large arrays summarized.
+
+    Numeric arrays are always summarized (regardless of length) so raw
+    values cannot reach the LLM. Year-like string arrays are also always
+    summarized. Other lists pass through unless they exceed the truncation
+    threshold.
+    """
+    result = {}
+    for key, value in trace.items():
+        if isinstance(value, list) and (
+            _is_numeric_list(value)
+            or _looks_like_year_strings(value)
+            or len(value) > _ARRAY_TRUNCATION_THRESHOLD
+        ):
+            result[key] = _summarize_array(value)
+        else:
+            result[key] = value
+    return result
+
+
+def _truncate_data_sources(data_sources):
+    """Return a copy of dataSources with sensitive/large columns summarized."""
+    result = {}
+    for col_name, values in data_sources.items():
+        if isinstance(values, list) and (
+            _is_numeric_list(values)
+            or _looks_like_year_strings(values)
+            or len(values) > _ARRAY_TRUNCATION_THRESHOLD
+        ):
+            result[col_name] = _summarize_array(values)
+        else:
+            result[col_name] = values
+    return result
+
+
+def _summarize_array(values):
+    """Summarize an array into a compact description string.
+
+    Numeric arrays → ``"[N numeric values]"``. Year-like string arrays →
+    ``"[N year-like values]"``. Other arrays keep first/last samples and
+    unique count because labels like country codes are useful qualitative
+    keywords for retrieval.
+
+    Quantitative details (min/max/mean) are deliberately omitted: agents
+    that consume this output are instructed to avoid numeric claims, and
+    summary text cannot leak what is not present.
+    """
+    n = len(values)
+
+    if _is_numeric_list(values):
+        return f"[{n} numeric values]"
+    if _looks_like_year_strings(values):
+        return f"[{n} year-like values]"
+
+    first = values[:3]
+    last = values[-3:]
+    try:
+        unique_count = len(set(str(v) for v in values))
+    except Exception:
+        unique_count = "?"
+
+    return f"[{n} values, {unique_count} unique, first: {first}, last: {last}]"

eea_plotly-2.5/eea/plotly/generate.py

@@ -0,0 +1,53 @@
+"""Chart generation using agent system."""
+
+import logging
+
+from plone import api
+
+from eea.genai.core.agent import AgentDeps
+from eea.genai.core.utils import get_executor
+from eea.plotly.controlpanel import IPlotlySettings
+
+logger = logging.getLogger("eea.plotly")
+
+
+def generate_chart(prompt, data_sources=None, context=None, request=None):
+    """Generate a full visualization content from a natural language description."""
+    deps = AgentDeps(
+        context=context, request=request, data_sources=data_sources or None
+    )
+    result = get_executor().run_with_agent(
+        "plotly_generator", user_prompt=prompt, deps=deps
+    )
+    _inject_theme(result)
+    return result.model_dump()
+
+
+def _inject_theme(result):
+    """Replace __ACTIVE_THEME__ placeholder with the actual theme object."""
+    layout = result.visualization.layout
+    if layout.get("template") != "__ACTIVE_THEME__":
+        return
+
+    theme = _get_active_theme()
+    if theme is None:
+        logger.warning("No active theme found; removing template placeholder")
+        layout.pop("template", None)
+        return
+
+    result.visualization.layout["template"] = theme
+
+
+def _get_active_theme():
+    """Return the first non-hidden theme from the Plotly control panel."""
+    try:
+        themes = api.portal.get_registry_record(
+            "themes", interface=IPlotlySettings, default=[]
+        )
+    except Exception:
+        return None
+
+    for theme in themes:
+        if not theme.get("hidden"):
+            return theme
+    return None

eea_plotly-2.5/eea/plotly/models.py

@@ -0,0 +1,89 @@
+"""Pydantic models for structured LLM output of chart operations."""
+
+from pydantic import BaseModel, Field
+
+
+class VisualizationData(BaseModel):
+    """The Plotly chart configuration stored in the visualization field."""
+
+    data: list[dict] = Field(
+        description=(
+            "Ordered list of Plotly trace objects. Each trace must have a "
+            "'type' field, '*src' references to dataSources columns, "
+            "dereferenced inline data arrays, and 'meta.columnNames' mapping."
+        ),
+    )
+
+    layout: dict = Field(
+        description=(
+            "Plotly layout object with title, axis definitions, and "
+            "layout.template set to the active site theme."
+        ),
+        default_factory=dict,
+    )
+
+    dataSources: dict = Field(
+        description=(
+            "Dict of column_name -> array of values. Contains all data "
+            "columns referenced by traces via *src fields."
+        ),
+        default_factory=dict,
+    )
+
+    columns: list[dict] = Field(
+        description=(
+            "List of column metadata objects, each with a 'key' field "
+            "matching a dataSources column name. "
+            'Example: [{"key": "Country"}, {"key": "Value"}]'
+        ),
+        default_factory=list,
+    )
+
+
+class ChartGenerationResult(BaseModel):
+    """Result of LLM visualization content generation.
+
+    Represents a full visualization content type with metadata and chart data.
+    """
+
+    title: str = Field(
+        description="Content title for the visualization page.",
+    )
+
+    description: str = Field(
+        description=(
+            "Short summary describing the visualization (1-3 sentences). "
+            "Used as the page description / Dublin Core description."
+        ),
+        default="",
+    )
+
+    visualization: VisualizationData = Field(
+        description=(
+            "The complete Plotly visualization with traces (data), layout, "
+            "dataSources (raw dataset columns), and columns metadata."
+        ),
+    )
+
+    topics: list[str] = Field(
+        description=(
+            "List of EEA topic tags relevant to this visualization "
+            "(e.g. 'Climate change mitigation', 'Air pollution')."
+        ),
+        default_factory=list,
+    )
+
+    temporal_coverage: list[int] = Field(
+        description=(
+            "List of years covered by the data (e.g. [2018, 2019, 2020, 2021, 2022])."
+        ),
+        default_factory=list,
+    )
+
+    geo_coverage: list[str] = Field(
+        description=(
+            "List of geographic areas covered by the data "
+            "(e.g. ['Europe', 'EU-27', 'Germany'])."
+        ),
+        default_factory=list,
+    )

{eea_plotly-2.3 → eea_plotly-2.5}/eea/plotly/profiles/default/types/visualization.xml

@@ -20,6 +20,7 @@
    <property name="klass">collective.folderishtypes.dx.content.FolderishDocument</property>
    <property name="behaviors">
       <element value="eea.plotly.visualization" />
+      <element value="eea.genai.summary" />
       <element value="eea.coremetadata.behavior" />
       <element value="eea.dataconnector.figure_note" />
       <element value="plone.namefromtitle" />