@panda-agent/panda-cli 0.1.28 → 0.1.30

package/skills/xlsx/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: xlsx
+version: 1.0.1
+description: "Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved."
+description_zh: "当电子表格文件是主要输入或输出时使用此技能。包括：打开、读取、编辑或修复现有的 .xlsx、.xlsm、.csv 或 .tsv 文件（如添加列、计算公式、格式化、图表、清洗数据）；从零或其他数据源创建新电子表格；在表格文件格式之间转换。当用户提及电子表格文件名或路径时触发——即使是随意提及（如\"下载文件夹里的 xlsx\"）——并希望对其进行操作或生成电子表格。也适用于将混乱的表格数据文件（格式错误的行、错位的表头、垃圾数据）清理重组为规范的电子表格。交付物必须是电子表格文件。当主要交付物是 Word 文档、HTML 报告、独立 Python 脚本、数据库管道或 Google Sheets API 集成时，即使涉及表格数据也不要触发。"
+license: Proprietary. LICENSE.txt has complete terms
+---
+
+# Spreadsheet Creation, Editing, and Analysis
+
+You have access to multiple tools and workflows for working with `.xlsx` files — from reading and analysing data, through programmatic creation and editing, to formula recalculation and error checking.
+
+## Tooling Primer
+
+| Library | Best for |
+|---------|----------|
+| **pandas** | Bulk data manipulation, statistical analysis, quick CSV↔XLSX conversion |
+| **openpyxl** | Cell-level formatting, Excel formulas, charts, conditional formatting |
+
+## ⛔ CRITICAL — Use Formulas, Never Hardcode Calculations
+
+> **🚨 MANDATORY RULE — ZERO EXCEPTIONS 🚨**
+>
+> **Every computed value must be an Excel formula, not a Python-calculated literal.**
+> This keeps the workbook dynamic and self-updating.
+> Violations will produce stale, non-updating spreadsheets.
+
+```python
+# ── WRONG — baking Python results into cells ──
+total = df['Sales'].sum()
+ws['B10'] = total          # static 5000
+
+growth = (df.iloc[-1]['Revenue'] - df.iloc[0]['Revenue']) / df.iloc[0]['Revenue']
+ws['C5'] = growth          # static 0.15
+
+avg = sum(vals) / len(vals)
+ws['D20'] = avg            # static 42.5
+```
+
+```python
+# ── RIGHT — let Excel do the maths ──
+ws['B10'] = '=SUM(B2:B9)'
+ws['C5']  = '=(C4-C2)/C2'
+ws['D20'] = '=AVERAGE(D2:D19)'
+```
+
+This rule applies to **all** calculations — sums, ratios, percentages, differences, etc.
+
+## Step-by-Step Workflow
+
+1. **Pick a library** — pandas for data; openpyxl for formatting / formulas.
+2. **Open or create** the workbook.
+3. **Modify** — add/edit data, formulas, and styles.
+4. **Save** to disk.
+5. **Recalculate** (mandatory when formulas are present):
+   ```bash
+   python scripts/recalc.py output.xlsx
+   ```
+6. **Inspect the JSON output** and fix any errors:
+   - `status: "errors_found"` → see `error_summary` for types and locations.
+   - Common errors: `#REF!` (bad reference), `#DIV/0!` (zero denominator), `#VALUE!` (type mismatch), `#NAME?` (unknown function).
+
+## Reading & Analysing Data
+
+```python
+import pandas as pd
+
+df = pd.read_excel('file.xlsx')                        # first sheet
+sheets = pd.read_excel('file.xlsx', sheet_name=None)   # all sheets → dict
+
+df.head(); df.info(); df.describe()                    # quick overview
+
+df.to_excel('result.xlsx', index=False)                # write back
+```
+
+## Building New Workbooks
+
+```python
+from openpyxl import Workbook
+from openpyxl.styles import Font, PatternFill, Alignment
+
+wb = Workbook()
+ws = wb.active
+
+ws['A1'] = 'Header'
+ws.append(['Row', 'of', 'data'])
+ws['B2'] = '=SUM(A1:A10)'
+
+ws['A1'].font = Font(bold=True, color='FF0000')
+ws['A1'].fill = PatternFill('solid', start_color='FFFF00')
+ws['A1'].alignment = Alignment(horizontal='center')
+ws.column_dimensions['A'].width = 20
+
+wb.save('output.xlsx')
+```
+
+## Modifying Existing Files
+
+```python
+from openpyxl import load_workbook
+
+wb = load_workbook('existing.xlsx')
+ws = wb.active                           # or wb['SheetName']
+
+for name in wb.sheetnames:
+    print("Sheet: {}".format(name))
+
+ws['A1'] = 'Updated'
+ws.insert_rows(2)
+ws.delete_cols(3)
+
+extra = wb.create_sheet('Extra')
+extra['A1'] = 'New data'
+
+wb.save('modified.xlsx')
+```
+
+## Formula Recalculation
+
+Workbooks produced by openpyxl contain formula *strings* but no cached results.  Use the bundled helper to populate those values:
+
+```bash
+python scripts/recalc.py <excel_file> [timeout_seconds]
+```
+
+**What it does:**
+- Deploys a LibreOffice Basic macro (first run only)
+- Invokes LibreOffice headless to recalculate every formula
+- Scans all cells for Excel error markers
+- Emits structured JSON
+
+**Prerequisite:** LibreOffice must be installed.  The helper handles first-run configuration automatically, including sandboxed environments where Unix sockets are restricted (via `scripts/office/soffice.py`).
+
+### Interpreting the Output
+
+```json
+{
+  "status": "success",
+  "total_errors": 0,
+  "total_formulas": 42,
+  "error_summary": {}
+}
+```
+
+When `status` is `"errors_found"`, `error_summary` lists each error type with count and cell locations (up to 20 per type).
+
+---
+
+# Output Quality Standards
+
+## General — All Workbooks
+
+| Area | Requirement |
+|------|-------------|
+| **Typography** | Use a single professional typeface (Arial, Times New Roman, …) throughout, unless the user specifies otherwise |
+| **Error-free delivery** | Zero formula errors — no `#REF!`, `#DIV/0!`, `#VALUE!`, `#N/A`, `#NAME?` |
+| **Template fidelity** | When editing an existing file, study and replicate its formatting conventions exactly; never impose a different style on an already-patterned workbook |
+
+## Financial Models
+
+### Colour Conventions (override only when the user or template says otherwise)
+
+| Element | RGB | Hex | Meaning |
+|---------|-----|-----|---------|
+| Blue text | `(0,0,255)` | `#0000FF` | Hard-coded inputs / scenario toggles |
+| Black text | `(0,0,0)` | `#000000` | All formulas and calculated values |
+| Green text | `(0,128,0)` | `#008000` | Cross-sheet references within the same workbook |
+| Red text | `(255,0,0)` | `#FF0000` | External links to other files |
+| Yellow fill | `(255,255,0)` | `#FFFF00` | Key assumptions or cells requiring review |
+
+### Number Formatting
+
+| Data Type | Excel Format Code | Display Example | Notes |
+|-----------|-------------------|-----------------|-------|
+| Calendar years | _(plain text)_ | `2024` | Never `2,024` — no thousands separator |
+| Currency | `$#,##0` | `$1,250` | Always label units in headers, e.g. _Revenue ($mm)_ |
+| Zero values | `$#,##0;($#,##0);-` | `-` | Custom three-section format |
+| Percentages | `0.0%` | `12.5%` | One decimal place |
+| Multiples | `0.0x` | `3.2x` | For EV/EBITDA, P/E, etc. |
+| Negatives | `(#,##0)` | `(123)` | Parenthesised, never `-123` |
+
+### Formula Best Practices
+
+- **Centralise assumptions** — growth rates, margins, multiples belong in labelled assumption cells; formulas should reference those cells, not embed literals.
+  ```
+  =B5*(1+$B$6)   ✓
+  =B5*1.05        ✗
+  ```
+- **Prevent errors** — verify references, check range boundaries, confirm consistent formulas across projection periods, test edge cases.
+- **No circular references** — unless explicitly designed and documented.
+- **Document hard-codes** — add a cell comment or adjacent note:
+  `Source: Company 10-K, FY2024, Page 45, Revenue Note, [SEC EDGAR URL]`
+
+## Formula Verification Checklist
+
+- Spot-check 2–3 references before building the full model.
+- Confirm column mapping (column 64 → BL, not BK).
+- Remember row offsets (DataFrame row 5 = Excel row 6).
+- Guard against `NaN` — use `pd.notna()`.
+- Test far-right columns (FY data often sits in column 50+).
+- Handle multiple matches — search all occurrences, not just the first.
+
+### Testing Strategy
+
+1. Validate formulas on a small range first.
+2. Verify every referenced cell exists.
+3. Include zero, negative, and very large values.
+
+## Code Style
+
+When generating Python that manipulates spreadsheets:
+- Keep code concise — no verbose names, no gratuitous comments.
+- Skip unnecessary `print()` calls.
+
+For the workbook itself:
+- Comment cells that contain complex formulas or key assumptions.
+- Cite data sources for every hard-coded figure.
+- Add section headers and notes for major model blocks.
+
+## Library Tips
+
+### openpyxl
+
+- Indices are **1-based** — `(row=1, column=1)` is cell A1.
+- `data_only=True` reads cached values; **warning:** saving afterward strips all formulas permanently.
+- For large files: `read_only=True` (reading) or `write_only=True` (writing).
+- Formulas are stored as strings and require `scripts/recalc.py` to populate values.
+
+### pandas
+
+- Specify dtypes to avoid inference surprises: `pd.read_excel('f.xlsx', dtype={'id': str})`.
+- Limit columns on large files: `usecols=['A', 'C', 'E']`.
+- Parse dates explicitly: `parse_dates=['date_column']`.

package/skills/xlsx/scripts/office/helpers/__init__.py

@@ -0,0 +1 @@
+# helpers sub-package (intentionally empty)

package/skills/xlsx/scripts/office/helpers/merge_runs.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+# ──────────────────────────────────────────────────────────────────
+# Coalesce adjacent <w:r> elements sharing identical <w:rPr> in DOCX.
+#
+# Pre-processing steps that enable merging:
+#   1. Strip all proofErr elements (spell/grammar markers)
+#   2. Remove rsid* attributes from runs (revision metadata)
+#
+# After merging, adjacent <w:t> children inside the same run are
+# concatenated into a single text node.
+# ──────────────────────────────────────────────────────────────────
+
+from pathlib import Path
+
+import defusedxml.minidom
+
+
+def merge_runs(input_dir: str) -> tuple[int, str]:
+    """Entry point – merge runs in word/document.xml and return (count, msg)."""
+    doc = Path(input_dir) / "word" / "document.xml"
+
+    if not doc.exists():
+        return 0, "Error: {} not found".format(doc)
+
+    try:
+        tree = defusedxml.minidom.parseString(doc.read_text(encoding="utf-8"))
+        top = tree.documentElement
+
+        # housekeeping
+        _purge_by_tag(top, "proofErr")
+        _drop_rsid_attrs(top)
+
+        # collect unique parent containers of all <w:r>
+        parents = {nd.parentNode for nd in _query_tag(top, "r")}
+
+        merged = 0
+        for p in parents:
+            merged += _coalesce_in_container(p)
+
+        doc.write_bytes(tree.toxml(encoding="UTF-8"))
+        return merged, "Merged {} runs".format(merged)
+
+    except Exception as ex:
+        return 0, "Error: {}".format(ex)
+
+
+# ──────────────────────────────────────────────────────────────────
+# DOM traversal helpers
+# ──────────────────────────────────────────────────────────────────
+
+def _query_tag(root, local_name: str) -> list:
+    """Recursively find all elements whose local name matches *local_name*."""
+    hits = []
+
+    def _walk(nd):
+        if nd.nodeType != nd.ELEMENT_NODE:
+            return
+        tag = nd.localName or nd.tagName
+        if tag == local_name or tag.endswith(":{}".format(local_name)):
+            hits.append(nd)
+        for ch in nd.childNodes:
+            _walk(ch)
+
+    _walk(root)
+    return hits
+
+
+def _child_by_tag(parent, local_name: str):
+    """Return the first direct child element matching *local_name*."""
+    for ch in parent.childNodes:
+        if ch.nodeType != ch.ELEMENT_NODE:
+            continue
+        tag = ch.localName or ch.tagName
+        if tag == local_name or tag.endswith(":{}".format(local_name)):
+            return ch
+    return None
+
+
+def _children_by_tag(parent, local_name: str) -> list:
+    """Return every direct child element matching *local_name*."""
+    return [
+        ch for ch in parent.childNodes
+        if ch.nodeType == ch.ELEMENT_NODE
+        and ((ch.localName or ch.tagName) == local_name
+             or (ch.localName or ch.tagName).endswith(":{}".format(local_name)))
+    ]
+
+
+def _only_whitespace_between(a, b) -> bool:
+    """True when nothing meaningful sits between siblings *a* and *b*."""
+    cur = a.nextSibling
+    while cur is not None:
+        if cur is b:
+            return True
+        if cur.nodeType == cur.ELEMENT_NODE:
+            return False
+        if cur.nodeType == cur.TEXT_NODE and cur.data.strip():
+            return False
+        cur = cur.nextSibling
+    return False
+
+
+# ──────────────────────────────────────────────────────────────────
+# Cleanup passes
+# ──────────────────────────────────────────────────────────────────
+
+def _purge_by_tag(root, local_name: str):
+    """Remove every element whose local name matches *local_name*."""
+    for nd in _query_tag(root, local_name):
+        if nd.parentNode is not None:
+            nd.parentNode.removeChild(nd)
+
+
+def _drop_rsid_attrs(root):
+    """Strip revision-save-ID attributes from all <w:r> elements."""
+    for r in _query_tag(root, "r"):
+        doomed = [a for a in r.attributes.values() if "rsid" in a.name.lower()]
+        for a in doomed:
+            r.removeAttribute(a.name)
+
+
+# ──────────────────────────────────────────────────────────────────
+# Core merging logic
+# ──────────────────────────────────────────────────────────────────
+
+def _tag_is_run(nd) -> bool:
+    tag = nd.localName or nd.tagName
+    return tag == "r" or tag.endswith(":r")
+
+
+def _next_elem(nd):
+    """Return the next element sibling (skip text/comment nodes)."""
+    s = nd.nextSibling
+    while s is not None:
+        if s.nodeType == s.ELEMENT_NODE:
+            return s
+        s = s.nextSibling
+    return None
+
+
+def _next_run_sibling(nd):
+    """Walk forward until we hit another <w:r> element."""
+    s = nd.nextSibling
+    while s is not None:
+        if s.nodeType == s.ELEMENT_NODE and _tag_is_run(s):
+            return s
+        s = s.nextSibling
+    return None
+
+
+def _first_run_child(container):
+    """Return the first child that is a run element."""
+    for ch in container.childNodes:
+        if ch.nodeType == ch.ELEMENT_NODE and _tag_is_run(ch):
+            return ch
+    return None
+
+
+def _runs_compatible(a, b) -> bool:
+    """Two runs are compatible when their <w:rPr> serialisations match."""
+    rpr_a = _child_by_tag(a, "rPr")
+    rpr_b = _child_by_tag(b, "rPr")
+    if (rpr_a is None) != (rpr_b is None):
+        return False
+    return True if rpr_a is None else rpr_a.toxml() == rpr_b.toxml()
+
+
+def _absorb_run(dst, src):
+    """Move non-rPr children from *src* into *dst*."""
+    for ch in list(src.childNodes):
+        if ch.nodeType != ch.ELEMENT_NODE:
+            continue
+        tag = ch.localName or ch.tagName
+        if tag == "rPr" or tag.endswith(":rPr"):
+            continue
+        dst.appendChild(ch)
+
+
+def _squash_text_nodes(run):
+    """Concatenate adjacent <w:t> children into one."""
+    t_nodes = _children_by_tag(run, "t")
+
+    idx = len(t_nodes) - 1
+    while idx > 0:
+        cur, prev = t_nodes[idx], t_nodes[idx - 1]
+
+        if _only_whitespace_between(prev, cur):
+            txt_prev = prev.firstChild.data if prev.firstChild else ""
+            txt_cur = cur.firstChild.data if cur.firstChild else ""
+            combined = txt_prev + txt_cur
+
+            if prev.firstChild:
+                prev.firstChild.data = combined
+            else:
+                prev.appendChild(run.ownerDocument.createTextNode(combined))
+
+            if combined.startswith(" ") or combined.endswith(" "):
+                prev.setAttribute("xml:space", "preserve")
+            elif prev.hasAttribute("xml:space"):
+                prev.removeAttribute("xml:space")
+
+            run.removeChild(cur)
+
+        idx -= 1
+
+
+def _coalesce_in_container(container) -> int:
+    """Merge compatible adjacent runs inside *container*."""
+    count = 0
+    cur = _first_run_child(container)
+
+    while cur is not None:
+        # absorb as many consecutive compatible runs as possible
+        while True:
+            nxt = _next_elem(cur)
+            if nxt is not None and _tag_is_run(nxt) and _runs_compatible(cur, nxt):
+                _absorb_run(cur, nxt)
+                container.removeChild(nxt)
+                count += 1
+            else:
+                break
+
+        _squash_text_nodes(cur)
+        cur = _next_run_sibling(cur)
+
+    return count

package/skills/xlsx/scripts/office/helpers/simplify_redlines.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+# ──────────────────────────────────────────────────────────────────
+# Merge adjacent tracked-change wrappers (<w:ins> / <w:del>) when they
+# share the same author.  Reduces visual clutter in heavily-redlined
+# DOCX documents without altering semantics.
+#
+# Constraints:
+#   • Only merges elements of the *same* tag (ins↔ins, del↔del)
+#   • Author must match (timestamps are ignored)
+#   • Only merges truly adjacent elements (whitespace-only gap allowed)
+# ──────────────────────────────────────────────────────────────────
+
+import xml.etree.ElementTree as ET
+import zipfile
+from pathlib import Path
+
+import defusedxml.minidom
+
+_WML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
+
+
+def simplify_redlines(input_dir: str) -> tuple[int, str]:
+    """Merge adjacent tracked changes in word/document.xml."""
+    doc = Path(input_dir) / "word" / "document.xml"
+
+    if not doc.exists():
+        return 0, "Error: {} not found".format(doc)
+
+    try:
+        tree = defusedxml.minidom.parseString(doc.read_text(encoding="utf-8"))
+        top = tree.documentElement
+
+        total = 0
+        for box in _collect_elements(top, "p") + _collect_elements(top, "tc"):
+            total += _coalesce_tracked(box, "ins")
+            total += _coalesce_tracked(box, "del")
+
+        doc.write_bytes(tree.toxml(encoding="UTF-8"))
+        return total, "Simplified {} tracked changes".format(total)
+
+    except Exception as ex:
+        return 0, "Error: {}".format(ex)
+
+
+# ──────────────────────────────────────────────────────────────────
+# Internal helpers
+# ──────────────────────────────────────────────────────────────────
+
+def _coalesce_tracked(container, kind: str) -> int:
+    """Merge adjacent <w:ins> or <w:del> elements inside *container*."""
+    nodes = [
+        ch for ch in container.childNodes
+        if ch.nodeType == ch.ELEMENT_NODE and _matches_tag(ch, kind)
+    ]
+    if len(nodes) < 2:
+        return 0
+
+    merged = 0
+    pos = 0
+    while pos < len(nodes) - 1:
+        this, that = nodes[pos], nodes[pos + 1]
+        if _same_author_adjacent(this, that):
+            _move_children(this, that)
+            container.removeChild(that)
+            nodes.pop(pos + 1)
+            merged += 1
+        else:
+            pos += 1
+
+    return merged
+
+
+def _matches_tag(nd, local: str) -> bool:
+    tag = nd.localName or nd.tagName
+    return tag == local or tag.endswith(":{}".format(local))
+
+
+def _author_of(elem) -> str:
+    """Extract the w:author attribute value from a tracked-change element."""
+    val = elem.getAttribute("w:author")
+    if val:
+        return val
+    for attr in elem.attributes.values():
+        if attr.localName == "author" or attr.name.endswith(":author"):
+            return attr.value
+    return ""
+
+
+def _same_author_adjacent(a, b) -> bool:
+    """True when *a* and *b* share an author and have no elements between them."""
+    if _author_of(a) != _author_of(b):
+        return False
+    cur = a.nextSibling
+    while cur is not None and cur is not b:
+        if cur.nodeType == cur.ELEMENT_NODE:
+            return False
+        if cur.nodeType == cur.TEXT_NODE and cur.data.strip():
+            return False
+        cur = cur.nextSibling
+    return True
+
+
+def _move_children(dst, src):
+    """Transplant every child of *src* into *dst*."""
+    while src.firstChild is not None:
+        kid = src.firstChild
+        src.removeChild(kid)
+        dst.appendChild(kid)
+
+
+def _collect_elements(root, local: str) -> list:
+    """Depth-first collection of elements matching *local*."""
+    out = []
+
+    def _dfs(nd):
+        if nd.nodeType == nd.ELEMENT_NODE:
+            tag = nd.localName or nd.tagName
+            if tag == local or tag.endswith(":{}".format(local)):
+                out.append(nd)
+            for ch in nd.childNodes:
+                _dfs(ch)
+
+    _dfs(root)
+    return out
+
+
+# ──────────────────────────────────────────────────────────────────
+# Author-analysis utilities (used by pack.py / infer_author)
+# ──────────────────────────────────────────────────────────────────
+
+def get_tracked_change_authors(xml_path: Path) -> dict[str, int]:
+    """Count tracked-change occurrences per author in an XML file."""
+    if not xml_path.exists():
+        return {}
+    try:
+        parsed = ET.parse(xml_path)
+    except ET.ParseError:
+        return {}
+
+    ns = {"w": _WML_NS}
+    attr_key = "{{{}}}author".format(_WML_NS)
+
+    counts: dict[str, int] = {}
+    for tag in ("ins", "del"):
+        for el in parsed.getroot().findall(".//w:{}".format(tag), ns):
+            who = el.get(attr_key)
+            if who:
+                counts[who] = counts.get(who, 0) + 1
+    return counts
+
+
+def _get_authors_from_docx(docx: Path) -> dict[str, int]:
+    """Extract author counts from a packed .docx without full unpacking."""
+    try:
+        with zipfile.ZipFile(docx, "r") as zf:
+            if "word/document.xml" not in zf.namelist():
+                return {}
+            with zf.open("word/document.xml") as fh:
+                parsed = ET.parse(fh)
+                ns = {"w": _WML_NS}
+                attr_key = "{{{}}}author".format(_WML_NS)
+                counts: dict[str, int] = {}
+                for tag in ("ins", "del"):
+                    for el in parsed.getroot().findall(".//w:{}".format(tag), ns):
+                        who = el.get(attr_key)
+                        if who:
+                            counts[who] = counts.get(who, 0) + 1
+                return counts
+    except (zipfile.BadZipFile, ET.ParseError):
+        return {}
+
+
+def infer_author(modified_dir: Path, original_docx: Path, default: str = "Claude") -> str:
+    """Guess which author introduced new tracked changes."""
+    mod_xml = modified_dir / "word" / "document.xml"
+    mod_counts = get_tracked_change_authors(mod_xml)
+
+    if not mod_counts:
+        return default
+
+    orig_counts = _get_authors_from_docx(original_docx)
+
+    delta: dict[str, int] = {}
+    for who, n in mod_counts.items():
+        diff = n - orig_counts.get(who, 0)
+        if diff > 0:
+            delta[who] = diff
+
+    if not delta:
+        return default
+
+    if len(delta) == 1:
+        return next(iter(delta))
+
+    raise ValueError(
+        "Multiple authors added new changes: {}. "
+        "Cannot infer which author to validate.".format(delta)
+    )