@fectivnfy112358/minimax-code-web 0.1.0-preview.0 → 0.1.0-preview.3

package/server/assets/skills/xlsx/docs/raw-xml-escape-hatch.md

@@ -0,0 +1,201 @@
+# Raw-XML Escape Hatch
+
+> Distilled from spreadsheet interoperability lessons; supersedes the
+> deprecated legacy `xlsx` skill bundle. The library-first
+> workflow in `SKILL.md` covers >90% of edits — this doc is for the residual.
+
+`openpyxl` round-trips real-world workbooks far better than its detractors
+claim: charts, conditional formatting, named ranges, merged cells, frozen
+panes, and (with `keep_vba=True`) VBA all survive a `load_workbook` →
+`save` cycle. Reach for the surgical XML path **only** when the workbook
+contains one of the six artefacts listed in §1; everything else stays in
+the library cookbook.
+
+---
+
+## 1. When to fall back to raw XML
+
+The following six categories are the genuine failure modes. In each case
+`openpyxl` either drops the part on save or rewrites it with semantics
+the consuming application no longer accepts.
+
+| Artefact (path inside the .xlsx zip) | Where it appears | What `openpyxl` actually does on save | Why XML edit is the only safe path |
+|---|---|---|---|
+| `xl/vbaProject.bin` | Workbook saved as `.xlsm` with macros / form events | Dropped entirely unless the file was opened with `keep_vba=True`; even then any new sheet may shift cell-level event bindings | Binary VBA bytecode; one byte off and the macro project fails to load |
+| `xl/pivotCaches/pivotCacheDefinition*.xml` + `xl/pivotTables/*.xml` | Pivot tables built from worksheet ranges or external data | Pivot table objects are dropped; pivot caches lose `<cacheSource>` ref tracking | The cache and the table are tightly coupled — editing one without the other corrupts the pivot |
+| `xl/slicers/*.xml` (+ `xl/slicerCaches/*.xml`) | Slicers wired to a pivot or table | Slicer XML is silently removed; pivot still loads but UI filter is gone | Slicers reference pivot cache IDs by GUID; rewriting the parts without preserving GUIDs breaks the binding |
+| `xl/connections.xml` | Workbooks with external data connections (Power Query, ODBC, OLAP) | Connection node is dropped; data refresh button does nothing | The connection definition is the only source of truth for the refresh URL / DSN |
+| `xl/externalLinks/` (incl. `externalLink*.bin`) | `=[OtherBook.xlsx]Sheet!A1` style cross-file references | External link XML retained, `.bin` cache is lost; Excel re-prompts the user for every linked file on open | The `.bin` is a binary cache the format treats as opaque |
+| Form controls / drawing shapes / advanced conditional formatting (color-scale + icon-set composites) | Workbooks with ActiveX form controls, `xl/drawings/drawing*.xml` shape art, multi-rule color-scale + icon-set on the same range | Form controls dropped (they live in `xl/activeX/` plus a paired `.bin`); shapes preserved but bound `linkedCell` may detach; multi-rule conditional formatting collapses into the first rule | None of these are first-class openpyxl objects |
+
+Anything not in this table — column insertion, formula edits, style
+changes, named-range tweaks, merged-cell expansion, chart series ranges
+(via `openpyxl.chart`), simple conditional formatting (`openpyxl.formatting`)
+— belongs in the library cookbook (see `SKILL.md` §3 and
+`docs/create-edit-guide.md`).
+
+---
+
+## 2. The escape-hatch toolchain
+
+Two scripts live at `scripts/office/` and are shared with the docx /
+pptx skills (the OOXML zip layout is identical for all three formats):
+
+| Script | Role | Notes |
+|---|---|---|
+| `scripts/office/unpack.py` | Unzip into a working directory and pretty-print every `*.xml` / `*.rels` so diffs read cleanly | Prints an inventory + a warning when high-risk parts are detected |
+| `scripts/office/pack.py` | Re-zip the working directory back into a valid `.xlsx`, validating XML well-formedness before sealing the archive | Refuses to pack if any XML fails to parse |
+| `scripts/office/validate.py` | Run XSD validation against the bundled OOXML schemas | Flags malformed parts the parsers will choke on |
+
+Standard CLI workflow:
+
+```bash
+python scripts/office/unpack.py input.xlsx /tmp/work/
+# ... edit raw XML files in /tmp/work/xl/ ...
+python scripts/office/pack.py /tmp/work/ output.xlsx
+python scripts/office/validate.py output.xlsx                    # optional XSD pass
+python scripts/recalc.py output.xlsx 60                          # mandatory before delivery
+```
+
+The `pack.py` script catches the most common foot-guns (malformed XML,
+missing `[Content_Types].xml` entries) but it does **not** semantically
+validate the artefacts in §1 — that responsibility stays with the
+caller.
+
+---
+
+## 3. Surgical edit recipe — rename a sheet
+
+Sheet rename is the cleanest demonstrable example: it touches exactly
+one attribute on one part, and any cross-sheet formula referring to the
+old name needs the same rename. Anything more complex (column insertion,
+row shift) belongs in `openpyxl` — they are the very operations the
+deprecated `xlsx` helper scripts implemented and got wrong.
+
+```bash
+python scripts/office/unpack.py books.xlsx /tmp/books/
+```
+
+Edit `/tmp/books/xl/workbook.xml`:
+
+```xml
+<!-- before -->
+<sheet name="Sheet1" sheetId="1" r:id="rId1"/>
+
+<!-- after -->
+<sheet name="Revenue" sheetId="1" r:id="rId1"/>
+```
+
+If any other worksheet referenced the old name in a formula, update
+those `<f>` nodes too:
+
+```bash
+grep -rn 'Sheet1!' /tmp/books/xl/worksheets/
+```
+
+```xml
+<!-- before -->
+<c r="B5"><f>Sheet1!C10</f><v></v></c>
+
+<!-- after -->
+<c r="B5"><f>Revenue!C10</f><v></v></c>
+```
+
+Repack and recalc:
+
+```bash
+python scripts/office/pack.py /tmp/books/ books.xlsx
+python scripts/recalc.py books.xlsx 60
+```
+
+Rules of thumb that apply to every surgical edit:
+
+- Touch the minimum number of nodes. Never rewrite a whole part you do not need to change.
+- Preserve `sheetId` and every `r:id` — these are stable internal identifiers.
+- Sheet names in formulas are case-sensitive and need single-quoting if they contain spaces (`'Q1 Data'!B5`).
+- Update `<dimension ref="...">` if you extended the row / column range; reviewers and pivot caches consult it.
+
+---
+
+## 4. Files you must never touch
+
+These six entries are absolute no-go zones for raw XML edits — the
+artefacts are either binary or so tightly coupled that there is no safe
+in-place patch. If a delivery requires modifying any of them, change
+the upstream architecture (regenerate the workbook, replace the link
+with a Power Query, etc.) instead of editing bytes.
+
+| File / location | Why it is unfixable in-place |
+|---|---|
+| `xl/vbaProject.bin` | Binary VBA bytecode; any modification corrupts the macro project |
+| `xl/pivotCaches/pivotCacheDefinition*.xml` | The cache definition ties the pivot to its source data; editing without a paired `pivotTable*.xml` update corrupts the pivot |
+| `xl/pivotTables/*.xml` | Tightly coupled to its cache definition and to internal state Excel rebuilds on load |
+| `xl/slicers/*.xml` | Slicers are connected to specific cache IDs and pivot fields; breaking those connections silently corrupts the file |
+| `xl/connections.xml` | External data connections; editing breaks live data refresh |
+| `xl/externalLinks/` (`*.bin` included) | External workbook links; the binary `.bin` cache files in here must not be modified |
+
+If you need to "fix" any of the above, the answer is to rebuild the
+workbook from a clean source — never to hand-edit bytes.
+
+---
+
+## 5. Files you may touch with constraints
+
+The following parts can be edited surgically, but only specific
+attributes are safe — every other byte must remain untouched.
+
+| File | What you may update | What to leave alone |
+|---|---|---|
+| `xl/charts/chartN.xml` | Data series range references (`<numRef><f>`) after a row / column shift | Chart type, formatting, layout, axis configuration |
+| `xl/tables/tableN.xml` | `ref` attribute on `<table>` after appending rows | Column definitions, style info, header row metadata |
+| `xl/pivotCaches/pivotCacheDefinition*.xml` | `ref` attribute on `<cacheSource><worksheetSource>` after shifting source data | All other content (cache fields, refresh metadata, GUIDs) |
+
+Any edit that strays beyond the "may update" column lands you back in
+§4 territory.
+
+---
+
+## 6. Validation after a raw-XML edit
+
+Always run the full pipeline before delivering an XML-edited workbook —
+the XSD validator + a real recalculation are the only way to catch
+subtle malformations.
+
+```bash
+python scripts/office/pack.py /tmp/work/ output.xlsx
+python scripts/office/validate.py output.xlsx                  # XSD well-formedness
+python scripts/recalc.py output.xlsx 60                        # LibreOffice headless
+```
+
+`recalc.py` returns JSON with `status`, `total_errors`, and
+`total_formulas`. A delivery is acceptable only when `total_errors == 0`.
+If `validate.py` flags a malformed part, fix the XML and repeat — do not
+ship a workbook that fails XSD validation.
+
+For the `recalc.py` JSON schema, the seven Excel error markers, and the
+AF_UNIX sandbox shim, see [`docs/recalc-guide.md`](recalc-guide.md).
+
+---
+
+## 7. When NOT to use this
+
+The deprecated `xlsx` cookbook reached for raw-XML edits as a
+default; that policy was driven by the now-disproven claim that
+"openpyxl corrupts complex workbooks". In practice, **>90% of xlsx edits
+the agent is asked to perform are one to three lines of `openpyxl`**.
+
+Reach for the escape hatch only when the workbook genuinely contains an
+artefact from §1; reach for the library cookbook in every other case.
+
+| Common task | Library-first answer (use this) | Why escape-hatch is wrong here |
+|---|---|---|
+| Add a column at the end of a sheet | `sheet.cell(row=r, column=last+1, value=...)` then `book.save()` | XML insertion has to keep `<dimension>`, `<cols>`, every cell `r=` attribute, conditional-formatting `sqref`, table `ref`, and chart series ranges in sync; `openpyxl` does it for you |
+| Insert a row in the middle | `sheet.insert_rows(idx)` then refill the new row, then `book.save()` | The deprecated `xlsx_insert_row.py` had a silent `FileNotFoundError` on openpyxl-produced files (`os.path.join(work_dir, "xl", rel.get("Target"))` dropped `work_dir` when `Target` started with `/`); same root-cause bug in `xlsx_shift_rows.py` and `xlsx_add_column.py` |
+| Set cell values from a list | `for r, row in enumerate(rows, start=2): for c, v in enumerate(row, start=1): sheet.cell(row=r, column=c, value=v)` | XML write requires sharedStrings table updates with paired `count` / `uniqueCount` increments — easy to get wrong |
+| Restyle a range (font, fill, border) | `cell.font = Font(...)`, `cell.fill = PatternFill(...)`, etc. | XML restyle requires appending `<font>` / `<fill>` / `<border>` / `<xf>` entries with paired `count` updates and a fresh `cellXfs` index |
+| Add a chart | `openpyxl.chart.LineChart()` + `add_data` + `add_chart` | Chart XML has thirty-plus required nodes; hand-writing them is a waste of context |
+| Add conditional formatting (single rule) | `openpyxl.formatting.rule.CellIsRule(...)` + `add` | Multi-rule color-scale + icon-set composites belong to §1 — but a single rule is a one-line library call |
+| Recalculate after edits | `python scripts/recalc.py output.xlsx 60` | Already covered by `SKILL.md` §4 |
+
+If you find yourself reaching for `unpack.py` to do any of the rows in
+that table, stop and rewrite the operation in `openpyxl` first.

package/server/assets/skills/xlsx/docs/recalc-guide.md

@@ -0,0 +1,295 @@
+# recalc-guide — `scripts/recalc.py` reference
+
+> Detailed reference for the only wrapped script in this skill —
+> `scripts/recalc.py`, which drives LibreOffice headless to recompute every
+> formula in a workbook and then walks the result with openpyxl looking for
+> the seven Excel error markers. The minimal "what to type" lives in
+> [`SKILL.md`](../SKILL.md) §4.1; this file is the place for the macro path,
+> JSON schema, AF_UNIX shim, cross-platform timeout, and the offline
+> alternatives.
+
+## 1. Command
+
+```bash
+python scripts/recalc.py <workbook> [timeout_seconds]
+```
+
+`workbook` is positional; `timeout_seconds` is optional and defaults to 60.
+Exit codes:
+
+| Code | Meaning |
+|---|---|
+| `0` | LibreOffice ran to completion. The JSON on stdout decides success vs `errors_found` vs hard error. |
+| non-zero | The wrapper itself failed (LibreOffice missing, file unreadable, timeout exceeded). The JSON on stdout will be `{"error": "..."}`. |
+
+The wrapped soffice invocation already saves the recalculated values back
+to the input path — there is no `--out` flag and no temp file dance.
+
+## 2. JSON output schema
+
+Three shapes, exactly one of which appears on stdout per run.
+
+### 2.1 Success path
+
+```json
+{
+  "status": "success",
+  "total_errors": 0,
+  "total_formulas": 42
+}
+```
+
+### 2.2 `errors_found` path
+
+```json
+{
+  "status": "errors_found",
+  "total_errors": 4,
+  "total_formulas": 42,
+  "error_summary": {
+    "#REF!": {
+      "count": 2,
+      "locations": ["MAU_Forecast!B5", "MAU_Forecast!C10"]
+    },
+    "#DIV/0!": {
+      "count": 2,
+      "locations": ["Assumptions!E3", "Assumptions!E4"]
+    }
+  }
+}
+```
+
+`locations` is capped at 20 per error class; if the class has more, the
+list is truncated and `count` reflects the full total.
+
+### 2.3 Hard-error path
+
+```json
+{
+  "error": "soffice timed out after 60 seconds"
+}
+```
+
+Downstream callers should branch on `error` first (short-circuit on hard
+failure) and only then on `status`.
+
+## 3. The seven Excel error classes
+
+Alphabetical, matching the tokens scanned in `scripts/recalc.py`:
+
+| Marker | Diagnosis | First fix |
+|---|---|---|
+| `#DIV/0!` | Denominator evaluated to zero (or empty) | Wrap with `IFERROR(num/denom, 0)` or `IF(denom=0, "", num/denom)`; verify the upstream feed is not zero-padded |
+| `#N/A` | Lookup target missing in `VLOOKUP` / `MATCH` / `XLOOKUP` | Check key whitespace, case, type (`123` vs `"123"`); confirm the lookup column is contiguous |
+| `#NAME?` | Function name typo or missing add-in | Check spelling (`SUMM` / `VLOKUP`); LibreOffice ships add-ins separately from MS Excel |
+| `#NULL!` | Range intersection is empty (a literal space was used between two ranges) | Replace the space with a comma or colon; run `recalc.py` again to confirm |
+| `#NUM!` | Numeric out of bounds (`LOG(-1)`, IRR not converging, integer overflow) | Inspect the function args; clamp or use `IFERROR` for known boundary cases |
+| `#REF!` | A referenced cell or sheet was deleted | Search the workbook for the broken reference; never bulk-delete columns without first running `Find: =` to enumerate dependents |
+| `#VALUE!` | Type mismatch (text where a number is expected, or vice versa) | Coerce with `VALUE()` / `TEXT()`; for batch fixes set `engine="openpyxl"` and convert the column with pandas before resaving |
+
+Each class only ever appears in `error_summary` if at least one cell on
+the sheet contains the literal marker — recalc cannot infer "this looks
+like it might be wrong" beyond the seven tokens.
+
+## 4. LibreOffice macro install
+
+`recalc.py` ships a tiny LibreOffice Basic module that wraps
+`ThisComponent.calculateAll() + store() + close(True)`. On the first run
+the script writes the module to the user profile and never touches it
+again:
+
+| Platform | Path |
+|---|---|
+| Linux | `~/.config/libreoffice/4/user/basic/Standard/Module1.xba` |
+| macOS | `~/Library/Application Support/LibreOffice/4/user/basic/Standard/Module1.xba` |
+
+The XML payload is byte-fixed inside `recalc.py` — keep it untouched.
+LibreOffice resolves the macro by literal name (`Standard.Module1
+.RecalculateAndSave`); any rename breaks the call site.
+
+If the macro file already exists with the expected XML, `install_recalc
+_basic()` is a no-op — safe to re-run.
+
+## 5. Cross-platform timeout
+
+`recalc.py` wraps the soffice command with the platform-specific timeout
+binary so a hung LibreOffice never blocks the caller forever:
+
+| Platform | Wrapper |
+|---|---|
+| Linux | `timeout <sec> soffice ...` (`coreutils` is always present) |
+| macOS with `coreutils` | `gtimeout <sec> soffice ...` (`brew install coreutils`) |
+| macOS without `coreutils` | no wrapper; the wall-clock timeout falls back to `subprocess.run(timeout=…)` |
+
+The third path still terminates the soffice process tree, but the JSON
+on stdout is the Python wrapper's `{"error": "..."}` rather than a
+`gtimeout`-style 124 exit code. If you parse the timeout reason
+upstream, branch on the JSON, not the exit code.
+
+## 6. AF_UNIX sandbox shim (`scripts/office/soffice.py`)
+
+LibreOffice opens an internal AF_UNIX socket pair to coordinate the
+headless backend with the Basic interpreter. Some sandboxed
+environments deny AF_UNIX entirely:
+
+- macOS App Sandbox (Mavis daemon spawned from a sandboxed parent)
+- Linux containers with seccomp filters that block `socket(AF_UNIX,...)`
+
+`office/soffice.py::get_soffice_env()` probes for the restriction (one
+quick `socket.socket(AF_UNIX, SOCK_STREAM)` attempt) and, if blocked,
+compiles a small C shim into `/tmp/lo_socket_shim.so` and sets
+`LD_PRELOAD` on the env it returns. The shim intercepts `socket(AF_UNIX,
+...)` and serves it from a `socketpair()` instead — transparent to
+LibreOffice.
+
+```python
+from office.soffice import get_soffice_env
+import subprocess
+
+env = get_soffice_env()                      # already includes LD_PRELOAD if needed
+subprocess.run(["soffice", "--headless", ...], env=env)
+```
+
+`recalc.py` calls `get_soffice_env()` for you. Direct callers
+(unoserver, custom batch wrappers) should do the same so a single code
+path works on every host.
+
+The compiled `.so` is cached. **Do not edit the C source** in
+`soffice.py` — the file's bytes are part of the cache key, and any whitespace
+change invalidates a working shim on a sandboxed machine.
+
+## 7. Alternative — `unoserver`
+
+Long-running batch jobs (hundreds of workbooks per hour) can swap
+`soffice` for [`unoserver`](https://github.com/unoconv/unoserver), a
+daemon that keeps a single LibreOffice process alive and accepts requests
+over UNO's socket protocol.
+
+| Tradeoff | `soffice` per call (default) | `unoserver` daemon |
+|---|---|---|
+| Memory | Released after each call | ~250 MB resident permanently |
+| Cold start | 1.5–3 s per call | None after first start |
+| Concurrency | Serial unless you orchestrate | Pool inside the daemon |
+| Setup | Zero | `pip install unoserver` + systemd unit / launchd plist |
+
+The default in this skill stays `soffice` because it is zero-config and
+reproducible across machines. Switch to `unoserver` only when the
+per-call cold start dominates the workload.
+
+## 8. Alternative — `xlcalculator` / `formulas` (pure Python)
+
+In environments where LibreOffice cannot be installed (CI containers,
+serverless, Apple Silicon hosts without the cask), pure-Python
+calculators can recompute a subset of Excel functions:
+
+| Library | Coverage | Notable gaps |
+|---|---|---|
+| [`xlcalculator`](https://github.com/bradbase/xlcalculator) | ~140 of the most common functions (math, stats, lookup, text, date) | Array formulas, pivot tables, data validation, custom functions |
+| [`formulas`](https://github.com/vinci1it2000/formulas) | Wider function set, dependency-graph based | Slower on large workbooks; charts and conditional formatting unsupported |
+
+```python
+from xlcalculator import ModelCompiler, Evaluator
+
+compiler = ModelCompiler()
+model = compiler.read_and_parse_archive("model.xlsx")
+evaluator = Evaluator(model)
+print(evaluator.evaluate("MAU_Forecast!D2"))
+```
+
+Use these as **CI fallbacks** — for delivery, always rerun
+`scripts/recalc.py` on a host with LibreOffice so the cached values
+match what reviewers see when they open the file in Excel.
+
+## 9. LibreOffice → openpyxl mergeCell roundtrip caveat
+
+LibreOffice's `calculateAll() + store()` cycle aggressively materialises
+implicit merged regions into explicit `<mergeCell>` elements — a 22-sheet
+input with four merged regions has been observed to come back with 200+,
+some with `ref` attributes that openpyxl's `CellRange` parser cannot
+read. The `_scan_for_errors` path then dies with a bare
+`TypeError: expected <class 'int'>` at
+`openpyxl.descriptors.base._convert`, even though LibreOffice itself
+exited `0` and the recalculated values are correctly persisted on disk.
+
+`scripts/recalc.py` handles this transparently:
+
+1. It tries `_scan_for_errors` (openpyxl) first.
+2. If that raises `TypeError` or `ValueError`, it falls back to
+   `_scan_for_errors_via_xml`, which walks the `.xlsx` zip directly with
+   `zipfile + xml.etree.ElementTree`. The XML scan does the same job —
+   count `<f>` elements as formulas, look for the seven error tokens in
+   `<v>` cached values — without touching merged-cell parsing.
+3. The JSON shape is identical to §2 with one extra field:
+
+```json
+{
+  "status": "success",
+  "total_errors": 0,
+  "error_summary": {},
+  "total_formulas": 5,
+  "warning": "openpyxl could not parse the post-recalc workbook (expected <class 'int'>); verified via raw XML scan instead. LibreOffice did complete the recalculation."
+}
+```
+
+Callers should treat the verdict as authoritative regardless of which
+scanner produced it; the `warning` field is informational only — there
+is no need to retry or escalate. If both scanners fail (rare), the
+script returns `{"error": "..."}` listing both failure reasons.
+
+The fallback exists because LibreOffice's mergeCell rewrites are
+non-deterministic across versions; the right place to absorb them is
+inside `recalc.py`, not in every downstream caller.
+
+**Downstream readers are still affected.** The fallback patches
+`recalc.py` itself, but a *caller* who later does
+`load_workbook(path)` on the same post-recalc file will hit the same
+`TypeError`. Open with `load_workbook(path, read_only=True,
+data_only=True)` instead — the streaming reader bypasses the merged-cell
+parser and returns cached values directly. When you also need formula
+strings (or want to inspect the zip without instantiating openpyxl
+objects), use the raw-XML pattern in
+[`docs/create-edit-guide.md`](create-edit-guide.md) §11.
+
+## 10. Non-standard XLSX compatibility fallback
+
+Some real-world workbooks (observed in formula-summary / Starbucks-style
+workbooks) contain package metadata that openpyxl parses strictly but Excel and
+LibreOffice tolerate: expanded `mergeCell` refs, chart/drawing relationship
+quirks, styles/number-format oddities, or vendor-generated XML. In these cases,
+`recalc.py` must not crash after LibreOffice has already recalculated the file.
+
+Current behavior:
+
+1. LibreOffice recalculates and saves the workbook.
+2. `recalc.py` tries the normal openpyxl scanner.
+3. **Any openpyxl parse failure** falls back to raw worksheet XML scanning.
+4. The fallback ignores charts/drawings/styles/merged-cell objects and only
+   verifies what recalc needs: cached error tokens and `<f>` formula nodes.
+
+Fallback JSON includes extra diagnostic fields:
+
+```json
+{
+  "status": "success",
+  "total_errors": 0,
+  "error_summary": {},
+  "total_formulas": 42,
+  "scanner": "raw_xml_fallback",
+  "openpyxl_error_type": "TypeError",
+  "openpyxl_error": "expected <class 'int'>",
+  "compatibility_hint": "openpyxl failed while parsing merged-cell metadata ... do not retry blindly ..."
+}
+```
+
+Agent rule:
+
+- Treat `status: success` or `errors_found` from `scanner: raw_xml_fallback` as
+  authoritative for formula/error verification.
+- Do **not** retry the same command blindly just because openpyxl complained.
+- If downstream code must read values after such a warning, prefer
+  `load_workbook(path, read_only=True, data_only=True)` or raw XML inspection.
+- If downstream code must preserve charts/merged cells/styles, avoid saving the
+  workbook with openpyxl unless you have a separate round-trip verification.
+- If both openpyxl and raw XML fallback fail, inspect with
+  `scripts/office/unpack.py` and resave the workbook with Excel/LibreOffice.
+