pyOpenVBA 6.2.0__tar.gz → 6.3.0__tar.gz

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyOpenVBA
-Version: 6.2.0
+Version: 6.3.0
 Summary: Read and write VBA macros and Excel Power Query in pure Python, no dependencies.
 Project-URL: Homepage, https://github.com/WilliamSmithEdward/pyOpenVBA
 Project-URL: Repository, https://github.com/WilliamSmithEdward/pyOpenVBA
@@ -936,6 +936,18 @@ A project its application would not write is not written here either:
 Excel writes only the code names for one with no code in it, and
 PowerPoint nothing for one with no module.
 
+An Access database answers the same way until its first module, since
+Access makes no project for a database that has never held code.
+`AccessDatabase.add_vba_project()` gives it the one Access makes:
+
+```python
+with AccessDatabase("orders.accdb") as db:
+    if not db.has_vba_project():
+        db.add_vba_project()
+    db.add_module("Module1", "Public Function Ping() As Long\r\n    Ping = 1\r\nEnd Function")
+    db.save()
+```
+
 ### Access
 
 | Extension | What it is                   | Read | Write | create_new |

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/README.md

@@ -899,6 +899,18 @@ A project its application would not write is not written here either:
 Excel writes only the code names for one with no code in it, and
 PowerPoint nothing for one with no module.
 
+An Access database answers the same way until its first module, since
+Access makes no project for a database that has never held code.
+`AccessDatabase.add_vba_project()` gives it the one Access makes:
+
+```python
+with AccessDatabase("orders.accdb") as db:
+    if not db.has_vba_project():
+        db.add_vba_project()
+    db.add_module("Module1", "Public Function Ping() As Long\r\n    Ping = 1\r\nEnd Function")
+    db.save()
+```
+
 ### Access
 
 | Extension | What it is                   | Read | Write | create_new |

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/docs/architecture.md

@@ -230,7 +230,8 @@ follows and how it was measured is in
 [docs/access_engine.md](access_engine.md).
 
 `AccessDatabase` exposes the same surface as the other hosts:
-`module_names()`, `get_module()`, `set_module()`, `vba_project()` with
+`has_vba_project()` and `add_vba_project()` for a database that has
+never held code, `module_names()`, `get_module()`, `set_module()`, `vba_project()` with
 `add_module` / `rename_module` / `delete_module`, `pull_modules()` /
 `push_modules()`, `forms()` and `add_form()` with `add_control` /
 `remove_control` / `set_property` on the result, `save()`, `create_new()`.
@@ -246,11 +247,12 @@ the project from the module streams on the next open, as Access's own
 `/decompile` does. The code has to compile; a live gate has Access run
 what was written and compares the value the code returns.
 
-`AccessReader` predates the engine: a page scanner that finds the MS-OVBA
-blobs by signature and decodes `MSysObjects` on its own. It keeps its
-read API (`get_module`, `vba_modules`, `read_project_info`, `identifiers`,
-`disassemble_module`, the catalog readers, `pull_modules`) and
-`pull_access` still uses it; it writes nothing. The chronicle of the
+`AccessReader` predates the engine and now reads through it: modules,
+the project's streams and `MSysObjects` come from the rows the engine
+finds them in, and text is decoded in the project's code page.  It keeps
+its read API (`get_module`, `vba_modules`, `read_project_info`,
+`identifiers`, `disassemble_module`, the catalog readers,
+`pull_modules`) and `pull_access` still uses it; it writes nothing. The chronicle of the
 attempts that came before the engine is
 [docs/msaccess_lessons_learned.md](msaccess_lessons_learned.md).
 
@@ -488,9 +490,9 @@ tests/
   test_powerpoint.py           PowerPointFile facade, end-to-end
   test_pull_push.py            Disk workflow
   test_gates.py                Per-roadmap-gate regression tests
-  test_access.py               AccessReader read path: page walk,
-                               LVAL chain walk, MS-OVBA blob decode,
-                               byte-for-byte oracle parity (EXPERIMENTAL)
+  test_access.py               AccessReader read path: modules through
+                               MSysAccessStorage, code pages,
+                               byte-for-byte oracle parity
   test_access_engine.py        Storage engine read layer against the
                                Access-authored fixtures
   test_live_access_engine_gate.py

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/docs/research/access_write/README.md

@@ -720,11 +720,18 @@ What it writes:
   `Modules`, a 13-byte `PropData` under it, and the module's stream under
   `VBA` with a row name of 28 random capitals
 * an entry in `Modules/PropData`, the list of those folders:
-  `05 09 02 <folder name, one UTF-16 character> "CB0"`, eleven bytes each
+  `05 09 02 <folder name, one UTF-16 character> "CB0"`, eleven bytes each.
+  The `09` and `02` are sizes, of the rest of the entry and of the name,
+  so a two-digit folder is `05 0b 04 "10" "CB0"` (GitHub issue #34).
+  *2026-09-27:* Access writes this entry, and the module's
+  `MSysNavPaneGroupToObjects` row, the next time it opens the database,
+  not when it adds the module, so the library now leaves both to it
+  (`tests/live_access_test/first_module_reopened.accdb`)
 * a dir block of eleven records and PROJECTMODULES up by one
 * entries in DirData, PROJECTwm and PROJECT
 * an `MSysObjects` row of type -32761, a `MSysNavPaneObjectIDs` row and a
-  `MSysNavPaneGroupToObjects` row in group 8
+  `MSysNavPaneGroupToObjects` row in group 8 (the last, since 2026-09-27,
+  left to Access, which adds it on its next open)
 * the `Version` word of `_VBA_PROJECT` set to a value VBA does not know
 
 It works on the shipped blank template too, which an earlier note had
@@ -733,6 +740,20 @@ all it needed.
 
 ### Three allocations that are computed, not chosen
 
+> **Corrected 2026-09-27.** The folder base of `4` and the id step of four
+> below were measured with Access driven through pyvbaharness, which
+> injects three modules of its own (`PyVbaHarnessRunner`,
+> `PyVbaHarnessCall`, `PyVbaUserCode`) into the project it runs code in.
+> While they exist they hold folders `1`, `2`, `3` and the next three ids,
+> so the module Access added took `4` and an id four higher; the `{0, 5}`
+> case reusing `4` is the same effect.  Access driven over COM alone gives
+> the lowest free folder counting from `0` and the next id
+> (`tests/live_access_test/modules_added.accdb`), the library now does
+> the same, and the live gate lists and runs modules in folders `1` and
+> up.  The `AllModules(i).Name` failures that seemed to confirm the old
+> rules came from `\x03DirData` naming folder `4` for every module, which
+> was fixed the same day.
+
 Both were found the same way -- a created module that the VBE listed and
 ran, while `CurrentProject.AllModules(i).Name` failed with "refers to an
 object that is closed or doesn't exist".
@@ -743,7 +764,10 @@ not folders>)`. `Modules` holds four such rows -- `PropData`,
 `PropDataCopy`, `DirData` and `DirDataCopy` -- so its folders start at
 `4`, which is why the second module in a database gets `4` and never `1`.
 Only the computed name works: on the blank template `1`, `9` and `A` all
-fail while `4` succeeds.
+fail while `4` succeeds. Past `9` the name is the number in decimal,
+`10`, not the character after `9`: Access filled every container of the
+template past folder 9 and named them `10`, `11` and `12` (GitHub issue
+#34, `tests/live_access_test/folders_past_nine.accdb`).
 
 Two rules fit the first four measurements and only one fits all six. A
 count, `chr(0x30 + children - 1)`, works while folders stay contiguous.
@@ -765,6 +789,66 @@ next id: the step is what an object reserves, not a global stride.
 
 That last line is the whole trick, and it is why the list is short.
 
+### The order of `\x03DirData` and `PropData`: an `std::unordered_map`
+
+Access does not append a new object's entry. Adding `Macro1` to
+`Macro11` gave `Macro1`, `Macro10`, `Macro2`, `Macro11`, `Macro3` and on
+to `Macro9`, in one session or in eleven. Forms named `Macro1` to
+`Macro11` come out the same way and macros named `Form1` to `Form11` in
+order, so the order follows the names. Adding `Mod2` to `Mod13` one
+session each, with a copy of the file after every session, showed
+entries that the session never touched changing places:
+
+```
+after Mod5    Module1 Mod2 Mod3 Mod4 Mod5
+after Mod6    Mod6 Module1 Mod2 Mod3 Mod4 Mod5
+after Mod7    Module1 Mod6 Mod2 Mod3 Mod4 Mod5 Mod7
+after Mod11   Module1 Mod6 Mod11 Mod2 Mod3 Mod4 Mod5 Mod7 Mod8 Mod9 Mod10
+after Mod13   Module1 Mod6 Mod11 Mod2 Mod12 Mod3 Mod13 Mod4 Mod5 Mod7 Mod8 Mod9 Mod10
+```
+
+Guessing the hash from orders like these went nowhere, so the answer
+came from `MSACCESS.EXE`. The storage names sit in a table beside a
+stream class whose reader inserts each entry it reads into a container
+that guards itself with "unordered_map/set too long". A container's
+`\x03DirData` is an MSVC `std::unordered_map` keyed by object name, its
+`PropData` folder list another keyed by folder name, and each stream is
+the map's own order. From the code, and checked against every probe:
+
+- **The hash** (at `0x14006c060`) is 16 bits: for each character that
+  counts, `h = (h << 5) + (h >> 13) + 1 + (c & 0x1f)`. So case does not
+  matter and `0` hashes as `P`. A byte table decides what counts among
+  ASCII characters: `"`, `'`, `~` and control characters are skipped,
+  and a space or tab counts as 0. A leading `.` is skipped. From the
+  first character past ASCII, the rest of the name goes through the
+  system code page (`WideCharToMultiByte`, no best fit) and
+  `LCMapStringA` upper-casing, a byte at a time, every byte above 1
+  counting. A Turkish UI turns two bytes into `I`, which is not
+  reproduced.
+- **The map** starts with 8 buckets; a key's bucket is `hash & (buckets
+  - 1)`. A new key goes in front of the first key of its bucket, or at
+  the end when its bucket is empty. An insert that would leave more keys
+  than buckets first grows the map to a power of two at least the new
+  size, and eightfold while it is under 512 buckets. The rehash walks the
+  list and moves each key to the front of its new bucket.
+- **Loading** inserts the stored entries in their order. That is why a
+  session reverses some pairs and not others: the ninth entry grows the
+  map, so the first eight are reversed twice and the rest once.
+- **Writing** happens only when the map changed. A delete erases; a
+  rename erases and inserts, in both lists under `DoCmd.Rename`, while a
+  module renamed in the VBE leaves `PropData` alone. A new object's
+  `PropData` line waits for the next open, when the missing lines are
+  inserted in DirData's order.
+
+Checked on 13 histories, every snapshot included: 36 one-character names
+made in one session forwards and backwards and then reopened, `Mod2` to
+`Mod13` one session each, `Macro1` to `Macro11` both ways, the forms and
+reports, `modules_added.accdb` and three small fixtures. Also on a
+delete, two renames, seven names past ASCII and four with skipped
+characters, and on `_container_order.ps1`'s delete and rename of
+modules, a form and a report in copies of `folders_past_nine.accdb`.
+The library writes these orders (`_storage.access_order`).
+
 ### `_VBA_PROJECT` is a cache, so invalidate it instead of forging it
 
 *This is what shipped; see `pyopenvba/access/_vba.py`.*

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pyOpenVBA"
-version = "6.2.0"
+version = "6.3.0"
 description = "Read and write VBA macros and Excel Power Query in pure Python, no dependencies."
 readme = "README.md"
 license = { text = "MIT" }

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/src/pyopenvba/__init__.py

@@ -293,4 +293,4 @@ def push_power_query(
     """
     return _push_queries(src_dir, workbook, out=out, encoding=encoding, remove_missing=remove_missing)
 
-__version__ = "6.2.0"
+__version__ = "6.3.0"

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/src/pyopenvba/access/_designs.py

@@ -35,10 +35,11 @@ import re
 import struct
 import unicodedata
 from collections.abc import Iterable, Mapping, Sequence
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
 from pathlib import Path
 from typing import NamedTuple
 
+from pyopenvba.access._themes import OFFICE_2007, Theme, theme_color
 from pyopenvba.access_read import AccessError
 
 #: `<u32 id><u16 code><u32 type><u32 width><u32 length>`.
@@ -107,6 +108,11 @@ NAME_CODES = (20, 21)
 
 #: Where the captured designs live, one folder stream each.
 TEMPLATES = Path(__file__).parents[1] / "_templates" / "designs"
+#: A new form's or report's own `PropData` as Access writes it when it
+#: makes one: property 1, the flag `set_design_code` sets, and nothing
+#: else.  A design saved again in a later session gains property 0 as
+#: well, which is how the captured designs carried it.
+NEW_DESIGN_PROP_DATA = bytes.fromhex("00000000" "02" "01000000" "00000000")
 #: The record that carries a design's GUID, which the catalog row repeats.
 GUID_RECORD = 208
 GUID_LENGTH = 16
@@ -2180,21 +2186,134 @@ _PROTOTYPES: dict[str, dict[int, tuple[DesignRecord, ...]]] = {}
 DEFAULTS_FREE = frozenset({118})
 
 
-def prototype_records(kind: str, type_code: int) -> tuple[DesignRecord, ...]:
+def prototype_records(kind: str, type_code: int, theme: Theme = OFFICE_2007) -> tuple[DesignRecord, ...]:
     """The records of the control-defaults object for `type_code` on a
-    design of `kind`; refused for a type the fixture did not hold."""
+    design of `kind`, as `theme` gives them; refused for a type the
+    fixture did not hold."""
     if kind not in _PROTOTYPES:
         _header, objects, _trailer = parse_design(template(kind, "prototypes"))
         _PROTOTYPES[kind] = {o.type: o.records for o in objects[1:] if o.type is not None}
     try:
-        return _PROTOTYPES[kind][type_code]
+        records = _PROTOTYPES[kind][type_code]
     except KeyError:
         name = CONTROL_TYPES.get(type_code, type_code)
         raise AccessError(f"no control defaults were captured for a {name} on a {kind}") from None
+    type_name = CONTROL_TYPES.get(type_code, "")
+    measured = PROPERTY_SLOTS.get(type_name, {}).get("TextFontFamily")
+    return rethemed(records, theme, measured[0] if measured else FAMILY_IDS.get(type_name))
+
+
+# --- what a design takes from its database's theme -------------------------------
+# The templates and the control defaults were captured in a database on
+# Office's 2007 theme.  Access builds the same objects from whatever theme
+# the database holds (`_themes`), and the differences are these, measured
+# by having Access make one form and one report with a control of every
+# type in a database on each Office theme:
+#
+# * a font comes from the theme by its ThemeFontIndex, 0 the heading face
+#   and 1 the body face, with the face's family byte beside it
+#   (TextFontFamily) unless it is the default a sans face takes;
+# * a colour is worked out again from the slot, tint and shade beside it;
+# * the design itself carries the body face, its family byte and the
+#   theme's Background 2 colour.
+#
+# Access also sizes a tab control's strip and pages by the face's metrics,
+# which is not reproduced here.
+
+#: A defaults object's family-byte slot where the measured slots have none.
+FAMILY_IDS = {"NavigationButton": 368}
+THEME_FONT_INDEX = PROPERTY_CODES["ThemeFontIndex"]
+FONT_NAME = PROPERTY_CODES["FontName"]
+TEXT_FONT_FAMILY = PROPERTY_CODES["TextFontFamily"]
+#: The design's own face, its family byte's slot, and its Background 2.
+DESIGN_FONT = 160
+DESIGN_FONT_FAMILY_SLOT = (56, 244, 2, 1)
+DESIGN_BACKGROUND = 319
+BACKGROUND_2 = 3
+#: Each themed colour beside the slot, tint and shade it is worked out from.
+THEMED_COLORS = tuple(
+    (
+        PROPERTY_CODES[f"{part}Color"],
+        PROPERTY_CODES[f"{part}ThemeColorIndex"],
+        PROPERTY_CODES.get(f"{part}Tint"),
+        PROPERTY_CODES.get(f"{part}Shade"),
+    )
+    for part in ("Back", "Border", "Fore", "Gridline", "Hover", "Pressed", "HoverFore", "PressedFore")
+)
+
+
+def _signed(record: DesignRecord) -> int:
+    return int.from_bytes(record.value, "little", signed=True)
+
+
+def _themed_face(
+    out: dict[int, DesignRecord], face: str, family: DesignRecord | None, family_slot: tuple[int, ...] | None
+) -> None:
+    """Set the family byte `face` takes: none for the default, else the
+    record at `family_slot`, `(id, code, value type, width)`."""
+    byte = FONT_FAMILIES.get(face)
+    if byte is None:
+        if family is not None:
+            del out[family.id]
+    elif family is not None:
+        out[family.id] = replace(family, value=bytes((byte,)))
+    elif family_slot is not None:
+        slot_id, code, value_type, width = family_slot[:4]
+        out[slot_id] = DesignRecord(slot_id, code, value_type, width, bytes((byte,)))
+
+
+def rethemed(records: Sequence[DesignRecord], theme: Theme, family_id: int | None) -> tuple[DesignRecord, ...]:
+    """Records captured on Office's 2007 theme, as `theme` gives them.
+    `family_id` is where the object's family byte goes when the theme's
+    face needs one and the object has none."""
+    if theme == OFFICE_2007:
+        return tuple(records)
+    by_code = {r.code: r for r in records}
+    out = {r.id: r for r in records}
+    index = by_code.get(THEME_FONT_INDEX)
+    font = by_code.get(FONT_NAME)
+    if index is not None and font is not None and _signed(index) in (0, 1):
+        face = theme.major_font if _signed(index) == 0 else theme.minor_font
+        out[font.id] = replace(font, value=face.encode("utf-16-le"))
+        slot = None if family_id is None else (family_id, TEXT_FONT_FAMILY, 2, 1)
+        _themed_face(out, face, by_code.get(TEXT_FONT_FAMILY), slot)
+    for color, slot_code, tint_code, shade_code in THEMED_COLORS:
+        found, slot_record = by_code.get(color), by_code.get(slot_code)
+        if found is None or slot_record is None or _signed(slot_record) < 0:
+            continue
+        tint = by_code.get(tint_code) if tint_code is not None else None
+        shade = by_code.get(shade_code) if shade_code is not None else None
+        out[found.id] = replace(found, value=theme_color(
+            theme,
+            _signed(slot_record),
+            struct.unpack("<f", tint.value)[0] if tint is not None else 100.0,
+            struct.unpack("<f", shade.value)[0] if shade is not None else 100.0,
+        ))
+    return tuple(out[record_id] for record_id in sorted(out))
+
+
+def rethemed_design(blob: bytes, theme: Theme) -> bytes:
+    """An empty design from the templates, as `theme` gives it."""
+    if theme == OFFICE_2007:
+        return blob
+    header, objects, trailer = parse_design(blob)
+    own = {r.id: r for r in objects[0].records}
+    by_code = {r.code: r for r in objects[0].records}
+    font = by_code.get(DESIGN_FONT)
+    if font is not None:
+        own[font.id] = replace(font, value=theme.minor_font.encode("utf-16-le"))
+    background = by_code.get(DESIGN_BACKGROUND)
+    if background is not None:
+        own[background.id] = replace(background, value=theme_color(theme, BACKGROUND_2))
+    family = next((r for r in objects[0].records if r.code == DESIGN_FONT_FAMILY_SLOT[1]), None)
+    _themed_face(own, theme.minor_font, family, DESIGN_FONT_FAMILY_SLOT)
+    design = replace(objects[0], records=tuple(own[record_id] for record_id in sorted(own)))
+    rest = tuple(replace(o, records=rethemed(o.records, theme, None)) for o in objects[1:])
+    return build_design(header, (design, *rest), trailer)
 
 
 def with_prototype(
-    objects: tuple[DesignObject, ...], kind: str, type_code: int
+    objects: tuple[DesignObject, ...], kind: str, type_code: int, theme: Theme = OFFICE_2007
 ) -> tuple[DesignObject, ...]:
     """The design with the control-defaults object for `type_code` in
     place: inserted in type order among the ones there, the run of
@@ -2203,7 +2322,7 @@ def with_prototype(
     ahead = list(objects[1:first_section])
     if type_code in DEFAULTS_FREE or any(o.type == type_code for o in ahead):
         return objects
-    fresh = DesignObject(None, type_code, None, prototype_records(kind, type_code))
+    fresh = DesignObject(None, type_code, None, prototype_records(kind, type_code, theme))
     at = next((i for i, o in enumerate(ahead) if (o.type or 0) > type_code), len(ahead))
     ahead.insert(at, fresh)
     sections = [(i, o) for i, o in enumerate(objects) if o.is_section]
@@ -2296,6 +2415,8 @@ BUTTON_FILL_COMPANIONS: tuple[tuple[str, object], ...] = (("Gradient", 0),)
 #: from the installed font; these are the values it wrote for the fonts
 #: below (measured on 24 fonts), and a font not listed gets its name alone.
 FONT_FAMILIES: dict[str, int] = {
+    "Aptos": 0,
+    "Aptos Display": 0,
     "Book Antiqua": 18,
     "Cambria": 18,
     "Comic Sans MS": 66,
@@ -2467,6 +2588,7 @@ def add_control(
     height: int = 240,
     caption: str | None = None,
     kind: str = "form",
+    theme: Theme = OFFICE_2007,
 ) -> bytes:
     """A design with one more control on it.
 
@@ -2480,7 +2602,8 @@ def add_control(
     the section's.
 
     The first control of a type also brings the type's control-defaults
-    object ahead of the sections, as it does in Access (`with_prototype`).
+    object ahead of the sections, as it does in Access (`with_prototype`),
+    in the fonts and colours of `theme`, the database's.
     """
     header, objects, trailer = parse_design(blob)
     if any(o.name == name for o in objects):
@@ -2512,7 +2635,7 @@ def add_control(
     )
     # The first control of a type brings the type's control-defaults object
     # ahead of the sections, which moves the sections along.
-    objects = with_prototype(objects, kind, TYPE_CODES[control_type])
+    objects = with_prototype(objects, kind, TYPE_CODES[control_type], theme)
     at = next(i for i, o in enumerate(objects) if o.is_section and o.name == section)
     end = at + 1
     while end < len(objects) and not objects[end].is_section:

{pyopenvba-6.2.0 → pyopenvba-6.3.0}/src/pyopenvba/access/_macros.py

@@ -70,11 +70,9 @@ ACTION_IDS: dict[str, int] = {
 }
 ACTION_NAMES: dict[int, str] = {value: key for key, value in ACTION_IDS.items()}
 
-#: `MSysObjects.Type` and `MSysNavPaneObjectIDs.Type` for a macro.  A
-#: macro's object id steps by one where a module's steps by four.
+#: `MSysObjects.Type` and `MSysNavPaneObjectIDs.Type` for a macro.
 OBJECT_MACRO = -32766
 NAV_MACRO_TYPE = 32770
-OBJECT_ID_STEP = 1
 
 
 @dataclass(frozen=True)