splat64 0.35.2__tar.gz → 0.36.1__tar.gz

{splat64-0.35.2 → splat64-0.36.1}/CHANGELOG.md

@@ -1,5 +1,27 @@
 # splat Release Notes
 
+### 0.36.1
+* Files generated with `generate_asm_macros_files` will not be overwritten if they already exist on disk unchanged.
+* Updated GTE macros to better document opcode encodings
+
+### 0.36.0
+
+* New symbol attribute: `align`.
+  * Emit an alignment directive for the given symbol during disassembly.
+  * The given alignment must be positive, be a power of two and The symbol's address must already be aligned to the given custom alignment, otherwise splat will emit an error and halt.
+* New yaml option: `ld_gp_expression`:
+  * Allows setting a custom expression for the `_gp` symbol in the generated linker script, making this symbol to properly shift around instead of hardcoding it to the given `gp_value`.
+* New yaml option: `include_asm_macro_style`:
+  * Allows configuring the style used by the generated `INCLUDE_ASM` macro. It currently allows two possible values:
+  * `default`: Uses the default definition for the macro. This is the default.
+  * `maspsx_hack`: Changes the definition of the generated `INCLUDE_ASM` to be compatible with the one expected by `maspsx` when using the [reordering workaround hack](https://github.com/mkst/maspsx?tab=readme-ov-file#include_asm-reordering-workaround-hack). This value is only relevant for psx projects.
+* `incbin` segments now provide a default symbol if the user hasn't defined a symbol at that given address.
+  * This name is automatically escaped to be a valid symbol name.
+  * The name of this symbol is based on the segment's name, with a `__` prefix and a suffix depending on the type of incbin used.
+  * This naming scheme may change at any time.
+* `incbin` segments now emit a nonmatching marker.
+* `spimdisasm` 1.36.1 or above is now required.
+
 ### 0.35.2
 
 * Miscellaneous updates to generated macro labels.

{splat64-0.35.2 → splat64-0.36.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splat64
-Version: 0.35.2
+Version: 0.36.1
 Summary: A binary splitting tool to assist with decompilation and modding projects
 Project-URL: Repository, https://github.com/ethteck/splat
 Project-URL: Issues, https://github.com/ethteck/splat/issues
@@ -42,7 +42,7 @@ Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: n64img>=0.3.3; extra == 'dev'
 Requires-Dist: pygfxd; extra == 'dev'
 Requires-Dist: rabbitizer<2.0.0,>=1.12.0; extra == 'dev'
-Requires-Dist: spimdisasm<2.0.0,>=1.36.0; extra == 'dev'
+Requires-Dist: spimdisasm<2.0.0,>=1.36.1; extra == 'dev'
 Requires-Dist: types-colorama; extra == 'dev'
 Requires-Dist: types-pyyaml; extra == 'dev'
 Provides-Extra: mips
@@ -50,7 +50,7 @@ Requires-Dist: crunch64<1.0.0,>=0.5.1; extra == 'mips'
 Requires-Dist: n64img>=0.3.3; extra == 'mips'
 Requires-Dist: pygfxd; extra == 'mips'
 Requires-Dist: rabbitizer<2.0.0,>=1.12.0; extra == 'mips'
-Requires-Dist: spimdisasm<2.0.0,>=1.36.0; extra == 'mips'
+Requires-Dist: spimdisasm<2.0.0,>=1.36.1; extra == 'mips'
 Description-Content-Type: text/markdown
 
 # splat
@@ -76,7 +76,7 @@ The brackets corresponds to the optional dependencies to install while installin
 If you use a `requirements.txt` file in your repository, then you can add this library with the following line:
 
 ```txt
-splat64[mips]>=0.35.2,<1.0.0
+splat64[mips]>=0.36.1,<1.0.0
 ```
 
 ### Optional dependencies

{splat64-0.35.2 → splat64-0.36.1}/README.md

@@ -21,7 +21,7 @@ The brackets corresponds to the optional dependencies to install while installin
 If you use a `requirements.txt` file in your repository, then you can add this library with the following line:
 
 ```txt
-splat64[mips]>=0.35.2,<1.0.0
+splat64[mips]>=0.36.1,<1.0.0
 ```
 
 ### Optional dependencies

{splat64-0.35.2 → splat64-0.36.1}/docs/Adding-Symbols.md

@@ -209,3 +209,29 @@ __opPCc__Q23std34_RefCountedPtr<c,Q23std9_Array<c>>CFv = 0x00202850; // filename
 ```
 
 Gets written to `func_00202850.s`
+
+### `align`
+
+Emit an alignment directive for the given symbol during disassembly.
+
+Even though an explicit alignment directive is not necessary in most situations, it may be desirable to have one on shiftable builds of partially matched projects. This allows ensuring the symbol's required alignment even when it is still being disassembled.
+
+  Some situations that may require an explicit alignment directive include textures in N64 (0x8 alignment) or symbols sent to the IOP on PS2 (0x40 alignment).
+
+The symbol's address must already be aligned to the given custom alignment, or the value will be discarded.
+
+This value must be a power of two, otherwise it will be discarded.
+
+**Example**
+
+```ini
+rgb_texture = 0x82013118; // align:0x8
+```
+
+Produces a disassembly similar to the following:
+
+```mips
+.align 3
+dlabel rgb_texture
+# ...
+```

{splat64-0.35.2 → splat64-0.36.1}/docs/Configuration.md

@@ -138,6 +138,15 @@ Some files may not be generated depending on the selected platform and compiler,
 
 Defaults to `True`.
 
+### include_asm_macro_style
+
+Allows configuring the style used by the generated `INCLUDE_ASM` macro. It currently allows two possible values:
+
+- `default`: Uses the default definition for the macro. This is the default.
+- `maspsx_hack`: Changes the definition of the generated `INCLUDE_ASM` to be compatible with the one expected by `maspsx` when using the [reordering workaround hack](https://github.com/mkst/maspsx?tab=readme-ov-file#include_asm-reordering-workaround-hack).
+
+Defaults to `default`.
+
 ### o_as_suffix
 
 Used to determine the file extension of the built files that will be listed on the linker script.
@@ -155,6 +164,8 @@ Defaults to `False`.
 
 The value of the `$gp` register to correctly calculate offset to `%gp_rel` relocs.
 
+It is strongly encouraged to provide a [`ld_gp_expression`](#ld_gp_expression) once the sections of the binary are properly understood.
+Not setting this expression makes splat always emit a hardcoded value for `_gp` in the linker script, preventing proper shiftability.
 
 ### check_consecutive_segment_types
 
@@ -579,7 +590,7 @@ Defaults to `True`, meaning `bss` sections will be put on `NOLOAD` segments.
 
 Specify that segments should be aligned before starting them.
 
-This option specifies the desired alignment value, or `null` if no aligment should be imposed on the segment start.
+This option specifies the desired alignment value, or `null` if no alignment should be imposed on the segment start.
 
 This behavior can be customized per segment too. See [ld_align_segment_start](Segments.md#ld_align_segment_start) on the Segments section.
 
@@ -615,6 +626,46 @@ Sets the default option for the `bss_contains_common` attribute of all segments.
 
 Defaults to `False`.
 
+### ld_gp_expression
+
+Provides an expression for the `_gp` symbol to be emitted in the generated linker script.
+
+Most projects start by only setting a [`gp_value`](#gp_value) on their yamls. This is fine while matching the project, but it is bad for shiftable builds, becuase it prevents the `_gp` symbol to shift around as needed.
+
+This expression is used as-is in the generated linker script, so care must be taken to provide an expression that makes sense for shiftable builds and also matches the original gp value on matching builds.
+
+Usually this expression is relative to the start of an "small section" (`.sdata`, `.srodata`, `.sbss`, etc.) plus an optional offset (usually `0x7FF0` or `0x8000`).
+
+The recommended approach to know what to set here is to try to understand where the first small section is, take the difference between the [`gp_value`](#gp_value) and the address of the small section to know what the offset is and then use the linker symbol for the start of that small section.
+
+For example, say your gp value is `0x00397FF0` and you found the first small section on the rom is a `.sdata` section at vram address `0x00390000` (note *vram* address, not rom address) which is part of the top-level segment `main`. The difference between those two addresses is `0x7FF0`, so that's your offset. Given this information the expression you want for gp is `main_SDATA_START + 0x7FF0`; where `main` is the top-level segment, `SDATA` comes from the `.sdata` section and the `+ 0x7FF0` is your calculated offset. If you did this process right, then the matching build should still match after setting the gp expression.
+
+Note you should **not** remove the `gp_value` from your yaml when you set `ld_gp_expression`. `gp_value` is still used for the disassembly; without it you wouldn't get symbolized gp-accesses.
+
+#### Usage
+
+Not using `ld_gp_expression` makes splat hardcode the value:
+
+```yaml
+  gp_value: 0x0039DD70
+```
+
+```txt
+    _gp = 0x0039DD70;
+```
+
+When a `ld_gp_expression` is given then splat is able to emit a non-hardcoded expression for `_gp` without interfering with the disassembly process:
+
+```yaml
+  gp_value: 0x0039DD70
+  ld_gp_expression: main_SDATA_START + 0x7FF0
+```
+
+```txt
+    _gp = main_SDATA_START + 0x7FF0;
+```
+
+
 ## C file options
 
 ### create_c_files

{splat64-0.35.2 → splat64-0.36.1}/docs/Segments.md

@@ -318,6 +318,8 @@ Generating assembly files enables better customization of these binaries, like a
 
 If a known symbol (via a symbol_addrs file) matches the vram of a incbin segment then it will be emitted accordingly at the top. If the symbol contains a [`name_end`](Adding-Symbols.md#name_end) property then it will be emitted after the `.incbin` (useful for Nintendo64's RSP ucodes).
 
+If no symbol is provided then splat will emit a symbol based on the segment's name with a `__` prefix and a suffix depending on the type of incbin used. This autogenerated naming scheme may change at any time.
+
 Curretly there are 3 types of incbins, `textbin`, `databin` and `rodatabin`, which are intended for binary blobs of `.text`, `.data` and `.rodata` sections.
 
 If a `textbin` section has a corresponding `databin` and/or `rodatabin` section with the same name then those will be included in the same generated assembly file.
@@ -542,7 +544,7 @@ Defaults to the value of the global option.
 
 Specify the current segment should be aligned before starting it.
 
-This option specifies the desired alignment value, or `null` if no aligment should be imposed on the segment start.
+This option specifies the desired alignment value, or `null` if no alignment should be imposed on the segment start.
 
 If not set, then the global configuration is used. See [ld_align_segment_start](Configuration.md#ld_align_segment_start) on the Configuration section.
 

{splat64-0.35.2 → splat64-0.36.1}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "splat64"
 # Should be synced with src/splat/__init__.py
-version = "0.35.2"
+version = "0.36.1"
 description = "A binary splitting tool to assist with decompilation and modding projects"
 readme = "README.md"
 license = {file = "LICENSE"}
@@ -20,7 +20,7 @@ dependencies = [
 
 [project.optional-dependencies]
 mips = [
-    "spimdisasm>=1.36.0,<2.0.0", # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py
+    "spimdisasm>=1.36.1,<2.0.0", # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py
     "rabbitizer>=1.12.0,<2.0.0",
     "pygfxd",
     "n64img>=0.3.3",

{splat64-0.35.2 → splat64-0.36.1}/requirements.txt

@@ -4,7 +4,7 @@ tqdm
 intervaltree
 colorama
 # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py and pyproject.toml
-spimdisasm>=1.36.0
+spimdisasm>=1.36.1
 rabbitizer>=1.10.0
 pygfxd
 n64img>=0.1.4

{splat64-0.35.2 → splat64-0.36.1}/src/splat/__init__.py

@@ -1,7 +1,7 @@
 __package_name__ = __name__
 
 # Should be synced with pyproject.toml
-__version__ = "0.35.2"
+__version__ = "0.36.1"
 __author__ = "ethteck"
 
 from . import util as util

{splat64-0.35.2 → splat64-0.36.1}/src/splat/disassembler/spimdisasm_disassembler.py

@@ -7,7 +7,7 @@ from typing import Set
 
 class SpimdisasmDisassembler(disassembler.Disassembler):
     # This value should be kept in sync with the version listed on requirements.txt and pyproject.toml
-    SPIMDISASM_MIN = (1, 36, 0)
+    SPIMDISASM_MIN = (1, 36, 1)
 
     def configure(self):
         # Configure spimdisasm

{splat64-0.35.2 → splat64-0.36.1}/src/splat/scripts/create_config.py

@@ -286,6 +286,7 @@ options:
 
   find_file_boundaries: False
   gp_value: 0x{exe.initial_gp:08X}
+  # ld_gp_expression: main_SCOMMON_START + 0x7FF0
 
   o_as_suffix: True
   use_legacy_include_asm: False
@@ -309,6 +310,10 @@ options:
   data_string_encoding: ASCII
   rodata_string_guesser_level: 2
   data_string_guesser_level: 2
+
+  # Uncomment this line if you need to use the maspsx reorder workaround hack
+  # https://github.com/mkst/maspsx?tab=readme-ov-file#include_asm-reordering-workaround-hack
+  # include_asm_macro_style: maspsx_hack
 """
 
     segments = f"""\

{splat64-0.35.2 → splat64-0.36.1}/src/splat/segtypes/common/textbin.py

@@ -1,4 +1,5 @@
 from pathlib import Path
+import re
 from typing import Optional, TextIO
 
 from ...util import log, options
@@ -75,45 +76,66 @@ class CommonSegTextbin(CommonSegment):
 
         f.write(f"{self.get_section_asm_line()}\n\n")
 
+        sym_name = None
+        sym_name_end = None
+        sym_size_matches = None
+
         # Check if there's a symbol at this address
-        sym = None
         vram = self.rom_to_ram(self.rom_start)
         if vram is not None:
             sym = self.get_symbol(vram, in_segment=True)
-
-        if sym is not None:
-            f.write(f"{asm_label} {sym.name}\n")
-            if asm_label == ".globl":
-                if self.is_text():
-                    f.write(f".ent {sym.name}\n")
-                f.write(f"{sym.name}:\n")
-            sym.defined = True
+            if sym is not None:
+                sym.defined = True
+                sym_name = sym.name
+                sym_name_end = sym.given_name_end
+                if (
+                    sym.given_size is None
+                    or sym.given_size == self.rom_end - self.rom_start
+                ):
+                    sym_size_matches = self.rom_end - self.rom_start
+
+        if sym_name is None:
+            # Normalize stuff like slashes and such.
+            n = regex_sym_name_normalizer.sub("_", self.name)
+            if self.is_text():
+                suffix = "textbin"
+            elif self.is_data():
+                suffix = "databin"
+            elif self.is_rodata():
+                suffix = "rodatabin"
+            else:
+                suffix = "incbin"
+            sym_name = f"__{n}_{suffix}"
+
+        if options.opts.asm_nonmatching_label_macro != "":
+            siz = f", 0x{sym_size_matches:X}" if sym_size_matches is not None else ""
+            f.write(f"{options.opts.asm_nonmatching_label_macro} {sym_name}{siz}\n\n")
+
+        f.write(f"{asm_label} {sym_name}\n")
+        if asm_label == ".globl":
+            if self.is_text():
+                f.write(f".ent {sym_name}\n")
+            f.write(f"{sym_name}:\n")
 
         f.write(f'.incbin "{binpath.as_posix()}"\n')
 
-        if sym is not None:
-            if options.opts.asm_emit_size_directive:
-                f.write(f".size {sym.name}, . - {sym.name}\n")
+        if options.opts.asm_emit_size_directive:
+            f.write(f".size {sym_name}, . - {sym_name}\n")
 
-            if self.is_text() and options.opts.asm_end_label != "":
-                f.write(f"{options.opts.asm_end_label} {sym.name}\n")
-            elif self.is_data() and options.opts.asm_data_end_label != "":
-                f.write(f"{options.opts.asm_data_end_label} {sym.name}\n")
+        if self.is_text() and options.opts.asm_end_label != "":
+            f.write(f"{options.opts.asm_end_label} {sym_name}\n")
+        elif options.opts.asm_data_end_label != "":
+            f.write(f"{options.opts.asm_data_end_label} {sym_name}\n")
 
-            if sym.given_name_end is not None:
-                if (
-                    sym.given_size is None
-                    or sym.given_size == self.rom_end - self.rom_start
-                ):
-                    f.write(f"{asm_label} {sym.given_name_end}\n")
-                    if asm_label == ".globl":
-                        f.write(f"{sym.given_name_end}:\n")
-                    if self.is_text() and options.opts.asm_end_label != "":
-                        f.write(f"{options.opts.asm_end_label} {sym.given_name_end}\n")
-                    elif self.is_data() and options.opts.asm_data_end_label != "":
-                        f.write(
-                            f"{options.opts.asm_data_end_label} {sym.given_name_end}\n"
-                        )
+        if sym_name_end is not None and sym_size_matches is not None:
+            f.write(f"{asm_label} {sym_name_end}\n")
+            if asm_label == ".globl":
+                f.write(f"{sym_name_end}:\n")
+
+            if self.is_text() and options.opts.asm_end_label != "":
+                f.write(f"{options.opts.asm_end_label} {sym_name_end}\n")
+            elif options.opts.asm_data_end_label != "":
+                f.write(f"{options.opts.asm_data_end_label} {sym_name_end}\n")
 
     def split(self, rom_bytes):
         if self.rom_end is None:
@@ -159,3 +181,6 @@ class CommonSegTextbin(CommonSegment):
         return (
             self.extract and self.should_scan()
         )  # only split if the segment was scanned first
+
+
+regex_sym_name_normalizer = re.compile(r"[^0-9a-zA-Z_]")

{splat64-0.35.2 → splat64-0.36.1}/src/splat/segtypes/linker_entry.py

@@ -204,8 +204,10 @@ class LinkerWriter:
         if not self.is_partial:
             self._writeln(f"__romPos = {options.opts.ld_rom_start};")
 
-            if options.opts.gp is not None:
-                self._writeln("_gp = " + f"0x{options.opts.gp:X};")
+            if options.opts.ld_gp_expression is not None:
+                self._writeln(f"_gp = {options.opts.ld_gp_expression};")
+            elif options.opts.gp is not None:
+                self._writeln(f"_gp = 0x{options.opts.gp:X};")
 
     # Write a series of statements which compute a symbol that represents the highest address among a list of segments' end addresses
     def write_max_vram_end_sym(self, symbol: str, overlays: List[Segment]):

{splat64-0.35.2 → splat64-0.36.1}/src/splat/segtypes/n64/i1.py

@@ -1,9 +1,9 @@
-import n64img.image
-
-from .img import N64SegImg
-
-
-class N64SegI1(N64SegImg):
-    def __init__(self, *args, **kwargs):
-        kwargs["img_cls"] = n64img.image.I1
-        super().__init__(*args, **kwargs)
+import n64img.image
+
+from .img import N64SegImg
+
+
+class N64SegI1(N64SegImg):
+    def __init__(self, *args, **kwargs):
+        kwargs["img_cls"] = n64img.image.I1
+        super().__init__(*args, **kwargs)