splat64 0.34.3__tar.gz → 0.35.0__tar.gz

{splat64-0.34.3 → splat64-0.35.0}/CHANGELOG.md

@@ -1,6 +1,29 @@
 # splat Release Notes
 
+### 0.35.0
+
+* Add `pair_segment` option to segments.
+  * Allows pairing the sections of two different segments together, making cross-segment rodata migration possible.
+* Now `create_config` for N64 games can create basic `symbol_addrs.txt` and `reloc_addrs.txt` files from the information inferred from its analysis.
+* splat will now start generating and regenerating files related to assembly macros, allowing less painful updates to newer splat versions.
+  * This includes generating files like `include/include_asm.h`, `include/macro.inc`, `include/labels.inc` and `include/gte_macros.inc`.
+  * This makes easier for splat to introduce new kinds of labels or update the current definitions with minimal pain for the end-user.
+  * This behaviour can be slightly customized with `generated_macro_inc_content` or completely turned off with `generate_asm_macros_files`.
+  * Both `splat` and `create_config` have this behaviour.
+* New yaml options:
+  * `asm_nonmatching_label_macro`: Set the label to be used to signal an assembly symbol haven't been matched yet. Defaults to `nonmatching`.
+  * `asm_data_end_label`: Set the label used at the end of a data symbol. Defaults to `enddlabel`.
+  * `generated_macro_inc_content`: A string defining custom contents to be emitted at the generated `include/macro.inc` file.
+  * `generate_asm_macros_files`: Controls if the assembly macros files should be regenerated by splat or not.
+* Change the default value for multiple assembly macro labels.
+  * `asm_function_alt_macro`: Changed from `glabel` to `alabel`.
+  * `asm_jtbl_label_macro`: Changed from `glabel` to `jlabel`.
+  * `asm_data_macro`: Changed from `glabel` to `dlabel`.
+  * `asm_end_label`: Changed from empty to `endlabel`.
+* `spimdisasm` 1.36.0 or above is now required.
+
 ### 0.34.3
+
 * `asm_emit_size_directive` now also affects generated assembly from `textbin`, `databin`, and `rodatabin` segments.
 
 ### 0.34.2

{splat64-0.34.3 → splat64-0.35.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splat64
-Version: 0.34.3
+Version: 0.35.0
 Summary: A binary splitting tool to assist with decompilation and modding projects
 Project-URL: Repository, https://github.com/ethteck/splat
 Project-URL: Issues, https://github.com/ethteck/splat/issues
@@ -76,7 +76,7 @@ The brackets corresponds to the optional dependencies to install while installin
 If you use a `requirements.txt` file in your repository, then you can add this library with the following line:
 
 ```txt
-splat64[mips]>=0.34.3,<1.0.0
+splat64[mips]>=0.35.0,<1.0.0
 ```
 
 ### Optional dependencies

{splat64-0.34.3 → splat64-0.35.0}/README.md

@@ -21,7 +21,7 @@ The brackets corresponds to the optional dependencies to install while installin
 If you use a `requirements.txt` file in your repository, then you can add this library with the following line:
 
 ```txt
-splat64[mips]>=0.34.3,<1.0.0
+splat64[mips]>=0.35.0,<1.0.0
 ```
 
 ### Optional dependencies

splat64-0.35.0/docs/Advanced-Reloc.md

@@ -0,0 +1,200 @@
+# Advanced: Manual relocation handling
+
+Sometimes the disassembler is unable to determine the correct symbol to be referenced by a given function or data symbol. For cases like this you can manually tell the disassembler what symbol should be used on each specific instance and how it should be referenced in a `reloc_addrs.txt` file.
+
+Assembly function or data symbol references other symbol by using relocations (relocs for short). This section will explain how to override the automatic relocations generated by the disassembler.
+
+By default splat will read the `reloc_addrs.txt` file from the root of the project, if any, but it is possible to rename the file, move it to a different folder or even provide multiple files for organizational purposes. Please refer to the [`reloc_addrs_path`](Configuration.md#reloc_addrs_path) for more information.
+
+## Syntax
+
+To override relocs you need to have at least one `reloc_addrs.txt` file. Each line of this file corresponds to a reloc override entry. Empty lines are allowed and comments are used with `//`.
+
+The format for defining an entry is:
+
+```ini
+rom:0x04B440 reloc:MIPS_HI16 symbol:BonusWait addend:-0x3
+```
+
+Each attribute is defined as follows:
+
+- `rom`: The rom address of the instruction or data reference you want to affect is located.
+- `symbol`: The symbol that you want to reference at this given place.
+- `addend`: Optional. A displacement into the symbol. It can be either positive or negative.
+- `reloc`: The relocation kind to be used. It mirrors the standard MIPS relocations. The following values are valid for this attribute:
+  - Function relocs:
+    - `MIPS_HI16`: Corresponds to the `%hi` reloc operator.
+    - `MIPS_LO16`: Corresponds to the `%lo` reloc operator.
+    - `MIPS_GPREL16`: Corresponds to the `%gp_rel` reloc operator.
+    - `MIPS_GOT16`: Corresponds to the `%got` reloc operator.
+    - `MIPS_CALL16`: Corresponds to the `%call16` reloc operator.
+    - `MIPS_GOT_HI16`: Corresponds to the `%got_hi` reloc operator.
+    - `MIPS_GOT_LO16`: Corresponds to the `%got_lo` reloc operator.
+    - `MIPS_CALL_HI16`: Corresponds to the `%call_hi` reloc operator.
+    - `MIPS_CALL_LO16`: Corresponds to the `%call_lo` reloc operator.
+    - `MIPS_26`: No direct operator. Used in `jal` (jump and link) and `j` (jump) instructions.
+    - `MIPS_PC16`: No direct operator. Used in branch instructions.
+  - Data relocs:
+    - `MIPS_32`: Corresponds to `.word`.
+    - `MIPS_GPREL32`: Corresponds to `.gpword`.
+  - No reloc:
+    - `MIPS_NONE`: Makes no reloc to be used at all, making the disassembler to use the raw value instead. Useful for fake positives on the symbol detector.
+
+## Examples
+
+Next are a common patterns where providing manual relocs is useful.
+
+### Negative offsets
+
+A common that can be seen on many C functions is to access a global array with an index substraction, making the compiler to emit an assembly access that ends up pointing to a different index than the one would expect.
+
+Take for example the following C code.
+
+```c
+int some_sym = 0;
+int some_array[3] = {0};
+
+int get_value(int index) {
+    return some_array[index - 1];
+}
+```
+
+Some compilers with some optimization flags enabled may optimize the generated assembly into something that doesn't need to actually perform the `- 1` subtraction, as seen in the following example assembly:
+
+```mips
+sll     $t6, $a0, 0x2
+lui     $v0, %hi(some_array - 0x4)
+addu    $v0, $v0, $t6
+jr      $ra
+ lw      $v0, %lo(some_array - 0x4)($v0)
+```
+
+This then gets build and linked into a final binary. That binary won't have those explicit relocations, it will just have raw addresses, meaning that code will techincally reference the symbol "behind" the array we want to actually use, in this case we end up refering to `some_sym` instead.
+
+A direct disassembly of this assembly would look like similar to the following assembly. Note there's no mention of `some_array` anywhere.
+
+```mips
+/* 0000 80000000 00047080 */ sll     $t6, $a0, 0x2
+/* 0004 80000004 3C028000 */ lui     $v0, %hi(some_sym)
+/* 0008 80000008 004E1021 */ addu    $v0, $v0, $t6
+/* 000C 8000000C 03E00008 */ jr      $ra
+/* 0010 80000010 8C420020 */  lw      $v0, %lo(some_sym)($v0)
+```
+
+While this assembly is very likely to build to a matching binary again, it may cause some issues under a few circunstances, like when a poject aims to achieve proper shiftability while it haven't been completely matched.
+
+To fix this disassembly it is needed to provide reloc entries in a `reloc_addrs.txt` file like the following:
+
+```ini
+rom:0x0004 reloc:MIPS_HI16 symbol:some_array addend:-0x4
+rom:0x0010 reloc:MIPS_LO16 symbol:some_array addend:-0x4
+```
+
+This tells the disassembler to reference `some_array - 0x4` at the instruction at rom address `0x0004` (the `lui` instruction), and that it should use the `%hi` reloc operator to do so. A similar logic is used for the instruction at rom address `0x0010` (the `lw`), but instead we told it to use the `%lw` reloc operator instead. This generates an assembly like the following:
+
+```mips
+/* 0000 80000000 00047080 */ sll     $t6, $a0, 0x2
+/* 0004 80000004 3C028000 */ lui     $v0, %hi(some_array - 0x4)
+/* 0008 80000008 004E1021 */ addu    $v0, $v0, $t6
+/* 000C 8000000C 03E00008 */ jr      $ra
+/* 0010 80000010 8C420020 */  lw      $v0, %lo(some_array - 0x4)($v0)
+```
+
+### Segment symbols in code
+
+A common pattern seen on N64 projects is when the code references special symbols known as "segment symbols" to load some fragments or segments of the ROM into VRAM. N64 games do this because it isn't possible to load the whole ROM into VRAM, and also the N64 games lack a proper filesystem.
+
+Segment symbols exist to describe addresses or sizes of specific regions of the ROM, like the start and end of the ROM addresses of a given segment, the expected start and end of the VRAM addresses of that segment, the size of the segment, etc.
+
+Sadly the disassembler is unable to properly disambiguate these segment addresses from other symbols or even plain numbers, so disassembly of code referencing segment symbols tend to be far from optimal.
+
+Take for example the following C code:
+
+```c
+/* segment symbols */
+u32 segment_menu_ROM_START[];
+u32 segment_menu_ROM_END[];
+
+void load_menu_segment(void *dst) {
+    load_segment(segment_menu_ROM_START, (u32)segment_menu_ROM_END - (u32)segment_menu_ROM_START, dst);
+}
+```
+
+After compiling this code, the generated assembly would look like the following:
+
+```mips
+addiu    $sp, $sp, -0x18
+sw       $ra, 0x10($sp)
+addu     $a2, $a0, $zero
+lui      $a0, %hi(segment_menu_ROM_START)
+addiu    $a0, $a0, %lo(segment_menu_ROM_START)
+lui      $a1, %hi(segment_menu_ROM_END)
+addiu    $a1, $a1, %lo(segment_menu_ROM_END)
+jal      load_segment
+ subu    $a1, $a1, $a0
+lw       $ra, 0x10($sp)
+addiu    $sp, $sp, 0x18
+jr       $ra
+ nop
+```
+
+But when this gets linked into a ROM those symbols get replaced with their raw numeric values, and since they point to rom addresses or vram addresses that are at the boundaries of each segment, the disassembler strugles symbolizing them, so it ends up symbolizing them into generic symbols, like the following:
+
+```mips
+/* 0000 80000000 27BDFFE8 */ addiu    $sp, $sp, -0x18
+/* 0004 80000004 AFBF0010 */ sw       $ra, 0x10($sp)
+/* 0008 80000008 00803021 */ addu     $a2, $a0, $zero
+/* 000C 8000000C 3C040000 */ lui      $a0, %hi(D_000FB480)
+/* 0010 80000010 24840000 */ addiu    $a0, $a0, %lo(D_000FB480)
+/* 0014 80000014 3C050000 */ lui      $a1, %hi(D_00101A80)
+/* 0018 80000018 24A50000 */ addiu    $a1, $a1, %lo(D_00101A80)
+/* 001C 8000001C 0C000000 */ jal      load_segment
+/* 0020 80000020 00A42823 */  subu    $a1, $a1, $a0
+/* 0024 80000024 8FBF0010 */ lw       $ra, 0x10($sp)
+/* 0028 80000028 27BD0018 */ addiu    $sp, $sp, 0x18
+/* 002C 8000002C 03E00008 */ jr       $ra
+/* 0030 80000030 00000000 */  nop
+```
+
+To fix the disassembly and make it use the proper segment symbols, we add more entries to the reloc_addrs.txt file. Note here we don't need to specify an `addend`, since we just want to refer to the symbol without any other calculation.
+
+```ini
+rom:0x000C reloc:MIPS_HI16 symbol:segment_menu_ROM_START
+rom:0x0010 reloc:MIPS_LO16 symbol:segment_menu_ROM_START
+rom:0x0014 reloc:MIPS_HI16 symbol:segment_menu_ROM_END
+rom:0x0018 reloc:MIPS_LO16 symbol:segment_menu_ROM_END
+```
+
+### Segment symbols in data
+
+On the other side, segment symbols may be referenced in data structures like arrays or structs. [Here you can read a bit more about what segment symbols are](#segment-symbols-in-code).
+
+Take for example the following C code and its corresponding compiled assembly:
+
+```c
+u32 segment_menu_ROM_START[];
+u32 segment_menu_ROM_END[];
+
+u32 *menu_addresses[] = {
+    segment_menu_ROM_START, segment_menu_ROM_END,
+}
+```
+
+```mips
+.word segment_menu_ROM_START
+.word segment_menu_ROM_END
+```
+
+But when we try disassembling a rom with this data, the disassembler won't be able to recognize these as segment symbols:
+
+```mips
+/* 0100 80000100 000FB480 */ .word D_000FB480 # It may use a D_ symbol
+/* 0104 80000104 000FB480 */ .word 0x000FB480 # Or it may even fail completely to symbolize it
+```
+
+In this case we can use the `MIPS_32` reloc in our `reloc_addrs` file to fix this kind of issue.
+
+```ini
+rom:0x0100 reloc:MIPS_32 symbol:segment_menu_ROM_START
+rom:0x0104 reloc:MIPS_32 symbol:segment_menu_ROM_END
+```

{splat64-0.34.3 → splat64-0.35.0}/docs/Configuration.md

@@ -122,6 +122,21 @@ String that is placed before the contents of newly-generated assembly (`.s`) fil
 generated_s_preamble: .set fp=64
 ```
 
+### generated_macro_inc_content
+
+String that is placed after the contents of the splat-generated `macro.inc` file.
+
+### generate_asm_macros_files
+
+Tells splat to regenerate files containing assembly macros and C macros each time splat is run.
+
+Specifically splat generates `include/include_asm.h`, `include/macro.inc`, `include/labels.inc` and `include/gte_macros.inc`, which contain the proper C and assembly macro definitions expected for building the generated assembly correctly. This allows splat to update the macro definitions with minimal user intervention and headaches.
+
+Turning this off can be useful in case the user wants to control exactly the contents of those files, but it is not recommended, since the user definitions may get outdated. Before turning this option off consider using the [`generated_macro_inc_content`](#generated_macro_inc_content) option to customize the contents of the generated `macro.inc` file.
+
+Some files may not be generated depending on the selected platform and compiler, because those setups don't require them. For example `include_asm.h` won't be generated if the compiler is set to `IDO` or `MWCCPS2`. `include/gte_macros.inc` is only generated on psx projects.
+
+Defaults to `True`.
 
 ### o_as_suffix
 
@@ -203,9 +218,15 @@ symbol_addrs_path: path/to/symbol_addrs
 
 
 
-### reloc_addrs_paths
+### reloc_addrs_path
+
+Determines the path to the reloc addresses file(s). A `reloc_addrs` file contains metadata to override relocations within the generated assembly. For more information about the syntax and how to use it refer to the corresponding [reloc_addrs chapter](Advanced-Reloc.md).
 
+It's possible to use more than one file by supplying a list instead of a string.
+
+#### Default
 
+`reloc_addrs.txt`
 
 ### build_path
 Path that built files will be found. Used for generation of the linker script.
@@ -647,19 +668,33 @@ Determines the macro used to declare functions in asm files
 
 ### asm_function_alt_macro
 
-Determines the macro used to declare symbols in the middle of functions in asm files (which may be alternative entries)
+Determines the macro used to declare symbols in the middle of functions in asm files (which may be alternative entries).
+
+Defaults to `alabel`.
 
 ### asm_jtbl_label_macro
 
-Determines the macro used to declare jumptable labels in asm files
+Determines the macro used to declare jumptable labels in asm files.
+
+Defaults to `jlabel`.
 
 ### asm_data_macro
 
-Determines the macro used to declare data symbols in asm files
+Determines the macro used to declare data symbols in asm files.
+
+Defaults to `dlabel`.
 
 ### asm_end_label
 
-Determines the macro used at the end of a function, such as endlabel or .end
+Determines the macro used at the end of a function, such as `endlabel` or `.end`.
+
+Defaults to `endlabel`.
+
+### asm_data_end_label
+
+Determines the macro used at the end of a data symbol.
+
+Defaults to `enddlabel`.
 
 ### asm_ehtable_label_macro
 
@@ -667,6 +702,14 @@ Determines the macro used to declare ehtable labels in asm files.
 
 Defaults to `ehlabel`
 
+### asm_nonmatching_label_macro
+
+Determines the macro used to declare the given symbol is a non matching one.
+
+Explicitly specifying that a symbol haven't been matched yet in the generated assembly is useful for other tools that consume the build artifacts of the project. This information can be used by those tools for stuff like progress reporting.
+
+Defaults to `nonmatching`
+
 ### asm_emit_size_directive
 
 Toggles the .size directive emitted by the disassembler

{splat64-0.34.3 → splat64-0.35.0}/docs/General-Workflow.md

@@ -118,7 +118,7 @@ The macros to include text/rodata assembly are different for GCC vs IDO compiler
 **GCC**: `INCLUDE_ASM` & `INCLUDE_RODATA` (text/rodata respectively)
 **IDO**: `GLOBAL_ASM`
 
-These macros must be defined in an included header, which splat currently does not produce.
+These macros must be defined in an included header, which splat generates and updates by default for GCC-based projects.
 
 For a GCC example, see the [include.h](https://github.com/AngheloAlf/drmario64/blob/master/include/include_asm.h) from the Dr. Mario project.
 
@@ -130,27 +130,75 @@ For MWCC, you will need [mwccgap](https://github.com/mkst/mwccgap) to include as
 
 splat relies on some assembly macros for the asm generation. They usually live on the `include/macro.inc` file. Without these macros then an assembler would not be able to build our disassemblies.
 
+By default splat will generate files with the required assembly macros.
+
 Those macros usually look like this:
 
 ```mips
-.macro glabel label
-    .global \label
+# A function symbol.
+.macro glabel label, visibility=global
+    .\visibility \label
     .type \label, @function
     \label:
+        .ent \label
 .endm
 
-.macro dlabel label
-    .global \label
+# The end of a function symbol.
+.macro endlabel label
+    .size \label, . - \label
+    .end \label
+.endm
+
+# An alternative entry to a function.
+.macro alabel label, visibility=global
+    .\visibility \label
+    .type \label, @function
     \label:
+        .aent \label
 .endm
 
+# A label referenced by an error handler table.
+.macro ehlabel label, visibility=global
+    .\visibility \label
+    \label:
+.endm
+
+
+# A label referenced by a jumptable.
 .macro jlabel label
     .global \label
     \label:
 .endm
+
+
+# A data symbol.
+.macro dlabel label, visibility=global
+    .\visibility \label
+    .type \label, @object
+    \label:
+.endm
+
+# End of a data symbol.
+.macro enddlabel label
+    .size \label, . - \label
+.endm
+
+
+# Label to signal the symbol haven't been matched yet.
+.macro nonmatching label, size=1
+    .global \label\().NON_MATCHING
+    .type \label\().NON_MATCHING, @object
+    .size \label\().NON_MATCHING, \size
+    \label\().NON_MATCHING:
+.endm
 ```
 
-Where `glabel` is used for functions, `dlabel` is used for data, rodata and bss variables and `jlabel` is used for branch labels used by jumptables.
+The most commonly used labels are:
+
+- `glabel` and `endlabel` which are used to define function symbols.
+- `dlabel` and `enddlabel` which are used to defined data, rodata and bss symbols.
+- `jlabel` is used for defining branch labels used by jumptables.
+- `nonmatching` is used to define the symbol haven't been matched yet.
 
 Asm differ tools can sometimes struggle to show diffs with `jlabel`s when combined with certain compilers. A workaround for this issue is to mark the `jlabel` as a function, like this:
 

{splat64-0.34.3 → splat64-0.35.0}/docs/Segments.md

@@ -583,3 +583,36 @@ Defaults to the global option.
     vram: 0x80000460
     suggestion_rodata_section_start: False
 ```
+
+### `pair_segment`
+
+Allows pairing sections of two different segments together.
+
+The main purpose of this is to make the automatic rodata-to-function migration possible, since the default behavior only allows pairing different sections of the same name under the same segment only. This kind of ROM layout can be seen on some TLB games from N64 projects.
+
+This value expects the name of the other segment that should be paired to the current one. Only one of the two to-be-paired segments should have this attribute.
+
+**Example:**
+
+```yaml
+  - name:  init
+    type:  code
+    start: 0x00001000
+    vram:  0x10001000
+    pair_segment: init_data # This is the name of the following segment.
+    subsegments:
+      # -- snip --
+      - [0x15550, c, libultra/audio/init_15550]
+      # -- snip --
+
+  - name:  init_data
+    type:  code
+    start: 0x000290D0
+    vram:  0x800290D0
+    bss_size: 0x16690
+    # Note there's no `pair_segment: init` on this segment.
+    subsegments:
+      # -- snip --
+      - [0x2C6B0, .rodata, libultra/audio/init_15550]
+      # -- snip --
+```

{splat64-0.34.3 → splat64-0.35.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "splat64"
 # Should be synced with src/splat/__init__.py
-version = "0.34.3"
+version = "0.35.0"
 description = "A binary splitting tool to assist with decompilation and modding projects"
 readme = "README.md"
 license = {file = "LICENSE"}

{splat64-0.34.3 → splat64-0.35.0}/requirements.txt

@@ -4,7 +4,7 @@ tqdm
 intervaltree
 colorama
 # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py and pyproject.toml
-spimdisasm>=1.33.0
+spimdisasm>=1.36.0
 rabbitizer>=1.10.0
 pygfxd
 n64img>=0.1.4

{splat64-0.34.3 → splat64-0.35.0}/src/splat/__init__.py

@@ -1,7 +1,7 @@
 __package_name__ = __name__
 
 # Should be synced with pyproject.toml
-__version__ = "0.34.3"
+__version__ = "0.35.0"
 __author__ = "ethteck"
 
 from . import util as util

{splat64-0.34.3 → splat64-0.35.0}/src/splat/disassembler/spimdisasm_disassembler.py

@@ -7,7 +7,7 @@ from typing import Set
 
 class SpimdisasmDisassembler(disassembler.Disassembler):
     # This value should be kept in sync with the version listed on requirements.txt and pyproject.toml
-    SPIMDISASM_MIN = (1, 33, 0)
+    SPIMDISASM_MIN = (1, 36, 0)
 
     def configure(self):
         # Configure spimdisasm
@@ -93,9 +93,15 @@ class SpimdisasmDisassembler(disassembler.Disassembler):
         )
         spimdisasm.common.GlobalConfig.ASM_DATA_LABEL = options.opts.asm_data_macro
         spimdisasm.common.GlobalConfig.ASM_TEXT_END_LABEL = options.opts.asm_end_label
+        spimdisasm.common.GlobalConfig.ASM_DATA_END_LABEL = (
+            options.opts.asm_data_end_label
+        )
         spimdisasm.common.GlobalConfig.ASM_EHTBL_LABEL = (
             options.opts.asm_ehtable_label_macro
         )
+        spimdisasm.common.GlobalConfig.ASM_NM_LABEL = (
+            options.opts.asm_nonmatching_label_macro
+        )
 
         if options.opts.asm_emit_size_directive is not None:
             spimdisasm.common.GlobalConfig.ASM_EMIT_SIZE_DIRECTIVE = (