varbert 2.0.0__tar.gz

varbert-2.0.0/LICENSE

@@ -0,0 +1,24 @@
+BSD 2-Clause License
+
+Copyright (c) 2023, BinSync
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

varbert-2.0.0/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.1
+Name: varbert
+Version: 2.0.0
+Summary: The VarBERT API for renaming variables in decompiled code.
+License: BSD 2 Clause
+Project-URL: Homepage, https://github.com/binsync/varbert_api
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch
+Requires-Dist: transformers
+Requires-Dist: tqdm
+Requires-Dist: dailalib
+Requires-Dist: libbs
+
+# VarBERT API
+The VarBERT API is a Python library to access and use the latest models from the S&P 2024 work 
+[""Len or index or count, anything but v1": Predicting Variable Names in Decompilation Output with Transfer Learning"](), featuring VarBERT. 
+VarBERT is a BERT-based model that predicts variable names for decompiled code.
+To train new models and understand the pipeline, see the [VarBERT paper repo]().
+Specialized models exist for IDA Pro and Ghidra, but can be used on any decompiler. 
+
+<p align="center">
+    <img src="./assets/varbert_no_background.png" style="width: 50%;" alt="DAILA context menu"/>
+</p>
+
+The main focus of this project is to provide an library API and CLI access to VarBERT models, but, it has 
+been designed to be used in decompiler directly using the [DAILA](https://github.com/mahaloz/DAILA) project. 
+DAILA comes with the VarBERT API bundled, so you do not need to install VarBERT if you are using DAILA.
+
+## Install 
+```
+pip3 install varbert && varbert --download-models
+```
+
+This will install the VarBERT API library and download the models to be stored inside the VarBERT package.
+You can optionally provide a decompiler name to `--download-models` to only download the models for that decompiler.
+
+## Usage
+The VarBERT API can be used in three ways:
+- From the CLI, directly on decompiled text (without an attached decompiler)
+- As a scripting library 
+- As a decompiler plugin (using [DALIA](https://github.com/mahaloz/DAILA)) 
+
+### Command Line (without running a decompiler)
+Note that VarBERT runs better when it is directly hooked up to a decompiler because it can use additional semantic information that the decompiler knows about the decompiled code.
+However, we do have the ability to run VarBERT without a running decompiler, only operating on the text from the command line.
+
+Running the following will cause VarBERT to read a function from standard input and output the function with predicted variable names to standard out:
+```bash
+varbert --predict --decompiler ida
+```
+
+You can select different decompilers that will use different models that are trained on the different decompilers.
+If you do not specify a decompiler, the default is IDA Pro.
+As an example, you can also give no decompiler:
+```bash 
+ echo "__int64 sub_400664(char *a1,char *a2)\n {}" | varbert -p
+```
+
+### Scripting
+#### Without Decompiler
+```python
+from varbert import VariableRenamingAPI
+api = VariableRenamingAPI(decompiler_name="ida", use_decompiler=False)
+new_names, new_code = api.predict_variable_names(decompilation_text="__int64 sub_400664(char *a1,char *a2)\n {}", use_decompiler=False)
+print(new_code)
+```
+
+You can also find more examples in the [tests.py](./tests/tests.py) file.
+
+#### Inside Decompiler
+You can use VarBERT as a scripting library inside your decompiler, utilizing LibBS.
+```python
+from varbert import VariableRenamingAPI
+from libbs.api import DecompilerInterface
+dec = DecompilerInterface()
+api = VariableRenamingAPI(decompiler_interface=dec)
+for func_addr in dec.functions:
+    new_names, new_code = api.predict_variable_names(function=dec.functions[func_addr])
+    print(new_names)
+```
+
+### As a Decompiler Plugin
+If you would like to use VarBERT as a decompiler plugin, you can use [DAILA](https://github.com/mahaloz/DAILA).
+You should follow the instructions on the DAILA repo to install DAILA, but it's generally as simple as:
+```bash
+pip3 install dailalib && daila --install
+```
+
+## Citing 
+If you use VarBERT in your research, please cite our paper:
+```
+TODO
+```

varbert-2.0.0/README.md

@@ -0,0 +1,80 @@
+# VarBERT API
+The VarBERT API is a Python library to access and use the latest models from the S&P 2024 work 
+[""Len or index or count, anything but v1": Predicting Variable Names in Decompilation Output with Transfer Learning"](), featuring VarBERT. 
+VarBERT is a BERT-based model that predicts variable names for decompiled code.
+To train new models and understand the pipeline, see the [VarBERT paper repo]().
+Specialized models exist for IDA Pro and Ghidra, but can be used on any decompiler. 
+
+<p align="center">
+    <img src="./assets/varbert_no_background.png" style="width: 50%;" alt="DAILA context menu"/>
+</p>
+
+The main focus of this project is to provide an library API and CLI access to VarBERT models, but, it has 
+been designed to be used in decompiler directly using the [DAILA](https://github.com/mahaloz/DAILA) project. 
+DAILA comes with the VarBERT API bundled, so you do not need to install VarBERT if you are using DAILA.
+
+## Install 
+```
+pip3 install varbert && varbert --download-models
+```
+
+This will install the VarBERT API library and download the models to be stored inside the VarBERT package.
+You can optionally provide a decompiler name to `--download-models` to only download the models for that decompiler.
+
+## Usage
+The VarBERT API can be used in three ways:
+- From the CLI, directly on decompiled text (without an attached decompiler)
+- As a scripting library 
+- As a decompiler plugin (using [DALIA](https://github.com/mahaloz/DAILA)) 
+
+### Command Line (without running a decompiler)
+Note that VarBERT runs better when it is directly hooked up to a decompiler because it can use additional semantic information that the decompiler knows about the decompiled code.
+However, we do have the ability to run VarBERT without a running decompiler, only operating on the text from the command line.
+
+Running the following will cause VarBERT to read a function from standard input and output the function with predicted variable names to standard out:
+```bash
+varbert --predict --decompiler ida
+```
+
+You can select different decompilers that will use different models that are trained on the different decompilers.
+If you do not specify a decompiler, the default is IDA Pro.
+As an example, you can also give no decompiler:
+```bash 
+ echo "__int64 sub_400664(char *a1,char *a2)\n {}" | varbert -p
+```
+
+### Scripting
+#### Without Decompiler
+```python
+from varbert import VariableRenamingAPI
+api = VariableRenamingAPI(decompiler_name="ida", use_decompiler=False)
+new_names, new_code = api.predict_variable_names(decompilation_text="__int64 sub_400664(char *a1,char *a2)\n {}", use_decompiler=False)
+print(new_code)
+```
+
+You can also find more examples in the [tests.py](./tests/tests.py) file.
+
+#### Inside Decompiler
+You can use VarBERT as a scripting library inside your decompiler, utilizing LibBS.
+```python
+from varbert import VariableRenamingAPI
+from libbs.api import DecompilerInterface
+dec = DecompilerInterface()
+api = VariableRenamingAPI(decompiler_interface=dec)
+for func_addr in dec.functions:
+    new_names, new_code = api.predict_variable_names(function=dec.functions[func_addr])
+    print(new_names)
+```
+
+### As a Decompiler Plugin
+If you would like to use VarBERT as a decompiler plugin, you can use [DAILA](https://github.com/mahaloz/DAILA).
+You should follow the instructions on the DAILA repo to install DAILA, but it's generally as simple as:
+```bash
+pip3 install dailalib && daila --install
+```
+
+## Citing 
+If you use VarBERT in your research, please cite our paper:
+```
+TODO
+```

varbert-2.0.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["setuptools>=61.2"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "varbert"
+classifiers = [
+    "License :: OSI Approved :: BSD License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+]
+license = {text = "BSD 2 Clause"}
+description = "The VarBERT API for renaming variables in decompiled code."
+urls = {Homepage = "https://github.com/binsync/varbert_api"}
+requires-python = ">= 3.8"
+dependencies = [
+    "torch",
+    "transformers",
+    "tqdm",
+    "dailalib",
+    "libbs"
+]
+dynamic = ["version"]
+
+[project.readme]
+file = "README.md"
+content-type = "text/markdown"
+
+[project.scripts]
+varbert = "varbert.__main__:main"
+
+[tool.setuptools]
+include-package-data = true
+license-files = ["LICENSE"]
+
+[tool.setuptools.packages]
+find = {namespaces = false}
+
+[tool.setuptools.dynamic]
+version = {attr = "varbert.__version__"}
+

varbert-2.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

varbert-2.0.0/tests/tests.py

@@ -0,0 +1,321 @@
+import unittest
+import sys
+from typing import Dict
+
+from varbert import VariableRenamingAPI
+from yodalib.data import Function, FunctionArgument, FunctionHeader, StackVariable
+
+
+def function_with_new_names(function: Function, new_names: Dict[str, str]):
+    new_func: Function = function.copy()
+    for old_name, new_name in new_names.items():
+        for _, arg in new_func.args.items():
+            if arg.name == old_name:
+                arg.name = new_name
+
+        for _, svar in new_func.stack_vars.items():
+            if svar.name == old_name:
+                svar.name = new_name
+
+    return new_func
+
+
+class TestBinSyncRenaming(unittest.TestCase):
+    def test_renaming(self):
+        api = VariableRenamingAPI(use_decompiler=False, decompiler_name="ida")
+
+        # testing text
+        function_text = "__int64 __fastcall sub_5E007(__int64 a1, __int64 a2, const char *a3)\n{\n  __int64 v3; // rcx\n  __int64 v4; // r8\n  __int64 v5; // r9\n  unsigned int v7; // [rsp+20h] [rbp-10h]\n  int v8; // [rsp+24h] [rbp-Ch]\n  __int64 v9; // [rsp+28h] [rbp-8h]\n\n  v9 = qword_246250;\n  v7 = 0;\n  v8 = atoi(a3);\n  if ( v8 >= 0 )\n  {\n    while ( v9 )\n    {\n      *(v9 + 3240) = v8;\n      v9 = *(v9 + 21664);\n      ++v7;\n    }\n    if ( a2 )\n      sub_33178(\n        4,\n        \"WARN: [%s] plugin name not supported for key 'telemetry_dump_kafka_topic_rr'. Globalized.\\n\",\n        a1,\n        v3,\n        v4,\n        v5);\n    return v7;\n  }\n  else\n  {\n    sub_33178(4, \"WARN: [%s] 'telemetry_dump_kafka_topic_rr' has to be >= 0.\\n\", a1, v3, v4, v5);\n    return 0xFFFFFFFFLL;\n  }\n}\n// 5E064: variable 'v3' is possibly undefined\n// 5E064: variable 'v4' is possibly undefined\n// 5E064: variable 'v5' is possibly undefined\n// 246250: using guessed type __int64 qword_246250;\n"
+        svar_name_data = ["v3", "v4", "v5", "v7", "v8", "v9"]
+        args_name_data = ["a1", "a2", "a3"]
+
+        # WARNING: the offsets that these stack variables have as BinSync objects are not real and are only for this testcase
+        function = Function(0xdead, 0x1337, header=FunctionHeader("sub_5E007", 0xdead, args={}), stack_vars={})
+        for i, name in enumerate(svar_name_data):
+            function.stack_vars[i] = StackVariable(i, name, None, 8, function.addr)
+        for i, name in enumerate(args_name_data):
+            function.args[i] = FunctionArgument(i, name, None, 8)
+
+        new_names, _ = api.predict_variable_names(function, decompilation_text=function_text, use_decompiler=False)
+        new_function = function_with_new_names(function, new_names)
+
+        assert new_function.args[0] != function.args[0]
+        assert new_function.args[1] != function.args[1]
+        assert new_function.args[2] != function.args[2]
+
+        assert new_function.stack_vars[3] != function.stack_vars[3]
+
+    def test_renaming_1(self):
+        api = VariableRenamingAPI(use_decompiler=False, decompiler_name="ida")
+
+        # testing text
+        function_text = "_int64 __fastcall main(int a1, char **a2, char **a3)\n{\n  _BOOL4 v4; // [rsp+1Ch] [rbp-24h] BYREF\n  char v5[16]; // [rsp+20h] [rbp-20h] BYREF\n  char buf[16]; // [rsp+30h] [rbp-10h] BYREF\n\n  buf[8] = 0;\n  v5[8] = 0;\n  puts(\"Username: \");\n  read(0, buf, 8uLL);\n  read(0, &v4, 1uLL);\n  puts(\"Password: \");\n  read(0, v5, 8uLL);\n  read(0, &v4, 1uLL);\n  v4 = sub_400664(buf, v5);\n  if ( !v4 )\n    sub_4006FD();\n  return sub_4006ED(buf);\n}\n"
+        svar_name_data = ['v4', 'v5', 'buf']
+        args_name_data = ["a1", "a2", "a3"]
+
+        # WARNING: the offsets that these stack variables have as BinSync objects are not real and are only for this testcase
+        function = Function(0xdead, 0x1337, header=FunctionHeader("main", 0xdead, args={}), stack_vars={})
+        for i, name in enumerate(svar_name_data):
+            function.stack_vars[i] = StackVariable(i, name, None, 8, function.addr)
+        for i, name in enumerate(args_name_data):
+            function.args[i] = FunctionArgument(i, name, None, 8)
+
+        new_names, _ = api.predict_variable_names(function, decompilation_text=function_text, use_decompiler=False)
+        new_function = function_with_new_names(function, new_names)
+
+        assert new_function.args[0] != function.args[0]
+        assert new_function.args[1] != function.args[1]
+        assert new_function.args[2] != function.args[2]
+
+        assert new_function.stack_vars[1] != function.stack_vars[1]
+        assert new_function.stack_vars[2] != function.stack_vars[2]
+    def test_renaming_2(self):
+        api = VariableRenamingAPI(use_decompiler=False, decompiler_name="ida")
+
+        # testing text
+        function_text = '''__int64 __fastcall main(int a1, char **a2, char **a3, char a4)
+        {
+        const char *v5; // rbp
+
+        if ( a1 != 2 )
+            return 0LL;
+        sub_2B30(*a2);
+        setlocale(6, "");
+        bindtextdomain("coreutils", "/usr/share/locale");
+        textdomain("coreutils");
+        sub_5550(sub_2A80);
+        v5 = a2[1];
+        if ( !strcmp(v5, "--help") )
+            sub_2700(0);
+        if ( !strcmp(v5, "--version") )
+            sub_4DC0(stdout, "true", &unk_6084, Version, "Jim Meyering", 0, a4);
+        return 0LL;
+        }
+        '''
+        svar_name_data = ['v5']
+        args_name_data = ["a1", "a2", "a3", "a4"]
+        # WARNING: the offsets that these stack variables have as BinSync objects are not real and are only for this testcase
+        function = Function(0xdead, 0x1337, header=FunctionHeader("main", 0xdead, args={}), stack_vars={})
+        for i, name in enumerate(svar_name_data):
+            function.stack_vars[i] = StackVariable(i, name, None, 8, function.addr)
+        for i, name in enumerate(args_name_data):
+            function.args[i] = FunctionArgument(i, name, None, 8)
+
+        new_names, _ = api.predict_variable_names(function, decompilation_text=function_text, use_decompiler=False)
+        new_function = function_with_new_names(function, new_names)
+
+        assert new_function.args[0] != function.args[0]
+        assert new_function.args[1] != function.args[1]
+        assert new_function.args[2] != function.args[2]
+        assert new_function.args[3] != function.args[3]
+
+    def test_renaming_3(self):
+        api = VariableRenamingAPI(use_decompiler=False, decompiler_name="ida")
+
+        # testing text
+        function_text = '''  __int64 __fastcall sub_4760(__int64 a1, __int64 a2, __int64 a3, __int64 a4, __int64 a5)
+        {
+        __int128 v6[2]; // [rsp+0h] [rbp-48h] BYREF
+        __m128i si128; // [rsp+20h] [rbp-28h]
+        __int64 v8; // [rsp+30h] [rbp-18h]
+        unsigned __int64 v9; // [rsp+38h] [rbp-10h]
+
+        v9 = __readfsqword(0x28u);
+        v6[0] = _mm_load_si128(&xmmword_A1E0);
+        v8 = qword_A210;
+        LODWORD(v6[0]) = 10;
+        v6[1] = _mm_load_si128(&xmmword_A1F0);
+        si128 = _mm_load_si128(&xmmword_A200);
+        if ( !a2 || !a3 )
+            abort();
+        si128.m128i_i64[1] = a2;
+        v8 = a3;
+        return sub_3F20(a1, a4, a5, v6);
+        }'''
+        svar_name_data = ['v6', "s128", "v8", "v9"]
+        args_name_data = ["a1", "a2", "a3", "a4", "a5"]
+        # WARNING: the offsets that these stack variables have as BinSync objects are not real and are only for this testcase
+        function = Function(0xdead, 0x1337, header=FunctionHeader("sub_4760", 0xdead, args={}), stack_vars={})
+        for i, name in enumerate(svar_name_data):
+            function.stack_vars[i] = StackVariable(i, name, None, 8, function.addr)
+        for i, name in enumerate(args_name_data):
+            function.args[i] = FunctionArgument(i, name, None, 8)
+
+        new_names, _ = api.predict_variable_names(function, decompilation_text=function_text, use_decompiler=False)
+        new_function = function_with_new_names(function, new_names)
+
+        assert new_function.args[0] != function.args[0]
+        assert new_function.args[1] != function.args[1]
+        assert new_function.args[2] != function.args[2]
+        assert new_function.args[3] != function.args[3]
+        assert new_function.args[4] != function.args[4]
+
+    def test_renaming_4(self):
+        api = VariableRenamingAPI(use_decompiler=False, decompiler_name="ida")
+
+        # testing text
+        function_text =   '''__int64 __fastcall main(int a1, char **a2, char **a3)
+        {
+        char *v4; // r12
+        char *v5; // rax
+        char *v6; // rbp
+        const char *v7; // r13
+        const char *v8; // r13
+        char *v9; // rax
+        char *v10; // rax
+        FILE *v11; // r12
+        char *v12; // rax
+        FILE *v13; // r12
+        char *v14; // rax
+        char *v15; // rax
+        __int64 *v16; // rbp
+        const char *v17; // rsi
+        const char *v18; // r14
+        char *v19; // rax
+        char *v20; // rdi
+        char *v21; // rax
+        char *v22; // r12
+        char *v23; // rax
+        char *v24; // rax
+        char *v25; // rdi
+        char *v26; // rax
+        FILE *v27; // rbp
+        char *v28; // rax
+        __int64 v29[21]; // [rsp+0h] [rbp-A8h] BYREF
+
+        v29[15] = __readfsqword(0x28u);
+        if ( a1 != 2 )
+            return 0LL;
+        v4 = *a2;
+        if ( !*a2 )
+        {
+            fwrite("A NULL argv[0] was passed through an exec system call.\n", 1uLL, 0x37uLL, stderr);
+            abort();
+        }
+        v5 = strrchr(v4, 47);
+        v6 = v5;
+        if ( v5 )
+        {
+            v7 = v5 + 1;
+            if ( v5 + 1 - v4 > 6 && !strncmp(v5 - 6, "/.libs/", 7uLL) )
+            {
+            v4 = v7;
+            if ( !strncmp(v7, "lt-", 3uLL) )
+            {
+                v4 = v6 + 4;
+                program_invocation_short_name = v6 + 4;
+            }
+            }
+        }
+        qword_7028 = v4;
+        program_invocation_name = v4;
+        setlocale(6, "");
+        bindtextdomain("coreutils", "/usr/share/locale");
+        textdomain("coreutils");
+        sub_39D0(sub_3390);
+        v8 = a2[1];
+        if ( !strcmp(v8, "--help") )
+        {
+            v9 = dcgettext(0LL, "Usage: %s [ignored command line arguments]\n  or:  %s OPTION\n", 5);
+            __printf_chk(1LL, v9, v4, v4);
+            v10 = dcgettext(0LL, "Exit with a status code indicating success.", 5);
+            __printf_chk(1LL, "%s\n\n", v10);
+            v11 = stdout;
+            v12 = dcgettext(0LL, "      --help     display this help and exit\n", 5);
+            fputs_unlocked(v12, v11);
+            v13 = stdout;
+            v14 = dcgettext(0LL, "      --version  output version information and exit\n", 5);
+            fputs_unlocked(v14, v13);
+            v15 = dcgettext(
+                    0LL,
+                    "\n"
+                    "NOTE: your shell may have its own version of %s, which usually supersedes\n"
+                    "the version described here.  Please refer to your shell's documentation\n"
+                    "for details about the options it supports.\n",
+                    5);
+            __printf_chk(1LL, v15, "true");
+            v29[2] = "coreutils";
+            v29[1] = "test invocation";
+            v16 = v29;
+            v17 = "[";
+            v29[3] = "Multi-call invocation";
+            v29[6] = "sha256sum";
+            v29[4] = "sha224sum";
+            v29[8] = "sha384sum";
+            v29[0] = "[";
+            v29[5] = "sha2 utilities";
+            v29[7] = "sha2 utilities";
+            v29[9] = "sha2 utilities";
+            v29[10] = "sha512sum";
+            v29[11] = "sha2 utilities";
+            v29[12] = 0LL;
+            v29[13] = 0LL;
+            do
+            {
+            if ( !strcmp("true", v17) )
+                break;
+            v17 = v16[2];
+            v16 += 2;
+            }
+            while ( v17 );
+            v18 = v16[1];
+            if ( v18 )
+            {
+            v19 = dcgettext(0LL, "\n%s online help: <%s>\n", 5);
+            __printf_chk(1LL, v19, &unk_4047, "https://www.gnu.org/software/coreutils/");
+            v20 = setlocale(5, 0LL);
+            if ( !v20 || !strncmp(v20, "en_", 3uLL) )
+            {
+        LABEL_19:
+                v21 = dcgettext(0LL, "Full documentation <%s%s>\n", 5);
+                v22 = " invocation";
+                __printf_chk(1LL, v21, "https://www.gnu.org/software/coreutils/", "true");
+                if ( v18 != "true" )
+                v22 = "";
+        LABEL_21:
+                v23 = dcgettext(0LL, "or available locally via: info '(coreutils) %s%s'\n", 5);
+                __printf_chk(1LL, v23, v18, v22);
+                exit(0);
+            }
+            }
+            else
+            {
+            v24 = dcgettext(0LL, "\n%s online help: <%s>\n", 5);
+            __printf_chk(1LL, v24, &unk_4047, "https://www.gnu.org/software/coreutils/");
+            v25 = setlocale(5, 0LL);
+            if ( !v25 || !strncmp(v25, "en_", 3uLL) )
+            {
+                v26 = dcgettext(0LL, "Full documentation <%s%s>\n", 5);
+                v18 = "true";
+                __printf_chk(1LL, v26, "https://www.gnu.org/software/coreutils/", "true");
+                v22 = " invocation";
+                goto LABEL_21;
+            }
+            v18 = "true";
+            }
+            v27 = stdout;
+            v28 = dcgettext(0LL, "Report any translation bugs to <https://translationproject.org/team/>\n", 5);
+            fputs_unlocked(v28, v27);
+            goto LABEL_19;
+        }
+        if ( !strcmp(v8, "--version") )
+            sub_3410(stdout, v29[0]);
+        return 0LL;
+        }
+        '''
+        svar_name_data = ['v4', 'v5', 'v6', 'v7', 'v8', 'v9', 'v10', 'v11', 'v12', 'v13', 'v14', 'v15', 'v16', 'v17', 'v18', 'v19',
+                          'v20', 'v21', 'v22', 'v23', 'v24', 'v25', 'v26', 'v27', 'v28', 'v29']
+        args_name_data = ["a1", "a2", "a3"]
+        # WARNING: the offsets that these stack variables have as BinSync objects are not real and are only for this testcase
+        function = Function(0xdead, 0x1337, header=FunctionHeader("sub_4760", 0xdead, args={}), stack_vars={})
+        for i, name in enumerate(svar_name_data):
+            function.stack_vars[i] = StackVariable(i, name, None, 8, function.addr)
+        for i, name in enumerate(args_name_data):
+            function.args[i] = FunctionArgument(i, name, None, 8)
+
+        new_names, _ = api.predict_variable_names(function, decompilation_text=function_text, use_decompiler=False)
+        assert new_names != {}
+
+
+if __name__ == "__main__":
+    unittest.main(argv=sys.argv)

varbert-2.0.0/varbert/__init__.py

@@ -0,0 +1,108 @@
+__version__ = "2.0.0"
+
+import importlib.resources
+import tarfile
+from pathlib import Path
+import urllib.request
+import hashlib
+import math
+import platform
+import shutil
+
+from tqdm import tqdm
+from libbs.decompilers import GHIDRA_DECOMPILER, IDA_DECOMPILER
+
+# initialize logging for the entire project
+import logging
+logging.getLogger("varbert").addHandler(logging.NullHandler())
+from .logger import Loggers
+loggers = Loggers()
+del Loggers
+
+from .api import VariableRenamingAPI
+
+MODELS_PATH = Path(Path(str(importlib.resources.files("varbert"))) / "models").absolute()
+SUPPORTED_MODELS = {GHIDRA_DECOMPILER, IDA_DECOMPILER}
+SUBSTITUTE_DECOMPILER_MODEL = IDA_DECOMPILER
+MODEL_FOLDER = "DECOMPILER-OPT-Function"
+# all models are found here: https://www.dropbox.com/scl/fo/socl7rd5lsv926whylqpn/h?rlkey=i0x74bdipj41hys5rorflxawo
+MODEL_URLS = {
+    # function based models:
+    f"{GHIDRA_DECOMPILER}-O0": "https://www.dropbox.com/scl/fi/8xsmmlzypd45icn8csk6y/Ghidra-O0-Function.tar.gz?rlkey=1b92b9ejktoyewjztvo3ns8q1&dl=1",
+    f"{IDA_DECOMPILER}-O0": "https://www.dropbox.com/scl/fi/dmmfqqwvwhkswiv48ltfs/IDA-O0-Function.tar.gz?rlkey=3unxmiydbm5si3n7jh5r43qjp&dl=1",
+    f"{GHIDRA_DECOMPILER}-O2": "https://www.dropbox.com/scl/fi/x5ci28s0aw3i852kg9w1j/Ghidra-O2-Function.tar.gz?rlkey=wpe08afvxelcblgcqndrxmvtm&dl=1",
+    f"{IDA_DECOMPILER}-O2": "https://www.dropbox.com/scl/fi/ku26eebbwvug5fu2pc4ek/IDA-O2-Function.tar.gz?rlkey=edlri604hhuohh8n5d7d02tnd&dl=1",
+    # binary based models:
+    #f"{GHIDRA_DECOMPILER}-O2": "https://www.dropbox.com/scl/fi/nbk5b068z6ffsdl0kgbuw/Ghidra-O2-Binary.tar.gz?rlkey=m83iit4jh5fg6icl5cf2z3yhq&dl=1",
+    #f"{IDA_DECOMPILER}-O2": "https://www.dropbox.com/scl/fi/vk0ybwu4uoru4fl61yztw/IDA-O2-Binary.tar.gz?rlkey=9rt8js8qrhkqp2cvvttxrlwd2&dl=1",
+}
+
+_l = logging.getLogger(__name__)
+
+
+def install_model(decompiler, opt_level="O0", reinstall=False):
+    if decompiler not in SUPPORTED_MODELS:
+        _l.warning("Model for decompiler is not supported yet, using model for %s", SUBSTITUTE_DECOMPILER_MODEL)
+        decompiler = SUBSTITUTE_DECOMPILER_MODEL
+
+    # check if the model exists
+    decompiler_model = MODELS_PATH / decompiler
+    if decompiler_model.exists():
+        if reinstall:
+            shutil.rmtree(decompiler_model)
+        else:
+            _l.info(f"Model for {decompiler} already exists. Skipping download.")
+            return
+
+    # saved models on the remote side have some messed up names, so we have to do some
+    # string matching here to make sure we download and move the correct stuff
+    compliant_decompiler_name = decompiler
+    if decompiler == GHIDRA_DECOMPILER:
+        decompiler = "Ghidra"
+    elif decompiler == IDA_DECOMPILER:
+        decompiler = "IDA"
+
+    dl_model_folder = MODELS_PATH / Path(MODEL_FOLDER.replace("DECOMPILER", decompiler).replace("OPT", opt_level))
+    url = MODEL_URLS[f"{compliant_decompiler_name}-{opt_level}"]
+    _l.info(f"Downloading model for {compliant_decompiler_name} now...")
+    tar_file_path = _download_file(url, MODELS_PATH / f"model.tar.gz")
+    with tarfile.open(tar_file_path, "r:gz") as tar:
+        tar.extractall(path=MODELS_PATH)
+
+    # move the model folder to be in a compliant form
+    dl_model_folder.rename(MODELS_PATH / compliant_decompiler_name)
+    # delete the old tar
+    tar_file_path.unlink()
+
+
+def _download_file(url: str, save_location: Path, verify_hash=False) -> Path:
+    # XXX: hacked code for non-ssl verification
+    if platform.system() == "Darwin":
+        import ssl
+        ssl._create_default_https_context = ssl._create_unverified_context
+
+    with urllib.request.urlopen(url) as response:
+        total_size = response.length
+        if response.status != 200:
+            raise Exception(f"HTTP error {response.status}: {response.reason}")
+
+        hasher = hashlib.md5()
+        chunk_size = 8192
+        mb_size = int(total_size / 1000000)
+        with open(save_location, 'wb') as f:
+            for _ in tqdm(range(math.ceil(total_size / chunk_size)), desc=f"Downloading model ~{mb_size} MB..."):
+                chunk = response.read(chunk_size)
+                hasher.update(chunk)
+                if not chunk:
+                    break
+
+                f.write(chunk)
+
+        # hash for extra security
+        #download_hash = hasher.hexdigest()
+        #if verify_hash and download_hash != JOERN_ZIP_HASH:
+        #    raise Exception(f"Files corrupted in download: {download_hash} != {JOERN_ZIP_HASH}")
+
+    return save_location
+
+