metasm 1.0.0

data/BUGS

@@ -0,0 +1,11 @@
+List of known bugs/missing features, in no particular order:
+
+PPC cpu cannot parse/encode code
+Disassembler is sloooow
+The GTK UI is quite sluggish too
+Disassembler backtracker does weird things
+Mach-O encoder does not work (binaries won't load on OSX)
+ELF encoder may need tweaks to handle OpenBSD
+Ia32 compile_c misses many features (divisions, bitfields), and needs a register allocator
+Asm parser does not handle well ; comments (eg "foo ; */* blargimdead") (c-style comments are parsed before asm-style, so multiline /* after ; is bad)
+The BUGS file is incomplete

data/CREDITS

@@ -0,0 +1,17 @@
+N: Yoann GUILLOT
+E: yoann at ofjj.net
+D: Lead developper
+
+N: Julien TINNES
+E: julien at cr0.org
+D: Senior Product Manager
+D: Ideas, bug hunting, Yoann-slapping
+D: Metasploit integration
+
+N: Arnaud CORNET
+E: arnaud.cornet at gmail.com
+D: Initial ELF support
+
+N: Raphael RIGO
+E: raphael at cr0.org
+D: Initial MIPS support and misc stuff

data/README

@@ -0,0 +1,270 @@
+Metasm, the Ruby assembly manipulation suite
+============================================
+
+* sample scripts in samples/ -- read comments at the beginning of the files
+* all files are licensed under the terms of the LGPL
+
+Author: Yoann Guillot <john at ofjj.net>
+
+
+Basic overview:
+
+Metasm allows you to interact with executables formats (ExeFormat):
+PE, ELF, Mach-O, Shellcode, etc.
+There are three approaches to an ExeFormat:
+ - compiling one up, from scratch
+ - decompiling an existing format
+ - manipulating the file structure
+
+
+Ready-to-use scripts can be found in the samples/ subdirectory, check the
+comments in the scripts headers. You can also try the --help argument if
+you're feeling lucky.
+
+
+Here is a short overview of the Metasm internals.
+
+
+Assembly:
+
+When compiling, you start from a source text (ruby String, consisting
+mostly in a sequence of instructions/data/padding directive), which is parsed.
+
+The string is handed to a Preprocessor instance (which handles #if, #ifdef,
+#include, #define, /* */ etc, should be 100% compatible with gcc -E), which is
+encapsulated in an AsmPreprocessor for assembler sources (to handles asm macro
+definitions, 'equ' and asm ';' comments).
+The interface to do that is ExeFormat#parse(text[, filename, lineno]) or
+ExeFormat.assemble (which calls .new, #parse and #assemble).
+
+The (Asm)Preprocessor returns tokens to the ExeFormat, which parses them as Data,
+Padding, Labels or parser directives. Parser directives always start with a dot.
+They can be generic (.pad, .offset...) or ExeFormat-specific (.section,
+.import, .entrypoint...). They are handled by #parse_parser_instruction().
+If the ExeFormat does not recognize a word, it is handed to its CPU instance,
+which is responsible for parsing Instructions (or raise an exception).
+All those tokens are stored in one or more arrays in the @source attribute of
+the ExeFormat (Shellcode's @source is an Array, for PE/ELF it is a hash
+[section name] => [Array of parsed data])
+Every immediate value can be an arbitrary Expression (see later).
+
+You can then assemble the source to binary sections using ExeFormat#assemble.
+
+Once the section binaries are available, the whole binary executable can be
+written to disk using ExeFormat#encode_file(filename[, format]).
+
+PE and ELF include an autoimport feature that allows automatic creation of
+import-related data for known OS-specific functions (e.g. unresolved calls to
+'strcpy' will generate data so that the binary is linked against the libc
+library at runtime).
+
+The samples/{exe,pe,elf}encode.rb can take an asm source file as argument
+and compile it to a working executable.
+
+The CPU classes are responsible for parsing and encoding individual
+instructions. The current Ia32 parser uses the Intel syntax (e.g. mov eax, 42).
+The generic parser recognizes labels as a string at the beginning of a line
+followed by a colon (e.g. 'some_label:'). GCC-style local labels may be used
+(e.g. '1:', refered to using '1b' (backward) or '1f' (forward) ; may be
+redefined as many times as needed.)
+Data are specified using 'db'-style notation (e.g. 'dd 42h', 'db "blabla", 0')
+See samples/asmsyntax.rb
+
+
+EncodedData:
+
+In Metasm all binary data is stored as an EncodedData.
+EncodedData has 3 main attributes:
+ - #data which holds the raw binary data (generally a ruby String, but see
+VirtualString)
+ - #export which is a hash associating an export name (label name) to an offset
+within #data
+ - #reloc which is a hash whose keys are offsets within #data, and whose values
+are Relocation objects.
+A Relocation object has an endianness (:little/:big), a type (:u32 for unsigned
+32bits) and a target (the intended value stored here).
+The target is an arbitrary arithmetic/logic Expression.
+
+EncodedData also has a #virtsize (for e.g. .bss sections), and a #ptr (internal
+offset used when decoding things)
+
+You can fixup an EncodedData, with a Hash variable name => value (value should
+be an Expression or a numeric value). When you do that, each relocation's target
+is bound using the binding, and if the result is calculable (no external variable
+name used in the Expression), the result is encoded using the relocation's
+size/sign/endianness information. If it overflows (try to store 128 in an 8bit
+signed relocation), an EncodeError exception is raised. Use the :a32 type to
+allow silent overflow truncating.
+If the relocation's target is not numeric, the target is unchanged if you use 
+EncodedData#fixup, or it is replaced with the bound target with #fixup! .
+
+
+Disassembly:
+
+This code is found in the metasm/decode.rb source file, which defines the
+Disassembler class.
+
+The disassembler needs a decoded ExeFormat (to be able to say what data is at
+which virtual address) and an entrypoint (a virtual address or export name).
+It can then start to disassemble instructions. When it encounters an
+Opcode marked as :setip, it asks the CPU for the jump destination (an
+Expression that may involve register values, for e.g. jmp eax), and backtraces
+instructions until it finds the numeric value.
+
+On decoding, the Disassembler maintains a #decoded hash associating addresses
+(expressions/integer #normalize()d) to DecodedInstructions.
+
+The disassembly generates an InstructionBlock graph. Each block holds a list of
+DecodedInstruction, and pointers to the next/previous block (by address).
+
+The disassembler also traces data accesses by instructions, and stores Xrefs
+for them.
+The backtrace parameters can be tweaked, and the maximum depth to consider
+can be specifically changed for :r/:w backtraces (instruction memory xrefs)
+using #backtrace_maxblocks_data.
+When an Expression is backtracked, each walked block is marked so that loops
+are detected, and so that if a new code path is found to an existing block,
+backtraces can be resumed using this new path.
+
+The disassembler makes very few assumptions, and in particular does not
+suppose that functions will return ; they will only if the backtrace of the
+'ret' instructions is conclusive. This is quite powerful, but also implies
+that any error in the backtracking process can lead to a full stop ; and also
+means that the disassembler is quite slow.
+
+The special method #disassemble_fast can be used to work around this when the
+code is known to be well-formed (ie it assumes that all calls returns)
+
+When a subfunction is found, a special DecodedFunction is created, which holds
+a summary of the function's effects (like a DecodedInstruction on steroids).
+This allows the backtracker to 'step over' subfunctions, which greatly improves
+speed. The DecodedFunctions may be callback-based, to allow a very dynamic
+behaviour.
+External function calls create dedicated DecodedFunctions, which holds some
+API information (e.g. stack fixup information, basic parameter accesses...)
+This information may be derived from a C header parsed beforehand.
+If no C function prototype is available, a special 'default' entry is used,
+which assumes that the function has a standard ABI.
+
+Ia32 implements a specific :default entry, which handles automatic stack fixup
+resolution, by assuming that the last 'call' instruction returns. This may lead
+to unexpected results ; for maximum accuracy a C header holding information for
+all external functions is recommanded (see samples/factorize-headers-peimports
+for a script to generate such a header from a full Visual Studio installation
+and the target binary).
+
+Ia32 also implements a specific GetProcAddress/dlsym callback, that will
+yield the correct return value if the parameters can be backtraced.
+
+The scripts implementing a full disassembler are samples/disassemble{-gui}.rb
+See the comments for the GUI key bindings.
+
+
+ExeFormat manipulation:
+
+You can encode/decode an ExeFormat (ie decode sections, imports, headers etc)
+
+Constructor: ExeFormat.decode_file(str), ExeFormat.decode_file_header(str)
+Methods: ExeFormat#encode_file(filename), ExeFormat#encode_string
+
+PE and ELF files have a LoadedPE/LoadedELF counterpart, that is able to work
+with memory-mmaped versions of those formats (e.g. to debugging running
+processes)
+
+
+VirtualString:
+
+A VirtualString is a String-like object: you can read and may rewrite slices of
+it. It can be used as EncodedData#data, and thus allows virtualization
+of most Metasm algorithms.
+You cannot change a VirtualString length.
+Taking a slice of a VirtualString will return either a String (for small sizes)
+or another VirtualString (a 'window' into the other). You can force getting a
+small VirtualString using the #dup(offset, length) method.
+Any unimplemented method called on it is forwarded to a frozen String which is
+a full copy of the VirtualString (should be avoided if possible, the underlying
+string may be very big & slow to access).
+
+There are currently 3 VirtualStrings implemented:
+- VirtualFile, whichs loads a file by page-sized chunks on demand,
+- WindowsRemoteString, which maps another process' virtual memory (uses the
+windows debug api through WinDbgAPI)
+- LinuxRemoteString, which maps another process' virtual memory (need ptrace
+rights, memory reading is done using /proc/pid/mem)
+
+The Win/Lin version are quite powerful, and allow things like live process
+disassembly/patching easily (using LoadedPE/LoadedELF as ExeFormat)
+
+
+Debugging:
+
+Metasm includes a few interfaces to allow live debugging.
+The WinOS and LinOS classes offer access to the underlying OS processes (e.g.
+OS.current.find_process('foobar') will retrieve a running process with foobar
+in its filename ; then process.mem can be used to access its memory.)
+
+The Windows and Linux debugging APIs (x86 only) have a basic ruby interface
+(PTrace32, extended in samples/rubstop.rb ; and WinDBG, a simple mapping of the
+windows debugging API) ; those will be more worked on/integrated in the future.
+
+A linux console debugging interface is available in samples/lindebug.rb ; it
+uses a SoftICE-like look and feel.
+This interface can talk to a gdb-server through samples/gdbclient.rb ; use
+[udp:]<host:port> as target.
+
+The disassembler scripts allow live process interaction by using as target
+'live:<pid or part of filename>'.
+
+A generic debugging interface is available, it is defined in metasm/os/main.rb
+It may be accessed using the Metasm::OS.current.create_debugger('foo')
+
+It can be viewed in action using the GUI and 'open live' target.
+
+
+C Parser:
+
+Metasm includes a hand-written C Parser.
+It handles all the constructs i am aware of, except hex floats:
+ - static const L"bla"
+ - variable arguments
+ - incomplete types
+ - __attributes__(()), __declspec()
+ - #pragma once
+ - #pragma pack()
+ - C99 declarators - type bla = { [ 2 ... 14 ].toto = 28 };
+ - Nested functions
+ - __int8 etc native types
+ - Label addresses (&&label)
+Also note that all those things are parsed, but most of them will fail to
+compile on the Ia32 backend (the only one implemented so far.)
+
+When you parse a C String using C::Parser.parse(text), you receive a Parser
+object. It holds a #toplevel field, which is a C::Block, which holds #structs,
+#symbols and #statements. The top-level functions are found in the #symbol hash
+whose keys are the symbol names, associated to a C::Variable object holding
+the functions. The function parameter/attributes are accessible through
+func.type, and the code is in func.initializer, which is itself a C::Block.
+Under it you'll find a tree-like structure of C::Statements (If, While, Asm,
+CExpressions...)
+
+A C::Parser may be #precompiled to transform it into a simplified version that
+is easier to compile: typedefs are removed, control sequences are transformed
+in if () goto ; etc.
+
+To compile a C program, use PE/ELF.compile_c, that will create a C::Parser with
+exe-specific macros defined (eg __PE__ or __ELF__).
+
+The prefered way to create a C::Parser is to initialize it with a CPU and the
+desired ExeFormat, so that it is
+correctly initialized (eg type sizes: is long 4 or 8 bytes? etc) ; and
+may define preprocessor macros needed to correctly parse standard headers.
+Vendor-specific headers may need to use either #pragma prepare_visualstudio
+(to parse the Microsoft Visual Studio headers) or prepare_gcc (for gcc), the
+latter may be auto-detected (or may not).
+Vendor headers tested are VS2003 (incl. DDK) and gcc4 ; ymmv.
+
+Currently the CPU#compilation of a C code will generate an asm source (text),
+which may then be parsed & assembled to binary code.
+
+See ExeFormat#compile_c, and samples/exeencode.rb
+

data/TODO

@@ -0,0 +1,114 @@
+List of TODO items, by section, in random order
+
+Ia32
+ emu fpu
+ add all sse2 instrs
+ realmode
+
+X86_64
+ decompiler
+
+CPU
+ Sparc
+ Cell
+
+Parser
+ Allow single-file multiplexer (C code + Asm + asm16bit + ...)
+ Fix the asm prepro comment issue: '; a /* b\n c ; */' should see 'c'
+
+Assembler
+ Handle cpu pseudo-instrs (mips 'li' -> lui high + ori low)
+ SplitReloc? (for pseudo-instrs)
+ Ia32 GAS syntax
+ Make the autoimport depend on the target platform and not on the exeformat
+ Encode FPU constants
+ 
+Disasm
+ DecodedData
+ Exe decoding generate decodeddata ?
+ Function-local namespace (esp+12 -> esp+var_42)
+ Fix thunk detection (thunk: mov ecx, 42  jmp [iat_thiscall] is not a thunk)
+ Test with ET_REL style exe
+ Store stuff out of mem (to handle big binaries)
+ Better :default usage
+  good on call eax, but not on <600k instrs> ret
+  use binary personality ? (uses call vs uses pushret..)
+ Improve backtrace -> patch di.instr.args exprs
+ path-specific backtracking ( foo: call a ; a: jmp retloc ; bar: call b ; b: jmp retloc ; retloc: ret ; call foo ; ret : last ret trackback should only reach a:)
+ Decode pseudo/macro-instrs (mips 'li')
+ Deoptimizer (instr reordering for readability)
+ Optimizer (deobfuscating)
+ Per-instr context (allows to mix cell/ppc, x86 32/16bits, arm/armthumb..)
+
+Compiler
+ Optimizer
+ Register allocator
+ Instr reordering
+ Asm intrinsics
+ Asm inline
+ inline functions
+ Separate partial compilation + linking (src1.c -> obj1.o, src2.c -> obj2.o, obj1.o+obj2.o -> bin)
+ Make generic compiler from cpu.instr_binding ?
+  create a cpu.what_instr_has_binding(:a => (:a + :b)) => 'add a, b' ?
+ Shellcode compiler (exit() => mov eax, 1  int 80h inline)
+ 
+Decompiler
+ Fix decompiling on loaded savefile
+ Rewrite cpu-specific to really dumb
+  Just translate di.binding to C
+   maybe w/ trivial var dependency check for unused regs, but beware :incomplete instrs deps
+  Check interdependency ('xadd')
+ Move frame pointer checks / stack var detection to C code
+ Update asm listing from info in C (stack vars, stack var names..)
+ Handle renaming/retyping register vars / aliases
+ Handle switch() / computed goto
+ Fix inline asm reg dependencies
+ Handle direct syscalls (mov eax, 1  int 80h => exit())
+ Autodecode structs
+ FPU
+ Handle/hide compiler-generated stuff (getip, stack cookie setup/check..)
+ Handle call 1f ; 1: pop eax
+ More user control (force/forbid register arg, return type, etc)
+
+Debugger
+ OSX
+ Detour-style functionnality to patch binary code (also static to patch exe files?)
+ Move constants in a data/ folder (ptrace reg numbers, syscalls, etc)
+ Generic remote process manip
+  create blank state
+  linux virtualallocex
+  pax-compatible code patch through mmap
+ Remote debugging (small standalone C client)
+ Support dbghelp.dll (ms symbol server info)
+ Support debugee function call (gdb 'call')
+ Manipulate memory through C struct casts
+
+ExeFormat
+ Handle minor editing without decode/reencode (eg patch ELF entrypoint)
+
+ELF
+ test encoding openbsd binaries
+ handle symbol versions
+ LoadedELF.dump
+ Check relocation encoding (eg samples/dynamic_ruby with cpu.generate_PIC=false)
+
+MachO
+
+PE
+ resource editor ?
+ rc compiler ?
+ add simple accessor for resource stuff (manifest, icon, ...)
+
+GUI
+ debugger
+  specialize widgets
+   show breakpoints
+   show jump direction from current flag values
+ have a console frontend
+ better graph positionning fallback
+ zoom font when zooming graph
+ copy/paste, selection
+ map (part of) the binary & debug it (map a PE on a linux host & run it)
+
+Ruby
+ compile ruby AST to native optimized code

data/doc/code_organisation.txt

@@ -0,0 +1,146 @@
+Metasm source code organisation
+===============================
+
+The metasm source code takes advantage of the ruby language facilities,
+which allows splitting the definition of a single class in multiple files.
+
+Each file in the source tree holds code related to a particular feature of
+the framework.
+
+Directories
+-----------
+
+The top-level directories are :
+
+* `doc/`: this documentation
+* `metasm/`: the framework core
+* `samples/`: a set of sample scripts showing various functionnalities of the framework
+* `tests/`: a few unit tests (too few..)
+* `misc/`: misc ruby scripts, not directly related to metasm
+
+The core
+--------
+
+The `metasm/` directory holds most of the code of the framework, along with the
+main `metasm.rb` file in the top directory.
+
+The top-level `metasm.rb` has code to load parts of the framework source on demand
+in the ruby interpreter, which is implemented with ruby's <const_missing.txt>
+
+
+Executable formats
+##################
+
+The `exe_format/` subdirectory contains the implementations of the various
+binary file formats supported in the framework.
+
+Three files have a special meaning here:
+
+* `main.rb`: it defines the <core/ExeFormat.txt> class
+* `serialstruct.rb`: here you'll find the definitions of <core/SerialStruct.txt>
+* `autoexe.rb`: the implementation of <core/AutoExe.txt>, which allows the recognition of arbitrary files from their binary signature.
+
+The `main.rb` file is included in all other formats, as all file classes
+are subclasses of `ExeFormat`.
+
+The `serialstruct.rb` implements a helper class to ease the description of
+binary structures, and generate parsing/encoding functions for those.
+
+All other files implement a specific file format handler. The bigger files
+(`ELF` and `PE/COFF`) are split between the parsing/encoding functions and
+decoding/disassembly.
+
+
+CPUs
+####
+
+All supported architectures have a dedicated subdirectory, and a helper file
+that will simply include all the arch-specific files.
+
+All those files will contribute to add functions to the same class implementing
+the CPU interface. Not all CPUs implement all those features. They are:
+
+* `main.rb`: inner classes definitions (for registers etc), generic functions
+* `opcodes.rb`: initializes the opcode list for the architecture
+* `encode.rb`: methods to encode instructions
+* `decode.rb`: methods to decode/emulate instructions
+* `parse.rb`: methods to parse asm instructions from a source file
+* `render.rb`: methods to output an instruction to a string
+* `compile_c.rb`: the C compiler implementation
+* `decompile.rb`: the arch-specific part of the generic decompiler
+* `debug.rb`: arch-specific information used when debugging target of this architecture
+
+In some cases the files are small enough to be all merged into the `main.rb` file.
+
+
+Operating systems
+#################
+
+The `os/` subdirectory holds the code used to abstract an operating systems.
+
+The files here define an API allowing to enumerate running processes, and interact
+with them in various ways. The <core/Debugger.txt> class and subclasses are
+defined there.
+
+Those files also holds the list of known functions and in which system libraries
+they can be found (see <core/WindowsExports.txt> or <core/GNUExports.txt>), which
+are used when linking executable files.
+
+
+Graphical user-interface
+########################
+
+The `gui/` subdirectory contains the code needed by the metasm graphical user-interfaces.
+
+Currently those include the disassembler and the debugger (see the *samples* section).
+
+Those GUI elements are implemented using a custom GUI abstraction, and reside in the
+various `dasm_*.rb` and `debug.rb`.
+
+The actual implementation of the GUI are found in:
+
+* `win32.rb`: the native Win32 API backend
+* `gtk.rb`: a Gtk2 backend, intended for unix platforms
+* `qt.rb`: a Qt backend experiment
+
+Please note that the Qt backend does not work *at all*.
+
+The `gui.rb` file in the main directory is used to chose among the available GUI backend
+the most appropriate for the current session.
+
+
+Others
+######
+
+The other files directly in the `metasm/` directory are either support files
+(eg `encode.rb`, `parse.rb`) that hold generic functions to be used by
+specific cpu/exeformat instances, or implement arch-agnostic features.
+Those include:
+
+* `preprocessor.rb`: the C/asm preprocessor/lexer
+* `parse_c.rb`: this is the implementation of the C parser
+* `compile_c.rb`: this is a C precompiler, it generates a very simplified C from a standard source
+* `decompile.rb`: the generic decompiler code, it uses arch-specific functions defined in the arch folder
+* `dynldr.rb`: this module is used when interacting directly with the host operating system through <core/DynLdr.txt>
+
+
+The samples
+-----------
+
+The `samples/` directory contains a lot of small files that intend to be
+exemples of how to use the framework. It also holds experiments and
+work-in-progress for features that may later be integrated into the main
+framework.
+
+The comment at the beginning of the file should be clear about the purpose
+of the script, and the scripts are expected to be copy/pasted and tweaked
+for the specific task needed by the user (that's you).
+
+Some of those files however are full-featured applications:
+
+* `exeencode.rb`: a shellcode compiler, with its `peencode.rb`, `elfencode.rb`, `machoencode.rb` counterparts
+* `disassemble.rb`: a disassembler
+* `disassemble-gui.rb`: the graphical disassembler / debugger
+
+The `samples/dasm-plugins/` subdirectory holds various plugins for the disassembler.
+