metasm 1.0.1 → 1.0.2

data/README

@@ -21,6 +21,10 @@ Ready-to-use scripts can be found in the samples/ subdirectory, check the
 comments in the scripts headers. You can also try the --help argument if
 you're feeling lucky.
 
+For more information, check the doc/ subdirectory. The text files can be
+compiled to html using the misc/txt2html.rb script.
+
+
 
 Here is a short overview of the Metasm internals.
 
@@ -167,8 +171,8 @@ You can encode/decode an ExeFormat (ie decode sections, imports, headers etc)
 Constructor: ExeFormat.decode_file(str), ExeFormat.decode_file_header(str)
 Methods: ExeFormat#encode_file(filename), ExeFormat#encode_string
 
-PE and ELF files have a LoadedPE/LoadedELF counterpart, that is able to work
-with memory-mmaped versions of those formats (e.g. to debugging running
+PE and ELF files have a LoadedPE/LoadedELF counterpart, that are able to work
+with memory-mmaped versions of those formats (e.g. to debug running
 processes)
 
 
@@ -198,27 +202,31 @@ disassembly/patching easily (using LoadedPE/LoadedELF as ExeFormat)
 
 Debugging:
 
-Metasm includes a few interfaces to allow live debugging.
+Metasm includes a few interfaces to handle debugging.
 The WinOS and LinOS classes offer access to the underlying OS processes (e.g.
 OS.current.find_process('foobar') will retrieve a running process with foobar
 in its filename ; then process.mem can be used to access its memory.)
 
-The Windows and Linux debugging APIs (x86 only) have a basic ruby interface
-(PTrace32, extended in samples/rubstop.rb ; and WinDBG, a simple mapping of the
-windows debugging API) ; those will be more worked on/integrated in the future.
+The Windows and Linux low-level debugging APIs have a basic ruby interface
+(PTrace and WinAPI) ; which are used by the unified high-end Debugger class.
+Remote debugging is supported through the GDB server wire protocol.
 
-A linux console debugging interface is available in samples/lindebug.rb ; it
-uses a SoftICE-like look and feel.
-This interface can talk to a gdb-server through samples/gdbclient.rb ; use
-[udp:]<host:port> as target.
+High-level debuggers can be created with the following ruby line:
+Metasm::OS.current.create_debugger('foo')
 
-The disassembler scripts allow live process interaction by using as target
-'live:<pid or part of filename>'.
+Only one kind of host debugger class can exist at a time ; to debug multiple
+processes, attach to other processes using the existing class. This is due
+to the way the OS debugging API works on Windows and Linux.
 
-A generic debugging interface is available, it is defined in metasm/os/main.rb
-It may be accessed using the Metasm::OS.current.create_debugger('foo')
+The low-level backends are defined in the os/ subdirectory, the front-end is
+defined in debug.rb.
 
-It can be viewed in action using the GUI and 'open live' target.
+A linux console debugging interface is available in samples/lindebug.rb ; it
+uses a (simplified) SoftICE-like look and feel.
+It can talk to a gdb-server socket ; use a [udp:]<host:port> target.
+
+The disassembler-gui sample allow live process interaction when using as
+target 'live:<pid or part of program name>'.
 
 
 C Parser:
@@ -236,7 +244,11 @@ It handles all the constructs i am aware of, except hex floats:
  - __int8 etc native types
  - Label addresses (&&label)
 Also note that all those things are parsed, but most of them will fail to
-compile on the Ia32 backend (the only one implemented so far.)
+compile on the Ia32/X64 backend (the only one implemented so far.)
+
+Parsing C files should be done using an existing ExeFormat, with the
+parse_c_file method. This ensures that format-specific macros/ABI are correctly
+defined (ex: size of the 'long' type, ABI to pass parameters to functions, etc)
 
 When you parse a C String using C::Parser.parse(text), you receive a Parser
 object. It holds a #toplevel field, which is a C::Block, which holds #structs,
@@ -249,15 +261,11 @@ CExpressions...)
 
 A C::Parser may be #precompiled to transform it into a simplified version that
 is easier to compile: typedefs are removed, control sequences are transformed
-in if () goto ; etc.
+into 'if (XX) goto YY;' etc.
 
 To compile a C program, use PE/ELF.compile_c, that will create a C::Parser with
 exe-specific macros defined (eg __PE__ or __ELF__).
 
-The prefered way to create a C::Parser is to initialize it with a CPU and the
-desired ExeFormat, so that it is
-correctly initialized (eg type sizes: is long 4 or 8 bytes? etc) ; and
-may define preprocessor macros needed to correctly parse standard headers.
 Vendor-specific headers may need to use either #pragma prepare_visualstudio
 (to parse the Microsoft Visual Studio headers) or prepare_gcc (for gcc), the
 latter may be auto-detected (or may not).

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+
+require "rake/testtask"
+
+Rake::TestTask.new do |t|
+  t.test_files = FileList['tests/*.rb']
+end
+
+task default: :test
+

data/TODO

@@ -1,14 +1,13 @@
 List of TODO items, by section, in random order
 
 Ia32
- emu fpu
- add all sse2 instrs
  realmode
 
 X86_64
  decompiler
 
 CPU
+ Arm
  Sparc
  Cell
 
@@ -26,19 +25,20 @@ Assembler
 Disasm
  DecodedData
  Exe decoding generate decodeddata ?
- Function-local namespace (esp+12 -> esp+var_42)
  Fix thunk detection (thunk: mov ecx, 42  jmp [iat_thiscall] is not a thunk)
  Test with ET_REL style exe
  Store stuff out of mem (to handle big binaries)
  Better :default usage
   good on call eax, but not on <600k instrs> ret
   use binary personality ? (uses call vs uses pushret..)
- Improve backtrace -> patch di.instr.args exprs
+ Improve 'backtrace => patch di.instr.args'
  path-specific backtracking ( foo: call a ; a: jmp retloc ; bar: call b ; b: jmp retloc ; retloc: ret ; call foo ; ret : last ret trackback should only reach a:)
  Decode pseudo/macro-instrs (mips 'li')
  Deoptimizer (instr reordering for readability)
  Optimizer (deobfuscating)
  Per-instr context (allows to mix cell/ppc, x86 32/16bits, arm/armthumb..)
+ Better save/load dasm state
+ Parse symbol.map generated by IDA for ELF files
 
 Compiler
  Optimizer
@@ -69,6 +69,7 @@ Decompiler
  Handle/hide compiler-generated stuff (getip, stack cookie setup/check..)
  Handle call 1f ; 1: pop eax
  More user control (force/forbid register arg, return type, etc)
+ Preserve C decompiled line association to range of asm decoded addrs
 
 Debugger
  OSX
@@ -77,14 +78,9 @@ Debugger
  Generic remote process manip
   create blank state
   linux virtualallocex
-  pax-compatible code patch through mmap
  Remote debugging (small standalone C client)
  Support dbghelp.dll (ms symbol server info)
  Support debugee function call (gdb 'call')
- Manipulate memory through C struct casts
-
-ExeFormat
- Handle minor editing without decode/reencode (eg patch ELF entrypoint)
 
 ELF
  test encoding openbsd binaries
@@ -98,6 +94,7 @@ PE
  resource editor ?
  rc compiler ?
  add simple accessor for resource stuff (manifest, icon, ...)
+ parse PDB
 
 GUI
  debugger
@@ -105,10 +102,11 @@ GUI
    show breakpoints
    show jump direction from current flag values
  have a console frontend
- better graph positionning fallback
  zoom font when zooming graph
- copy/paste, selection
+ text selection
+ copy/paste
  map (part of) the binary & debug it (map a PE on a linux host & run it)
+ html frontend
 
 Ruby
- compile ruby AST to native optimized code
+ write a fast ruby-like interpreter

data/doc/code_organisation.txt

@@ -54,6 +54,8 @@ decoding/disassembly.
 CPUs
 ####
 
+The cpu-specific code is stored inside the `cpu/` subdirectory.
+
 All supported architectures have a dedicated subdirectory, and a helper file
 that will simply include all the arch-specific files.
 

data/doc/core/DynLdr.txt

@@ -0,0 +1,247 @@
+DynLdr
+======
+
+DynLdr is a class that uses metasm to dynamically add native methods,
+or native method wrappers, available to the running ruby interpreter.
+
+It leverages the built-in C parser / compiler.
+
+It is implemented in `metasm/dynldr.rb`.
+
+Currently only supported for <core/Ia32.txt> and <core/X86_64.txt> under
+Windows and Linux.
+
+
+Basics
+------
+
+Native library wrapper
+######################
+
+The main usage is to generate interfaces to native libraries.
+
+This is done through the `#new_api_c` method.
+
+The following exemple will read the specified C header fragment,
+define ruby constants for all `#define`/`enum`, and define ruby
+method wrappers to call the native functions whose prototype is
+present in the header.
+
+All referenced native functions must be exported by the given
+library file.
+
+  class MyInterface < DynLdr
+    c_header = <<EOS
+  #define SomeConst 42
+  enum { V1, V2 };
+  
+  __stdcall int methodist(char*, int);
+  EOS
+
+    new_api_c c_header, 'mylib.dll'
+  end
+
+Then you can call, from the ruby:
+
+  MyInterface.methodist("lol", MyInterface::SOMECONST)
+
+Constant/enum names are converted to full uppercase, and method
+names are converted to full lowercase.
+
+Dynamic native inline function
+##############################
+
+You can also dynamically compile native functions, that are compiled
+in memory and copied to RWX memory with the right ruby wrapper:
+
+  class MyInterface < DynLdr
+    new_func_c <<EOS
+  int bla(char*arg) {
+    if (strlen(arg) > 4)
+       return 1;
+    else
+       return 0;
+  }
+  EOS
+  end
+
+References to external functions are allowed, and resolved automatically.
+
+The ruby objects used as arguments to the wrapper method are
+automatically converted to the right C type.
+
+
+You can also write native functions in assembly, but you must specify a
+C prototype, used for argument and return value conversion.
+
+  class MyInterface < DynLdr
+    new_func_asm "int increment(int i);", <<EOS
+   mov eax, [esp+4]
+   inc eax
+   ret
+  EOS
+  
+  p increment(4)
+
+  end
+
+
+Structures
+----------
+
+`DynLdr` handles C structures.
+
+Once a structure is specified in the C part, you can create a ruby object
+using `MyClass.alloc_c_struct(structname)`, which will allocate an object of the
+right size to hold all the structure members, and with the right accessors.
+
+To access/modify struct members, you can either use a `Hash`-style access
+
+  structobj['membername'] = 42
+
+or `Struct`-style access
+
+  structobj.membername = 42
+
+Member names are matched case-insensitively, and nested structures/unions
+are also searched.
+
+The struct members can be initially populated by passing a `Hash` argument
+to the `alloc_c_struct` constructor. Additionally, this hash may use the
+special value `:size` to reference the byte size of the current structure.
+
+  class MyInterface < DynLdr
+    new_api_c <<EOS
+  struct sname {
+    int s_mysize;
+    int s_value;
+    union {
+      struct {
+        int s_bits:4;
+        int s_bits2:4;
+      };
+      int s_union;
+    }
+  };
+  EOS
+  end
+
+  # field s_mysize holds the size of the structure in bytes, ie 12
+  s_obj = MyInterface.alloc_c_struct('sname', :s_mysize => :size, :s_value => 42)
+
+  # we can access fields using Hash-style access
+  s_obj['s_UniOn'] = 0xa8
+
+  # or Struct-style access
+  puts '0x%x' % s_obj.s_BiTS2     #  =>  '0xa'
+
+This object can be directly passed as argument to a wrapped function, and
+the native function will receive a pointer to this structure (that it can
+freely modify).
+
+This object is a `C::AllocStruct`, defined in `metasm/parse_c.rb`.
+Internally, it is based on a ruby `String`, and has a reference to the parser's
+`Struct` to find the mapping membername -> offsets/length.
+
+See <core/CParser.txt> for more details.
+
+
+Callbacks
+---------
+
+`DynLdr` handles C callbacks, with arbitrary ABI.
+
+Any number of callbacks can be defined at any time.
+
+C callbacks are backed by a ruby `Proc`, eg `lambda {}`.
+
+
+  class MyInterface < DynLdr
+    new_api_c <<EOS
+  void qsort(void *, int, int, int(*)(void*, void*));
+  EOS
+
+    str = "sanotheusnaonetuh"
+    cmp = lambda { |p1, p2|
+      memory_read(p1, 1) <=> memory_read(p2, 1)
+    }
+    qsort(str, str.length, 1, cmp)
+    p str
+  end
+
+
+
+Argument conversion
+-------------------
+
+Ruby objects passed to a wrapper method are converted to the corresponding
+C type
+
+* `Strings` are converted to a C pointer to the byte buffer (also directly
+accessible from the ruby through `DynLdr.str_ptr(obj)`
+* `Integers` are converted to their C equivalent, according to the prototype
+(`char`, `unsigned long long`, ...)
+* `Procs` are converted to a C callback
+* `Floats` are not supported for now.
+
+
+Working with memory
+-------------------
+
+DynLdr provides different ways to allocate memory.
+
+* `alloc_c_struct` to allocate a C structure
+* `alloc_c_ary` to allocate C array of some type
+* `alloc_c_ptr`, which is just an ary of size 1
+* `memory_alloc` allocates memory from a new memory page
+
+`memory_alloc` works by calling `mmap` under linux and `VirtualAlloc` under windows,
+and is suitable for allocating memory where you want to control
+the memory permissions (read, write, execute). This is done through `memory_perm`.
+
+`memory_perm` takes for argument the start address, the length, and the new permission, specified as a String (e.g. 'r', 'rwx')
+
+To work with memory that may be returned by an API (e.g. `malloc`),
+DynLdr provides ways to read and write arbitrary pointers from the ruby
+interpreter memory.
+Take care, those may generate faults when called with invalid addresses that
+will crash the ruby interpreter.
+
+* `memory_read` takes a pointer and a length, and returns a String
+* `memory_read_int` takes a pointer, and returns an Integer (of pointer size,
+e.g. 64 bit in a 64-bit interpreter)
+* `memory_write` takes a pointer and a String, and writes it to memory
+* `memory_write_int`
+
+
+Hacking
+-------
+
+Internally, DynLdr relies on a number of features that are not directly
+available from the ruby interpreter.
+
+So the first thing done by the script is to generate a binary native module
+that will act as a C extension to the ruby interpreter.
+This binary is necessarily different depending on the interpreter.
+The binary name includes the target architecture, in the format
+dynldr-*arch*-*cpu*-*19*.so, e.g.
+
+* dynldr-linux-ia32.so
+* dynldr-windows-x64-19.so
+
+This native module is (re)generated if it does not exist, or is older than the
+`dynldr.rb` script.
+
+A special trick is used in this module, as it does not know the actual name
+of the ruby library used by the interpreter. So on linux, the `libruby` is
+removed from the `DT_NEEDED` library list, and on windows a special stub
+is assembled to manually resolve the ruby imports needed by the module from
+any instance of `libruby` present in the running process.
+
+The native file is written to a directory writeably by the current user.
+The following list of directories are tried, until a suitable one is found:
+
+* the `metasm` directory itself
+* the `$HOME`/`$APPDATA`/`$USERPROFILE` directory
+* the `$TMP`/`$TEMP`/current directory
+

data/doc/core/ExeFormat.txt

@@ -0,0 +1,43 @@
+ExeFormat
+=========
+
+This class is the parent of all executable format handlers.
+
+It is defined in `metasm/exe_format/main.rb`.
+
+It defines some standard shortcut functions, such as:
+
+* `Exe.decode_file(filename)`
+* `Exe.assemble(cpu,asm_source)`
+* `Exe.compile_c(cpu,c_source)`
+* `Exe#encode_file(filename)`
+
+These methods will instanciate a new Exe, and call the corresponding
+methods, *e.g.* `load` with the file content, and `decode`.
+
+The handling of the different structures in the binary format should be
+done using the <core/SerialStruct.txt> facility.
+
+The subclasses are expected to implement various functions, depending on the
+usage (refer to the ELF and COFF implementations for more details):
+
+File decoding/disassembly
+-------------------------
+
+* `#decode_header`: parse the raw data in `#encoded` only to parse the file header
+* `#decode`: parse all the raw data in `#encoded`
+* `#cpu_from_headers`: return a <core/CPU.txt> instance according to the exe header information
+* `#get_default_entrypoints`: the list of entrypoints (exported functions, etc)
+* `#dump_section_header`: return a string that may be assembled to recreate the specified section
+* `#section_info`: return a list of generic section informations for the disassembler
+
+
+File encoding/source parsing
+----------------------------
+
+* `#tune_prepro`: define exe-specific macros for the preprocessor (optional)
+* `#parse_init`: initialize the `@cursource` array to receive the parsed asm source
+* `#parse_parser_instruction`: parse exe-specific instructions, eg `.text`, `.import`...
+* `#assemble`: assemble the content of the @cursource into binary section contents
+* `#encode`: assemble the various sections and a binary header into `@encoded`
+