metasm 1.0.0 → 1.0.5

data/doc/core/ExeFormat.txt

@@ -0,0 +1,43 @@
+ExeFormat
+=========
+
+This class is the parent of all executable format handlers.
+
+It is defined in `metasm/exe_format/main.rb`.
+
+It defines some standard shortcut functions, such as:
+
+* `Exe.decode_file(filename)`
+* `Exe.assemble(cpu,asm_source)`
+* `Exe.compile_c(cpu,c_source)`
+* `Exe#encode_file(filename)`
+
+These methods will instanciate a new Exe, and call the corresponding
+methods, *e.g.* `load` with the file content, and `decode`.
+
+The handling of the different structures in the binary format should be
+done using the <core/SerialStruct.txt> facility.
+
+The subclasses are expected to implement various functions, depending on the
+usage (refer to the ELF and COFF implementations for more details):
+
+File decoding/disassembly
+-------------------------
+
+* `#decode_header`: parse the raw data in `#encoded` only to parse the file header
+* `#decode`: parse all the raw data in `#encoded`
+* `#cpu_from_headers`: return a <core/CPU.txt> instance according to the exe header information
+* `#get_default_entrypoints`: the list of entrypoints (exported functions, etc)
+* `#dump_section_header`: return a string that may be assembled to recreate the specified section
+* `#section_info`: return a list of generic section informations for the disassembler
+
+
+File encoding/source parsing
+----------------------------
+
+* `#tune_prepro`: define exe-specific macros for the preprocessor (optional)
+* `#parse_init`: initialize the `@cursource` array to receive the parsed asm source
+* `#parse_parser_instruction`: parse exe-specific instructions, eg `.text`, `.import`...
+* `#assemble`: assemble the content of the @cursource into binary section contents
+* `#encode`: assemble the various sections and a binary header into `@encoded`
+

data/doc/core/Expression.txt

@@ -0,0 +1,220 @@
+Expression
+==========
+
+Metasm uses this class to represent arbitrary symbolic arithmetic expressions, e.g.
+* `42`
+* `eax + 12`
+* `loc_4228h + 4*ebx - 12`
+
+These expressions can include `Integers`, `Symbols`, and `Strings`.
+
+The symbols and strings represent arbitrary variables, with the convention that
+strings represent fixed quantities (eg addresses, labels), whereas symbols
+represent more variable stuff (eg register values).
+
+There is also a special symbol that may be used, `:unknown`, to represent a
+value that is known to be unknown. See the `reduce` section.
+
+See also <core/Indirection.txt>.
+
+The Expression class holds all methods relative to Integer binary manipulation,
+that is `encoding` and `decoding` from/to a binary blob (see also
+<core/EncodedData.txt>)
+
+
+Members
+-------
+
+Expressions hold exactly 3 members:
+* `lexpr`, the left-hand side of the expression
+* `rexpr`, the right-hand side
+* `op`, the operator
+
+`lexpr` and `rexpr` can be any value, most often String, Symbol, Integer or
+Expression. For unary operators, `lexpr` is `nil`.
+
+`op` is a Symbol representing the operation.
+It should be from the list:
+* arithmetic: `+ - / * >> << & | ^`
+* boolean: `|| && == != > >= < <=`
+* unary: `+ - ~ !`
+
+
+Instantiation
+-------------
+
+In ruby code, use the class method `[]`. It takes 1 to 3 arguments, `lexpr`,
+`op`, and `rexpr`. `lexpr` defaults to `nil`, and `op` defaults to `:+` (except
+for negative numeric values, which is stored with `op` == `:-` and `rexpr` ==
+abs).
+
+If `lexpr` or `rexpr` are an `Array`, the `[]` constructor is called
+recursively, to ease the definition of nested Expressions.
+
+Exemples:
+
+  Expression[42]
+  Expression[:eax, :+, 12]
+  Expression[:-, 'my_var']
+  Expression[[:eax, :-, 4], :*, [:ebx, :+, 0x12]]
+
+The Expression class also includes a parser, to allow creating an expression
+from a string. `parse_string!` will create an Expression and update its
+argument to point after the last part read successfully into the expr.
+The parser handles standard C operator precedence.
+
+  str = "1 + var"
+  Expression.parse_string!(str)    # => Expression[1, :+, "var"]
+  str = "42 bla"
+  Expression.parse_string!(str)    # => Expression[42]
+  str                             # => "bla"
+
+Use `parse_string` without the ! to parse the string without updating it.
+
+External variables
+------------------
+
+The `externals` method will return all non-integer members of the Expression.
+
+  Expression[[:eax, :+, 42], :-, "bla"].externals  # => [:eax, "bla"]
+
+
+Pattern matching
+----------------
+
+The `match` method allows to check an Expression against a pattern without
+having to check individual members. The pattern should be an Expression,
+whose variable members should be Strings or Symbols, which are also passed as
+arguments to the match function. On successful match, the correspondance
+between variable patterns and their actual value matched is returned as a Hash.
+
+  Expression[1, :+, 2].match(Expression['var', :+, 2], 'var')
+  # => { 'var' => 1 }
+  Expression[1, :+, 2].match(Expression['var', :+, 'var'], 'var')
+  # => nil
+  Expression[1, :+, 1].match(Expression['var', :op, 'var'], 'var', :op)
+  # => { 'var' => 1, :op => :+ }
+  
+
+Reduction
+---------
+
+Metasm Expressions include a basic symbolic computation engine, that allows
+some simple transformations of the Expression. The reduction will also
+compute numerical values whenever possible. If the final result is fully
+numeric, an Integer is returned, otherwise a new Expression is returned.
+
+In this context, the special value `:unknown` has a particular meaning.
+
+  Expression[1, :+, 2].reduce
+  # => 3
+  Expression[:eax, :+, [:ebx, :-, :eax]].reduce
+  # => Expression[:ebx]
+  Expression[1, :+, [:eax, :+, 2]].reduce
+  # => Expression[:eax, :+, 3]
+  Expression[:unknown, :+, :eax].reduce
+  # => Expression[:unknown]
+
+The symbolic engine operates mostly on addition/substractions, and
+no-operations (eg shift by 0). It also handles some boolean composition.
+
+The detail can be found in the #replace_rec method body, in `metasm/main.rb`.
+
+The reduce method can also take a block argument, which will be called at
+every step in the recursive reduction, for custom operations. If the block
+returns nil, the result is unchanged, otherwise the new value is used as
+replacement. For exemple, if you operate on 32-bit values and want to get rid
+of `bla & 0xffffffff`, use
+
+  some_expr.reduce { |e|
+    if e.kind_of?(Expression) and e.op == :& and e.rexpr == 0xffff_ffff
+      e.lexpr
+    end
+  }
+   
+
+Binding
+-------
+
+An expression involving variable externals can be bound using a Hash. This will
+replace any occurence of a key of the Hash by its value in the expression
+members. The `bind` method will return a new Expression with the substitutions,
+and the `bind!` method will update the Expression in-place.
+
+  Expression['val', :+, 'stuff'].bind('val' => 4, 'stuff' => 8).reduce
+  # => 12
+  Expression[:eax, :+, :ebx].bind(:ebx => 42)
+  # Expression[:eax, :+, 42]
+  Expression[:eax, :+, :ebx].bind(:ebx => :ecx)
+  # Expression[:eax, :+, :ecx]
+
+You can use Expressions as keys, but they will only be used on perfect matches.
+
+
+Binary packing
+--------------
+
+Encoding
+########
+
+The `encode` method will generate an EncodedData holding the expression, either
+as binary if it can reduce to an integral value, or as a relocation.
+The arguments are the relocation type and the endianness, plus an optional
+backtrace (to notify the user where an overflowing relocation comes from).
+
+The `encode_imm` class method will generate a raw String for a given
+integral value, a type and an endianness.
+The type can be given as a byte size.
+
+  Expression.encode_imm(42, :u8, :little)  # => "*"
+  Expression.encode_imm(42, 1, :big)       # => "*"
+  Expression.encode_imm(256, :u8, :little) # raise EncodeError
+
+On overflows (value cannot be encoded in the bit field) an EncodeError
+exception is raised.
+
+Decoding
+########
+
+The `decode_imm` class method can be used to read a binary value into an
+Integer, with an optional offset into the binary string.
+
+  Expression.decode_imm("*", :u8, :little)               # => 42
+  Expression.decode_imm("bla\xfe\xff", :i16, :little, 3) # => -2
+
+
+Arithmetic coercion
+-------------------
+
+Expression implement the `:+` and `:-` ruby methods, so that `expr + 4`
+works as expected. The result is reduced.
+
+
+Integer methods
+---------------
+
+The Expression class offers a few methods to work with integers.
+
+make_signed
+###########
+
+`make_signed` will convert a raw unsigned to its equivalent signed value,
+given a bit size.
+
+  Expression.make_signed(1, 16)      # => 1
+  Expression.make_signed(0xffff, 16) # => -1
+
+
+in_range?
+#########
+
+`in_range?` can check if a given numeric value would fit in a particular
+<core/Relocation.txt> field. The method can return true or false if it
+fits or not, or `nil` if the result is unknown (eg the expr has no numeric
+value).
+
+  Expression.in_range?(42, :i8)                 # => true
+  Expression.in_range?(128, :i8)                # => false
+  Expression.in_range?(-128, :i8)               # => true
+  Expression.in_range?(Expression['bla'], :u32) # => nil
+

data/doc/core/GNUExports.txt

@@ -0,0 +1,27 @@
+GNUExports
+==========
+
+This class is defined in `metasm/os/gnu_exports.rb`
+
+It defines an `EXPORT` constant, a Hash, whose keys
+are the standard linux API symbol names, and values
+are the library name where you can find this symbol.
+
+The equivallent for windows is <core/WindowsExports.txt>
+
+Usage
+-----
+
+The main usage of this class is the automatic generation
+of the <core/ELF.txt> dynamic tag `DT_NEEDED` from the
+external symbols referenced by a binary during compilation.
+
+This is done in the `automagic_symbols` method.
+
+Symbols
+-------
+
+The current version holds the symbols of the debian
+glibc, from `libc.so.6` and `libdl.so.2`.
+
+Ruby symbols are also defined, from `libruby1.8.so.1.8`.

data/doc/core/Ia32.txt

@@ -0,0 +1,236 @@
+Ia32
+====
+
+The Ia32 architecture, aka *Intel_x86*, is the most advanced among the
+architectures implemented in the framework. It is a subclass of the
+generic <core/CPU.txt>.
+
+It can handle binary code for the 16 and 32bits modes of the processor.
+
+It is a superclass for the <core/X86_64.txt> object, a distinct processor
+that handles 64-bit *long_mode* (aka *x64*, *amd64*, *em64t*)
+
+The CPU `shortname` is `ia32` (`ia32_16` in 16-bit mode, and a `_be` suffix
+if bigendian)
+
+Opcodes
+-------
+
+The opcodes list can be customized to match that available on a specific
+version of the processor. The possibilities are:
+
+* 386_common
+* 386
+* 387
+* 486
+* pentium
+* p6
+* 3dnow
+* sse
+* sse2
+* sse3
+* vmx
+* sse42
+
+Most opcodes are available in the framework, with the notable exception of:
+
+* most sse2 simd instructions
+* the AVX instructions
+* amd-specific instructions
+
+The `386_common` family is the subset of 386 instruction that are most
+commonly found in standard usermode programs (no `in`/`out`/bcd
+arithmetic/far call/etc).
+This can be useful when manipulating stuff that in not known to be i386
+binary code.
+
+
+Initialization
+--------------
+
+An Ia32 <core/CPU.txt> object can be created using the following code:
+
+  Metasm::Ia32.new
+
+The `X86` alias may be used in place of `Ia32`.
+
+The constructor accepts optional arguments to specify the CPU size, the
+opcode family, and the endianness of the processor. The arguments can
+be given in any order. For exemple,
+
+  Metasm::Ia32.new(16, 'pentium', :big)
+
+will create a 16-bit mode cpu, with opcodes up to the 'pentium' CPU family,
+in big-endian mode.
+
+The Ia32 initializer has the convenience feature that it will create an
+X86_64 instance when given the 64 bit size (e.g. `Ia32.new(64)` returns an
+X86_64 instance)
+
+
+Assembler
+---------
+
+The parser handles only Intel-style asm syntax, *e.g.*
+
+  some_label:
+    mov eax, 10h
+    mov ecx, fs:[eax+16]
+    push dword ptr fs:[1Ch]
+    call ecx
+    test al, al
+    jnz some_label
+    ret
+    fmulp ST(4)
+
+
+Instruction arguments
+#####################
+
+The parser recognizes standard registers, such as
+
+* `eax`
+* `ah`
+* `mm4` (mmx 64bit register)
+* `xmm2` (xmm 128bit register)
+* `ST` (current top of the FPU stack)
+* `ST(3)` (FPU reg nr.3)
+* `cs` (segment register)
+* `dr3` (debug register)
+* `cr2` (control register)
+
+It also supports inexistant registers, such as
+
+* `cr7`
+* `dr4`
+* `segr6` (segment register nr.6)
+
+The indirections are called `ModRM`. They take the form:
+
+* `[eax]` (memory pointed by `eax`)
+* `byte ptr [eax]` (1-byte memory pointed by `eax`)
+* `byte [eax]` (same as previous)
+* `fs:[eax]` (offset `eax` from the base of the `fs` segment)
+* `[fs:eax]` (same as previous)
+
+The pointer itself can be:
+
+* `[eax]` (any register)
+* `[eax+12]` (base + numeric offset)
+* `[eax+ebx]` (base + register index)
+* `[eax + 4*ebx]` (base + 1,2,4 or 8 * index)
+* `[eax + 2*ebx + 42]` (both)
+
+Note that the form base + s*index cannot use `esp` as index with s != 1.
+
+For indirection sizes, the size is taken from the size of other arguments
+if it is not specified (eg `mov eax, [42]` will be 4 bytes, and `mov al, [42]`
+will be 1). The explicit size specifier can be:
+
+* `byte` (8bits)
+* `word` (16)
+* `dword` (32)
+* `qword` (64)
+* `oword` (128)
+* `_12bits` (12, arbitrary numbers can be used)
+
+
+Parser commands
+###############
+
+The following commands are recognized in an asm source:
+
+* `.mode`
+* `.bits`
+
+They are synonymous, and serve to change the mode of the processor to either
+16 or 32bits.
+
+They should be the first instruction in the source, changing the mode during
+parsing is not supported. This would change only the mode for the next
+instructions to be parsed, and for all instructions (incl. those already parsed
+at this point) when encoding, which is likely **not** what you want. See the
+`codeXX` prefixes.
+
+Note that changing the CPU size once it was created may have bad side-effects.
+For exemple, some preprocessor macros may already have been generated according
+to the original size of the CPU and will be incorrect from this point on.
+
+
+Prefixes
+########
+
+The following prefixes are handled:
+
+* `lock`
+* `rep`, `repz`, `repnz`, `repe`, `repne`
+* `code16`, `code32`
+* `hintjmp`, `hintnojmp` (aliases: `ht`, `hnt`)
+* `seg_cs` ... `seg_gs`
+
+The `repXX` prefixes are for string operations (`movsd` etc), but will be set
+for any opcode. Only the last of the family will be encoded.
+
+The `code16` will generate instructions to be run on a CPU in 16bit mode,
+independantly of the global CPU mode. For exemple,
+
+  code16 mov ax, 42h
+
+will generate `"\xb8\x42\x00"` (no opsz override prefix), and will decode or
+run incorrectly on an 32bit CPU.
+
+The `hintjmp` prefix is useful for conditional jumps to give a hint to the
+CPU branch predictor as to whether the branch is take or not.
+
+The `seg_cs` prefix family is used to declare arbitrary segment override.
+These should be used only in instructions with no ModRM argument.
+
+
+Suffixes
+########
+
+The parser implements a specific feature to allow the differenciation of
+otherwise ambiguous opcodes, in the form of instruction suffixes.
+
+By default, the assembler will generate the shortest encoding for a given
+instruction. To force encoding of another form you can add a specific
+suffix to the instruction. In general, metasm will use e.g. register sizes
+when possible to avoid this kind of situations, but with immediate-only
+displacement this is necessary.
+
+  or.a16 [1234h], eax  ; use a 16-bit address
+  or [bx], eax         ; use a 16-bit address (implicit from the bx register)
+  or eax, 1	; "\x83\xc8\x01"
+  or.i8 eax, 1	; "\x83\xc8\x01" (same, shortest encoding)
+  or.i eax, 1   ; "\x81\xc8\x01\x00\x00\x00" (constant stored in a 32bit field)
+  movsd.a16	; use a 16-byte address-size override prefix (copy dword [si] to [di])
+  push.i16 42h  ; push a 16-bit integer
+
+The suffixes are available as follow:
+
+* if the opcode takes an integer argument that can be encoded as either a 8bits or <cpu size>bits, the `.i` and `.i8` variants are created
+* if the opcode takes a memory indirection as argument, or is a string operation (`movsd`, `scasb`, etc) the `.a16` and `.a32` variants are created
+* if the opcode takes a single integer argument, a far pointer, or is a return instruction, the `.i16` and `.i32` variants are created
+
+
+C parser
+--------
+
+The Ia32 C parser will initialize the type sizes with the `ilp32` memory
+model, which is:
+
+* short = 16bits
+* int = 32bits
+* long = 32bits
+* long long = 64bits
+* pointer = 32bits
+
+In 16bit mode, the model is `ilp16`, which may not be correct (the 16bits
+compiler has not been tested anyway).
+
+The following macros are defined (in the asm preprocessor too)
+
+* `_M_IX86` = 500
+* `_X86_`
+* `__i386__`
+