smallworld-re 1.0.0__tar.gz

smallworld_re-1.0.0/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+© 2023 Massachusetts Institute of Technology
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

smallworld_re-1.0.0/PKG-INFO

@@ -0,0 +1,189 @@
+Metadata-Version: 2.2
+Name: smallworld-re
+Version: 1.0.0
+Summary: An emulation stack tracking library
+Home-page: https://github.com/smallworld-re/smallworld
+Author: MIT Lincoln Laboratory
+Author-email: smallworld@ll.mit.edu
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: unicorn
+Requires-Dist: angr
+Requires-Dist: capstone
+Requires-Dist: lief
+Requires-Dist: pyhidra
+Requires-Dist: pypcode
+Provides-Extra: development
+Requires-Dist: black; extra == "development"
+Requires-Dist: isort; extra == "development"
+Requires-Dist: flake8; extra == "development"
+Requires-Dist: mypy; extra == "development"
+Requires-Dist: pip-tools; extra == "development"
+Requires-Dist: pre-commit; extra == "development"
+Requires-Dist: sphinx; extra == "development"
+Requires-Dist: sphinxcontrib-programoutput; extra == "development"
+Dynamic: author
+Dynamic: author-email
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SmallWorld
+
+[![commit-style-image]][conventional]
+[![code-style-image]][black]
+[![license-image]][mit]
+
+Easier harnessing of code for analysis!
+
+## Description
+
+SmallWorld is an environment for streamlined harnessing of binary code for the
+purpose of dynamic analysis. If you have code that you got from somewhere and
+you'd like to run it and analyze those runs to understand what that code does
+or if it has bugs, then you should try SmallWorld!!
+
+There are two fundamental tenets behind SmallWorld
+* Harnessing should be easier
+* Analysis can accelerate harnessing
+
+The first of these tenets we hope to support with good software APIs. As a very
+simple example, consider the harnessing script
+[stack.py](https://github.com/smallworld-re/smallworld/blob/main/tests/square.py),
+composed using SmallWorld, in which registers are initialized and a stack is
+arranged for running the code in
+[stack.s](https://github.com/smallworld-re/smallworld/blob/main/tests/stack.s).
+For a more sophisticated example of SmallWorld's harnessing facitilites,
+consider the code snippet
+[struct.s](https://github.com/smallworld-re/smallworld/blob/main/tests/struct.s),
+which assumes a stack and input pointers to a linked list with very specific
+format. The harnessing script in this case is more complicated, including
+specifying type information for the linked list element structures as well as
+use of a simple allocator abstraction provided by SmallWorld to instantiate
+nodes and link them together appropriately:
+[struct.py](https://github.com/smallworld-re/smallworld/blob/main/tests/struct.py).
+
+The second tenet we address with purpose-built analyses which leverage a
+(possibly incomplete) harness script and that use techniques such as [Micro
+Execution](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/microx.pdf)
+and [Symbolic
+Execution](https://en.wikipedia.org/wiki/Symbolic_execution#:~:text=In%20computer%20science%2C%20symbolic%20execution,of%20a%20program%20to%20execute)
+to provide hints that can guide improving a harness. 
+
+This harness is the final output of SmallWorld and might be used in fuzzing or
+dynamic reverse engineering. Note that these are not applications which
+SmallWorld directly supports yet.
+
+
+## Installation
+
+To install SmallWorld from this repo, run:
+
+```bash
+pip install .
+```
+
+## Usage
+
+Print basic usage and help:
+
+```bash
+smallworld --help
+```
+
+## Contributing
+
+Pull requests and issues more than welcome.
+
+### Development
+
+To set up a development environment from this repo, install SmallWorld in
+editable mode with extras for development and testing. Use the include
+constraints to install frozen versions and ensure a consistent development
+environment.
+
+```bash
+pip install -e .[development] -c constraints.txt
+```
+
+#### Code Style
+
+Pre-commit hooks are available for automatic code formatting, linting, and type
+checking via [pre-commit](https://pre-commit.com/). To enable them (after
+installing development dependencies), run:
+
+```bash
+pre-commit install
+```
+
+### Documentation
+
+To build the full SmallWorld documentation, after installing SmallWorld with
+`development` extras enabled, from the `docs/` directory, run:
+
+```bash
+make html
+```
+
+Or other [supported Sphinx output formats](https://www.sphinx-doc.org/en/master/usage/builders/index.html).
+
+### Testing
+
+#### Prerequisites
+
+Building the test binaries requires some dependencies which can be installed
+with:
+
+```bash
+apt-get install `cat tests/dependencies/apt.txt`
+```
+
+You can then build the tests by running:
+
+```bash
+make -C tests
+```
+
+#### Running Tests
+
+Once the test files have been built and SmallWorld has been installed, you can
+run unit and integration tests:
+
+```bash
+python3 tests/unit.py
+python3 tests/integration.py
+```
+
+## Distribution
+
+DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
+unlimited.
+
+This material is based upon work supported by the Under Secretary of Defense
+for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any
+opinions, findings, conclusions or recommendations expressed in this material
+are those of the author(s) and do not necessarily reflect the views of the
+Under Secretary of Defense for Research and Engineering.
+
+Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS
+Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice,
+U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS
+252.227-7014 as detailed above. Use of this work other than as specifically
+authorized by the U.S. Government may violate any copyrights that exist in this
+work.
+
+[MIT License](LICENSE.txt)
+
+[commit-style-image]: https://img.shields.io/badge/commits-conventional-fe5196.svg
+[conventional]: https://www.conventionalcommits.org/en/v1.0.0/
+[code-style-image]: https://img.shields.io/badge/code%20style-black-000000.svg
+[black]: https://github.com/psf/black
+[license-image]: https://img.shields.io/badge/license-MIT-green.svg
+[mit]: ./LICENSE.txt

smallworld_re-1.0.0/README.md

@@ -0,0 +1,152 @@
+# SmallWorld
+
+[![commit-style-image]][conventional]
+[![code-style-image]][black]
+[![license-image]][mit]
+
+Easier harnessing of code for analysis!
+
+## Description
+
+SmallWorld is an environment for streamlined harnessing of binary code for the
+purpose of dynamic analysis. If you have code that you got from somewhere and
+you'd like to run it and analyze those runs to understand what that code does
+or if it has bugs, then you should try SmallWorld!!
+
+There are two fundamental tenets behind SmallWorld
+* Harnessing should be easier
+* Analysis can accelerate harnessing
+
+The first of these tenets we hope to support with good software APIs. As a very
+simple example, consider the harnessing script
+[stack.py](https://github.com/smallworld-re/smallworld/blob/main/tests/square.py),
+composed using SmallWorld, in which registers are initialized and a stack is
+arranged for running the code in
+[stack.s](https://github.com/smallworld-re/smallworld/blob/main/tests/stack.s).
+For a more sophisticated example of SmallWorld's harnessing facitilites,
+consider the code snippet
+[struct.s](https://github.com/smallworld-re/smallworld/blob/main/tests/struct.s),
+which assumes a stack and input pointers to a linked list with very specific
+format. The harnessing script in this case is more complicated, including
+specifying type information for the linked list element structures as well as
+use of a simple allocator abstraction provided by SmallWorld to instantiate
+nodes and link them together appropriately:
+[struct.py](https://github.com/smallworld-re/smallworld/blob/main/tests/struct.py).
+
+The second tenet we address with purpose-built analyses which leverage a
+(possibly incomplete) harness script and that use techniques such as [Micro
+Execution](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/microx.pdf)
+and [Symbolic
+Execution](https://en.wikipedia.org/wiki/Symbolic_execution#:~:text=In%20computer%20science%2C%20symbolic%20execution,of%20a%20program%20to%20execute)
+to provide hints that can guide improving a harness. 
+
+This harness is the final output of SmallWorld and might be used in fuzzing or
+dynamic reverse engineering. Note that these are not applications which
+SmallWorld directly supports yet.
+
+
+## Installation
+
+To install SmallWorld from this repo, run:
+
+```bash
+pip install .
+```
+
+## Usage
+
+Print basic usage and help:
+
+```bash
+smallworld --help
+```
+
+## Contributing
+
+Pull requests and issues more than welcome.
+
+### Development
+
+To set up a development environment from this repo, install SmallWorld in
+editable mode with extras for development and testing. Use the include
+constraints to install frozen versions and ensure a consistent development
+environment.
+
+```bash
+pip install -e .[development] -c constraints.txt
+```
+
+#### Code Style
+
+Pre-commit hooks are available for automatic code formatting, linting, and type
+checking via [pre-commit](https://pre-commit.com/). To enable them (after
+installing development dependencies), run:
+
+```bash
+pre-commit install
+```
+
+### Documentation
+
+To build the full SmallWorld documentation, after installing SmallWorld with
+`development` extras enabled, from the `docs/` directory, run:
+
+```bash
+make html
+```
+
+Or other [supported Sphinx output formats](https://www.sphinx-doc.org/en/master/usage/builders/index.html).
+
+### Testing
+
+#### Prerequisites
+
+Building the test binaries requires some dependencies which can be installed
+with:
+
+```bash
+apt-get install `cat tests/dependencies/apt.txt`
+```
+
+You can then build the tests by running:
+
+```bash
+make -C tests
+```
+
+#### Running Tests
+
+Once the test files have been built and SmallWorld has been installed, you can
+run unit and integration tests:
+
+```bash
+python3 tests/unit.py
+python3 tests/integration.py
+```
+
+## Distribution
+
+DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
+unlimited.
+
+This material is based upon work supported by the Under Secretary of Defense
+for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any
+opinions, findings, conclusions or recommendations expressed in this material
+are those of the author(s) and do not necessarily reflect the views of the
+Under Secretary of Defense for Research and Engineering.
+
+Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS
+Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice,
+U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS
+252.227-7014 as detailed above. Use of this work other than as specifically
+authorized by the U.S. Government may violate any copyrights that exist in this
+work.
+
+[MIT License](LICENSE.txt)
+
+[commit-style-image]: https://img.shields.io/badge/commits-conventional-fe5196.svg
+[conventional]: https://www.conventionalcommits.org/en/v1.0.0/
+[code-style-image]: https://img.shields.io/badge/code%20style-black-000000.svg
+[black]: https://github.com/psf/black
+[license-image]: https://img.shields.io/badge/license-MIT-green.svg
+[mit]: ./LICENSE.txt

smallworld_re-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

smallworld_re-1.0.0/setup.py

@@ -0,0 +1,35 @@
+from setuptools import find_packages, setup
+
+with open("README.md", "r", encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    name="smallworld-re",
+    version="1.0.0",
+    author="MIT Lincoln Laboratory",
+    author_email="smallworld@ll.mit.edu",
+    url="https://github.com/smallworld-re/smallworld",
+    description="An emulation stack tracking library",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    license="MIT",
+    license_files=["LICENSE.txt"],
+    packages=find_packages(),
+    python_requires=">=3.8",
+    install_requires=["unicorn", "angr", "capstone", "lief", "pyhidra", "pypcode"],
+    extras_require={
+        "development": [
+            "black",
+            "isort",
+            "flake8",
+            "mypy",
+            "pip-tools",
+            "pre-commit",
+            "sphinx",
+            "sphinxcontrib-programoutput",
+        ],
+    },
+    include_package_data=True,
+    zip_safe=False,
+    entry_points={"console_scripts": ["smallworld = smallworld.__main__:main"]},
+)

smallworld_re-1.0.0/smallworld/__init__.py

@@ -0,0 +1,35 @@
+from importlib import metadata as __metadata
+
+metadata = __metadata.metadata("smallworld")
+
+__title__ = metadata["name"]
+__description__ = metadata["Summary"]
+__author__ = metadata["Author"]
+__version__ = metadata["version"]
+
+
+from . import (
+    analyses,
+    emulators,
+    exceptions,
+    extern,
+    hinting,
+    instructions,
+    logging,
+    platforms,
+    state,
+)
+from .helpers import *  # noqa: F401, F403
+from .helpers import __all__ as __helpers__
+
+__all__ = __helpers__ + [
+    "analyses",
+    "emulators",
+    "exceptions",
+    "extern",
+    "hinting",
+    "instructions",
+    "logging",
+    "platforms",
+    "state",
+]

smallworld_re-1.0.0/smallworld/analyses/__init__.py

@@ -0,0 +1,14 @@
+from .analysis import *  # noqa: F401, F403
+from .analysis import __all__ as __analysis__
+from .colorizer import Colorizer
+from .colorizer_summary import ColorizerSummary
+from .field_detection import FieldDetectionAnalysis, ForcedFieldDetectionAnalysis
+from .forced_exec import ForcedExecution
+
+__all__ = __analysis__ + [
+    "Colorizer",
+    "ColorizerSummary",
+    "FieldDetectionAnalysis",
+    "ForcedFieldDetectionAnalysis",
+    "ForcedExecution",
+]

smallworld_re-1.0.0/smallworld/analyses/analysis.py

@@ -0,0 +1,88 @@
+import abc
+import logging
+import typing
+
+from .. import hinting, state, utils
+
+
+class Analysis(utils.MetadataMixin):
+    """An analysis that emits some information about some code, possibly to help with harnessing."""
+
+    @abc.abstractmethod
+    def run(self, machine: state.Machine) -> None:
+        """Run the analysis.
+
+        This function **should not** modify the provided Machine. Instead, it
+        should be coppied before modification.
+
+        Arguments:
+            machine: A machine state object on which this analysis should run.
+        """
+
+        pass
+
+
+class Filter(utils.MetadataMixin):
+    """Analyses that consume and sometimes produce additional hints.
+
+    Filter analyses are analyses that consume some part of the hint
+    stream and possibly emit new higher-level, synthetic hints. These
+    analyses do not inspect machine state directly, they just react to
+    hints from other analyses.
+
+    """
+
+    def __init__(self):
+        self.listeners = []
+
+    def listen(
+        self,
+        hint: typing.Type[hinting.Hint],
+        method: typing.Callable[[hinting.Hint], None],
+    ) -> None:
+        """Register a listener for a particular hint type on the hint stream.
+
+        Arguments:
+            hint: A hint type that should trigger this listener. Note: All
+                subclasses `hint` will trigger the listener.
+            method: The method to call when the given hint type is observed.
+        """
+
+        class Handler(logging.Handler):
+            def emit(self, record):
+                method(record.msg)
+
+        handler = Handler()
+        handler.setLevel(logging.DEBUG)
+        handler.addFilter(hinting.HintSubclassFilter(hint))
+        hinting.root.addHandler(handler)
+
+        self.listeners.append(handler)
+
+    @abc.abstractmethod
+    def activate(self) -> None:
+        """Activate this filter.
+
+        Implementations should make necessary calls to `listen()` here to
+        register hint listener functions. They will be unregistered
+        automatically on destruction or manual call to `deactivate()`.
+        """
+
+        pass
+
+    def deactivate(self) -> None:
+        """Deactivate this filter.
+
+        This is done automatically on destruction of this object - you likely
+        shouldn't need to call this manually.
+        """
+
+        for handler in self.listeners:
+            hinting.root.removeHandler(handler)
+
+    def __del__(self):
+        self.deactivate()
+        # super().__del__()
+
+
+__all__ = ["Analysis", "Filter"]

smallworld_re-1.0.0/smallworld/analyses/code_coverage.py

@@ -0,0 +1,31 @@
+import logging
+import typing
+
+from .. import emulators, hinting
+from . import analysis
+
+logger = logging.getLogger(__name__)
+hinter = hinting.get_hinter(__name__)
+
+
+class CodeCoverage(analysis.Analysis):
+    """A simple analysis that logs jumps, calls, and returns."""
+
+    name = "code-coverage"
+    description = ""
+    version = "0.0.1"
+
+    def run(self, machine) -> None:
+        emulator = emulators.UnicornEmulator(machine.get_platform())
+        coverage: typing.Dict[int, int] = {}
+
+        for step in machine.step(emulator):
+            cpu = step.get_cpu()
+            pc = cpu.pc.get_content()
+            if pc in coverage:
+                coverage[pc] += 1
+            else:
+                coverage[pc] = 1
+
+        hint = hinting.CoverageHint(message="Coverage for execution", coverage=coverage)
+        hinter.info(hint)