flipjump 1.2.0__py3-none-any.whl

flipjump/README.md

@@ -0,0 +1,95 @@
+# FlipJump Source Code
+
+In this documentation file you could find information about every python source file in the flipjump module.
+
+## The FlipJump Macro-Assembler
+
+The assembler has 4 steps:
+- parsing the .fj text files into a dictionary of macros and their ops ([fj_parser.py](assembler/fj_parser.py)).
+- resolving (unwinding) the macros (and reps) to get a straight stream of ops ([preprocessor.py](assembler/preprocessor.py)).
+- resolving the label values and getting the ops binary data ([assembler.py](assembler/assembler.py)). 
+- writing the binary data into the executable ([fjm_writer.py](fjm/fjm_writer.py)).
+
+The whole process is executed within the [assemble()](assembler/assembler.py) function.
+![Assembly of calc.fj](../res/calc__asm.jpg)
+
+- The [ops.py](assembler/inner_classes/ops.py) file contains the classes of the different assembly ops.
+- The [expr.py](assembler/inner_classes/expr.py) file contains the expression class (Expr), which is the code's representation of the assembly mathematical expressions. The expressions are based on numbers and labels.
+
+## The FlipJump Interpreter
+
+The Interpreter ([fjm_run.py](interpretter/fjm_run.py)) stores the entire memory in a dictionary {address: value}, and supports unaligned-word access. 
+
+The whole interpretation is done within the [run()](interpretter/fjm_run.py) function (also uses the [fjm_reader.py](fjm/fjm_reader.py) to read the fjm file - i.e. to get the flipjump program memory from the compiled fjm file).  
+More about [how to run](../README.md#how-to-run).
+![Running the compiled calculator](../res/calc__run.jpg)
+
+The Interpreter has a built-in debugger, and it's activated by specifying breakpoints when called (via the [breakpoints.py](interpretter/debugging/breakpoints.py)'s `BreakpointHandler`).
+The debugger can stop on the next breakpoint, or on a fixed number of executed ops after the current breakpoint.
+In order to call the debugger with the right labels, get familiar with the [generating label names](README.md#Generated-Label-Names) (and see the debugger-image there), and use the `-d`/`-b`/`-B` cli options.  
+More about [how to debug](../README.md#how-to-debug).
+
+The [macro_usage_graph.py](interpretter/debugging/macro_usage_graph.py) file exports a feature to present the macro-usage (which are the most used macros, and what % do they take from the overall flipjump ops) in a graph.  
+In order to view it, run the assembler with `--stats` (requires plotly to be installed (installed automatically with `pip install flipjump[stats]`)).  
+For example:
+![The macro-usage statistics of calc.fj](../res/calc_stats.png)
+
+## The Using-FlipJump Files
+
+- The [flipjump_cli.py](flipjump_cli.py) file is the main FlipJump cli-script. run with --help to see its capabilities. The `fj` utility runs the main() of this file.
+- The [flipjump_quickstart.py](flipjump_quickstart.py) file contains the fundamental assemble/run functions that are exposed to the users. They are wrappers to the inner api. These are the functions that will be exported when you `import flipjump`.
+
+### FJM versions
+
+The .fjm file currently has 4 versions:
+
+- Version 0: The basic version
+- Version 1: The normal version (more configurable than the basic version)
+- Version 2: The relative-jumps version (good for further compression)
+- Version 3: The compressed version
+
+You can specify the version you want with the `-v VERSION` flag.  
+The assembler chooses **by default** version **3** if the `--outfile` is specified, and version **1** if it isn't. 
+
+### Generated Label Names
+
+The generated label string is a concatenation of the macro call tree, each separated by '---', and finish with the label local-name.
+
+Each macro call string is as follows:\
+short_file_name **:** line **:** macro_called
+
+So if a->bit.b->hex.c->my_label: (a, bit.b called from file f2 lines 3,5; hex.c from file s1, line 72), the label's name will be:\
+f2:3:a---f2:5:bit.b---s1:72:hex.c---my_label
+
+On a rep-call (on index==i), the macro call string is:\
+short_file_name : line : rep{i} : macro_called\
+for example: f1:32:rep6:hex.print---f2:17:print_bit---print_label
+
+the short_file_name is (by default) s1,s2,s3,... for the standard library files (in the order of [stl/conf.json - all](stl/conf.json)),
+and f1,f2,f3,... for the compiled .fj files, in the order they are mentioned to the compiler (or appear in the test-line).
+
+You can place breakpoints to stop on specific labels using the `-d`, followed by a  `-b` and a label name (or `-B` and a part of a label name). For example:
+![Debugging Demo](../res/breakpoint.jpg)
+
+## More Files
+
+- The [fjm_consts.py](fjm/fjm_consts.py) contains the constants needed for interacting with the fjm format (used by the [fjm_reader.py](fjm/fjm_reader.py) + [fjm_writer.py](fjm/fjm_writer.py)).
+- The [utils/](utils) folder contains common utilities used and shared by the entire project:
+  - [utils/classes.py](utils/classes.py) - contains the common classes used in the entire project
+  - [utils/functions.py](utils/functions.py) - contains the common utility functions used in the entire project
+  - [utils/constants.py](utils/constants.py) - contains the project's constants and definitions.
+  - [utils/exceptions.py](utils/exceptions.py) - contains all the project's exceptions.
+- The [interpreter/io_devices/](interpretter/io_devices) folder contains modules for different Input/Output-handling classes (can be passed as a parameter to the interpreter). 
+  - The standard one is [StandardIO.py](interpretter/io_devices/StandardIO.py), which takes its input from the standard input, and write its output to the standard output.
+  - The tests use the [FixedIO.py](interpretter/io_devices/FixedIO.py), which takes a defined input and remembers its output.
+  - If you want to assert that your program takes no input and generates no output, use the [BrokenIO.py](interpretter/io_devices/BrokenIO.py), which raises exception on every input/output.
+  - Finally, the pure abstract IO handler class - [IODevice.py](interpretter/io_devices/IODevice.py).
+
+# Read More
+
+The FlipJump source is built in a way that allows simple addition of new features.
+
+Every addition should be supported from the parsing level, up to the phase that is disappears (and probably is replaced with some flipjump ops). See the `assemble()` function in [assembler](assembler/assembler.py) to better understand the assembler 'pipeline'.
+
+For example, if you want to add a new operation `a@b` that calculates _a^2+b^2_ or `a!` for _factorial(a)_, it is simple as adding a parsing rule in [fj_parser.py](assembler/fj_parser.py), and then adding the function to the op_string_to_function() in [expr.py](assembler/inner_classes/expr.py). That's it.
+

flipjump/__init__.py

@@ -0,0 +1,23 @@
+from flipjump.flipjump_cli import assemble_run_according_to_cmd_line_args
+from flipjump.flipjump_quickstart import (assemble, run, debug, run_test_output,
+                                          assemble_and_run, assemble_and_debug, assemble_and_run_test_output,
+                                          FJMVersion, TerminationCause, TerminationStatistics,
+                                          FixedIO, IODevice, StandardIO)
+from flipjump.utils.exceptions import IODeviceException, FlipJumpException
+from flipjump.interpretter.io_devices.BrokenIO import BrokenIO
+
+
+__all__ = [
+    'assemble_run_according_to_cmd_line_args',
+    'assemble',
+    'run',
+    'debug',
+    'run_test_output',
+    'assemble_and_run',
+    'assemble_and_debug',
+    'assemble_and_run_test_output',
+    'FJMVersion',
+    'TerminationCause',
+    'TerminationStatistics',
+    'FlipJumpException',
+]

flipjump/assembler/assembler.py

@@ -0,0 +1,241 @@
+import dataclasses
+from collections import defaultdict
+from pathlib import Path
+from typing import Deque, List, Dict, Tuple, Optional
+
+from flipjump.fjm.fjm_writer import Writer
+from flipjump.utils.constants import WFLIP_LABEL_PREFIX
+from flipjump.utils.functions import save_debugging_labels
+from flipjump.utils.classes import PrintTimer
+from flipjump.assembler.fj_parser import parse_macro_tree
+from flipjump.utils.exceptions import FlipJumpAssemblerException, FlipJumpException, FlipJumpWriteFjmException
+from flipjump.assembler.inner_classes.ops import FlipJump, WordFlip, LastPhaseOp, NewSegment, ReserveBits, Padding
+from flipjump.assembler.preprocessor import resolve_macros
+
+
+def assert_address_in_memory(memory_width: int, address: int):
+    if address < 0 or address >= (1 << memory_width):
+        raise FlipJumpAssemblerException(f"Not enough space with the {memory_width}-bits memory-width.")
+
+
+def validate_addresses(memory_width, first_address, last_address):
+    if first_address % memory_width != 0 or last_address % memory_width != 0:
+        raise FlipJumpAssemblerException(f'segment boundaries are unaligned: '
+                                         f'[{hex(first_address)}, {hex(last_address - 1)}].')
+
+    assert_address_in_memory(memory_width, first_address)
+    assert_address_in_memory(memory_width, last_address)
+
+
+def add_segment_to_fjm(memory_width: int,
+                       fjm_writer: Writer,
+                       first_address: int, last_address: int,
+                       fj_words: List[int], wflip_words: List[int]) -> None:
+    validate_addresses(memory_width, first_address, last_address)
+    if first_address == last_address:
+        return
+
+    data_words = fj_words + wflip_words
+    data_start = fjm_writer.add_data(data_words)
+
+    segment_start_address = first_address // memory_width
+    segment_length = (last_address - first_address) // memory_width
+
+    try:
+        fjm_writer.add_segment(segment_start_address, segment_length, data_start, len(data_words))
+    except FlipJumpWriteFjmException as e:
+        exception_message = (f"failed to add the segment: "
+                             f"{fjm_writer.get_segment_addresses_repr(segment_start_address, segment_length)}.")
+        raise FlipJumpAssemblerException(exception_message) from e
+
+    fj_words.clear()
+    wflip_words.clear()
+
+
+@dataclasses.dataclass
+class WFlipSpot:
+    list: List[int]
+    index: int
+    address: int
+
+
+class BinaryData:
+    def __init__(self, memory_width: int, first_segment: NewSegment, labels: Dict[str, int]):
+        self.memory_width = memory_width
+
+        self.first_address = first_segment.start_address
+        self.wflip_address = first_segment.wflip_start_address
+
+        self.labels = labels
+        self.wflips_so_far = 0
+
+        self.current_address = self.first_address
+
+        self.fj_words: List[int] = []
+        self.wflip_words: List[int] = []
+
+        self.padding_ops_indices: List[int] = []    # indices in self.fj_words
+
+        # return_address -> { (f3, f2, f1, f0) -> start_flip_address }
+        self.wflips_dict: Dict[int, Dict[Tuple[int, ...]]] = defaultdict(lambda: {})
+
+    def get_wflip_spot(self) -> WFlipSpot:
+        if self.padding_ops_indices:
+            index = self.padding_ops_indices.pop()
+            return WFlipSpot(self.fj_words, index, self.first_address + self.memory_width * index)
+
+        wflip_spot = WFlipSpot(self.wflip_words, len(self.wflip_words), self.wflip_address)
+        self.wflip_words += (0, 0)
+        self.wflip_address += 2*self.memory_width
+        return wflip_spot
+
+    def close_and_add_segment(self, fjm_writer: Writer) -> None:
+        add_segment_to_fjm(self.memory_width, fjm_writer, self.first_address, self.wflip_address, self.fj_words, self.wflip_words)
+
+    def _insert_wflip_label(self, address: int):
+        self.labels[f'{WFLIP_LABEL_PREFIX}{self.wflips_so_far}'] = address
+        self.wflips_so_far += 1
+
+    def insert_fj_op(self, flip: int, jump: int) -> None:
+        self.fj_words += (flip, jump)
+        self.current_address += 2*self.memory_width
+
+    def insert_wflip_ops(self, word_address: int, flip_value: int, return_address: int) -> None:
+        if 0 == flip_value:
+            self.insert_fj_op(0, return_address)
+        else:
+            assert_address_in_memory(self.memory_width, flip_value)
+
+            return_dict = self.wflips_dict[return_address]
+
+            # this is the order of flip_addresses (tested with many other orders) that produces the best
+            #  found-statistic for searching flip_bit[:i] with different i's in return_dict.
+            flip_addresses = [word_address + i for i in range(self.memory_width) if flip_value & (1 << i)][::-1]
+
+            # insert the first op
+            self.insert_fj_op(flip_addresses.pop(), 0)
+            last_return_address_index = self.fj_words, len(self.fj_words) - 1
+
+            while flip_addresses:
+                flips_key = tuple(flip_addresses)
+                ops_list, last_address_index = last_return_address_index
+
+                if flips_key in return_dict:
+                    # connect the last op to the already created wflip-chain
+                    ops_list[last_address_index] = return_dict[flips_key]
+                    return
+                else:
+                    # insert a new wflip op, and connect the last one to it
+                    wflip_spot = self.get_wflip_spot()
+                    self._insert_wflip_label(wflip_spot.address)
+
+                    ops_list[last_address_index] = wflip_spot.address
+                    return_dict[flips_key] = wflip_spot.address
+
+                    wflip_spot.list[wflip_spot.index] = flip_addresses.pop()
+                    last_return_address_index = wflip_spot.list, wflip_spot.index + 1
+
+            ops_list, last_address_index = last_return_address_index
+            ops_list[last_address_index] = return_address
+
+    def insert_padding(self, ops_count: int) -> None:
+        for i in range(len(self.fj_words), len(self.fj_words) + 2 * ops_count, 2):
+            self.padding_ops_indices.append(i)
+            self.fj_words += (0, 0)
+        self.current_address += ops_count * (2 * self.memory_width)
+
+    def insert_new_segment(self, fjm_writer: Writer, first_address: int, wflip_first_address: int) -> None:
+        self.close_and_add_segment(fjm_writer)
+
+        self.first_address = first_address
+        self.wflip_address = wflip_first_address
+        self.current_address = self.first_address
+
+        self.padding_ops_indices.clear()
+
+    def insert_reserve_bits(self, fjm_writer: Writer, new_first_address: int) -> None:
+        add_segment_to_fjm(self.memory_width, fjm_writer, self.first_address, new_first_address, self.fj_words, [])
+
+        self.first_address = new_first_address
+        self.current_address = self.first_address
+
+        self.padding_ops_indices.clear()
+
+
+def labels_resolve(ops: Deque[LastPhaseOp], labels: Dict[str, int],
+                   memory_width: int, fjm_writer: Writer) -> None:
+    """
+    resolve the labels and expressions to get the list of fj ops, and add all the data and segments into the fjm_writer.
+    @param ops:[in]: the list ops returned from the preprocessor stage
+    @param labels:[in]: dictionary from label to its resolved value
+    @param memory_width: the memory-width
+    @param fjm_writer: [out]: the .fjm file writer
+    """
+    first_segment: NewSegment = ops.popleft()
+    if not isinstance(first_segment, NewSegment):
+        raise FlipJumpAssemblerException(f"The first op must be of type NewSegment (and not {first_segment}).")
+
+    binary_data = BinaryData(memory_width, first_segment, labels)
+
+    for op in ops:
+        if isinstance(op, FlipJump):
+            try:
+                binary_data.insert_fj_op(op.get_flip(labels), op.get_jump(labels))
+            except FlipJumpException as e:
+                raise FlipJumpAssemblerException(f"{e} in op {op}.")
+
+        elif isinstance(op, WordFlip):
+            try:
+                binary_data.insert_wflip_ops(op.get_word_address(labels), op.get_flip_value(labels),
+                                             op.get_return_address(labels))
+            except FlipJumpException as e:
+                raise FlipJumpAssemblerException(f"{e} in op {op}.")
+
+        elif isinstance(op, Padding):
+            binary_data.insert_padding(op.ops_count)
+
+        elif isinstance(op, NewSegment):
+            binary_data.insert_new_segment(fjm_writer, op.start_address, op.wflip_start_address)
+
+        elif isinstance(op, ReserveBits):
+            binary_data.insert_reserve_bits(fjm_writer, op.first_address_after_reserved)
+
+        else:
+            raise FlipJumpAssemblerException(f"Can't resolve/assemble the next opcode - {str(op)}")
+
+    binary_data.close_and_add_segment(fjm_writer)
+
+
+def assemble(input_files: List[Tuple[str, Path]], memory_width: int, fjm_writer: Writer, *,
+             warning_as_errors: bool = True, debugging_file_path: Optional[Path] = None,
+             show_statistics: bool = False, print_time: bool = True)\
+        -> None:
+    """
+    runs the assembly pipeline. assembles the input files to a .fjm.
+    :param input_files:[in]: a list of (short_file_name, fj_file_path). The files will to be parsed in that given order.
+    :param memory_width: the memory-width
+    :param fjm_writer:[out]: the .fjm file writer
+    :param warning_as_errors: treat warnings as errors (stop execution on warnings)
+    :param debugging_file_path:[out]: is specified, save debug information in this file
+    :param show_statistics: if true shows macro-usage statistics
+    :param print_time: if true prints the times of each assemble-stage
+    """
+    try:
+        with PrintTimer('  parsing:         ', print_time=print_time):
+            macros = parse_macro_tree(input_files, memory_width, warning_as_errors)
+
+        with PrintTimer('  macro resolve:   ', print_time=print_time):
+            ops, labels = resolve_macros(memory_width, macros, show_statistics=show_statistics)
+
+        with PrintTimer('  labels resolve:  ', print_time=print_time):
+            labels_resolve(ops, labels, memory_width, fjm_writer)
+
+        with PrintTimer('  create binary:   ', print_time=print_time):
+            fjm_writer.write_to_file()
+            save_debugging_labels(debugging_file_path, labels)
+
+    except FlipJumpException as fj_exception:
+        raise fj_exception
+    except Exception as unknown_exception:
+        raise FlipJumpAssemblerException("Unknown exception during assembling the .fj files, please report this bug") \
+            from unknown_exception