ruby_memprofiler_pprof 0.0.1

data/ext/ruby_memprofiler_pprof/vendor/upb/docs/wrapping-upb.md

@@ -0,0 +1,444 @@
+
+<!---
+This document contains embedded graphviz diagrams inside ```dot blocks.
+
+To convert it to rendered form using render.py:
+  $ ./render.py wrapping-upb.in.md
+
+You can also live-preview this document with all diagrams using Markdown Preview Enhanced
+in Visual Studio Code:
+  https://marketplace.visualstudio.com/items?itemName=shd101wyy.markdown-preview-enhanced
+--->
+
+# Building a protobuf library on upb
+
+This is a guide for creating a new protobuf implementation based on upb.  It
+starts from the beginning and walks you through the process, highlighting
+some important design choices you will need to make.
+
+## Overview
+
+A protobuf implementation consists of two main pieces:
+
+1. a code generator, run at compile time, to turn `.proto` files into source
+   files in your language (we will call this "zlang", assuming an extension of ".z").
+2. a runtime component, which implements the wire format and provides the data
+   structures for representing protobuf data and metadata.
+
+<br/>
+
+```dot {align="center"}
+digraph {
+    rankdir=LR; 
+    newrank=true;
+    node [style="rounded,filled" shape=box]
+    "foo.proto" -> protoc;
+    "foo.proto" [shape=folder];
+    protoc [fillcolor=lightgrey];
+    protoc -> "protoc-gen-zlang";
+    "protoc-gen-zlang" -> "foo.z";
+    "protoc-gen-zlang" [fillcolor=palegreen3];
+    "foo.z" [shape=folder];
+    labelloc="b";
+    label="Compile Time";
+}
+```
+
+<br/>
+
+```dot {align="center"}
+digraph {
+    newrank=true;
+    node [style="rounded,filled" shape=box fillcolor=lightgrey]
+    "foo.z" -> "zlang/upb glue (FFI)";
+    "zlang/upb glue (FFI)" -> "upb (C)";
+    "zlang/upb glue (FFI)" [fillcolor=palegreen3];
+    labelloc="b";
+    label="Runtime";
+}
+```
+
+The parts in green are what you will need to implement.
+
+Note that your code generator (`protoc-gen-zlang`) does *not* need to generate
+any C code (eg. `foo.c`). While upb itself is written in C, upb's parsers and
+serializers are fully table-driven, which means there is never any need or even
+benefit to generating C code for each proto. upb is capable of full-speed
+parsing even when schema data is loaded at runtime from strings embedded into
+`foo.z`. This is a key benefit of upb compared with C++ protos, which have
+traditionally relied on generated parsers in `foo.pb.cc` files to achieve full
+parsing speed, and suffered a ~10x speed penalty in the parser when the schema
+data was loaded at runtime.
+
+## Prerequisites
+
+There are a few things that the language runtime must provide in order to wrap
+upb.
+
+1.  **FFI**: To wrap upb, your language must be able to call into a C API
+    through a Foreign Function Interface (FFI). Most languages support FFI in
+    some form, either through "native extensions" (in which you write some C
+    code to implement new methods in your language) or through a direct FFI (in
+    which you can call into regular C functions directly from your language
+    using a special library).
+2.  **Finalizers, Destructors, or Cleaners**: The runtime must provide
+    finalizers or destructors of some sort. There must be a way of triggering a
+    call to a C function when the language garbage collects or otherwise
+    destroys an object. We don't care much whether it is a finalizer, a
+    destructor, or a cleaner, as long as it gets called eventually when the
+    object is destroyed. upb allocates memory in C space, and a finalizer is our
+    only way of making sure that memory is freed and does not leak.
+3.  **HashMap with weak values**: (optional) This is not a strong requirement,
+    but it is sometimes helpful to have a global hashmap with weak values to act
+    as a `upb_msg* -> wrapper` object cache. We want the values to be weak (not
+    the keys). There is some question about whether we want to continue to use
+    this pattern going forward.
+
+## Reflection vs. MiniTables
+
+The first key design decision you will need to make is whether your generated
+code will access message data via reflection or minitables. Generally more
+dynamic languages will want to use reflection and more static languages will
+want to use minitables.
+
+### Reflection
+
+Reflection-based data access makes the most sense in highly dynamic language
+interpreters, where method dispatch is generally resolved via strings and hash
+table lookups.  
+
+In such languages, you can often implement a special method like `__getattr__`
+(Python) or `method_missing` (Ruby) that receives the method name as a string.
+Using upb's reflection, you can look up a field name using the method name,
+thereby using a hash table belonging to upb instead of one provided by the
+language.
+
+```python
+class FooMessage:
+  # Written in Python for illustration, but in practice we will want to
+  # implement this in C for speed.
+  def __getattr__(self, name):
+    field = FooMessage.descriptor.fields_by_name[name]
+    return field.get_value(self)
+```
+
+Using this design, we only need to attach a single `__getattr__` method to each
+message class, instead of defining a getter/setter for each field. In this way
+we can avoid duplicating hash tables between upb and the language interpreter,
+reducing memory usage.
+
+Reflection-based access requires loading full reflection at runtime. Your
+generated code will need to embed serialized descriptors (ie. a serialized
+message of `descriptor.proto`), which has some amount of size overhead and
+exposes all message/field names to the binary. It also forces a hash table
+lookup in the critical path of field access. If method calls in your language
+already have this overhead, then this is no added burden, but for statically
+dispatched languages it would cause extra overhead.
+
+If we take this path to its logical conclusion, all class creation can be
+performed fully dynamically, using only a binary descriptor as input. The
+"generated code" becomes little more than an embedded descriptor plus a
+library call to load it. Python has recently gone down this path. Generated
+code now looks something like this:
+
+```python
+# main_pb2.py
+from google3.net.proto2.python.internal import builder as _builder
+from google3.net.proto2.python.public import descriptor_pool as _descriptor_pool
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile("<...>")
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google3.main_pb2', globals())
+```
+
+This is all the runtime needs to create all of the classes for messages defined
+in that serialized descriptor.  This code has no pretense of readability, but
+a separate `.pyi` stub file provides a fully expanded and readable list of all
+methods a user can expect to be available:
+
+```python
+# main_pb2.pyi
+from google3.net.proto2.python.public import descriptor as _descriptor
+from google3.net.proto2.python.public import message as _message
+from typing import ClassVar as _ClassVar, Optional as _Optional
+
+DESCRIPTOR: _descriptor.FileDescriptor
+
+class MyMessage(_message.Message):
+    __slots__ = ["my_field"]
+    MY_FIELD_FIELD_NUMBER: _ClassVar[int]
+    my_field: str
+    def __init__(self, my_field: _Optional[str] = ...) -> None: ...
+```
+
+To use reflection-based access:
+
+1. Load and access descriptor data using the interfaces in google3/third_party/upb/upb/def.h.
+2. Access message data using the interfaces in google3/third_party/upb/upb/reflection.h.
+
+### MiniTables
+
+MiniTables are a "lite" schema representation that are much smaller that
+reflection. MiniTables omit names, options, and almost everything else from the
+`.proto` file, retaining only enough information to parse and serialize binary
+format.
+
+MiniTables can be loaded into upb through *MiniDescriptors*. MiniDescriptors are
+a byte-oriented format that can be embedded into your generated code and passed
+to upb to construct MiniTables. MiniDescriptors only use printable characters,
+and therefore do not require escaping when embedding them into generated code
+strings. Overall the size savings of MiniDescriptors are ~60x compared with
+regular descriptors.
+
+MiniTables and MiniDescriptors are a natural choice for compiled languages that
+resolve method calls at compile time. For languages that are sometimes compiled
+and sometimes interpreted, there might not be an obvious choice. When a method
+call is statically bound, we want to remove as much overhead as possible,
+especially from accessors. In the extreme case, we can use unsafe APIs to read
+raw memory at a known offset:
+
+```java
+// Example of a maximally-optimized generated accessor.
+class FooMessage {
+    public long getBarField() {
+        // Using Unsafe should give us performance that is comparable to a
+        // native member access.
+        //
+        // The constant "24" is obtained from upb at compile time.
+        sun.misc.Unsafe.getLong(this.ptr, 24);
+    }
+}
+```
+
+This design is very low-level, and tightly couples the generated code to one
+specific version of the schema and compiler.  A slower but safer version would
+look up a field by field number:
+
+```java
+// Example of a more loosely-coupled accessor.
+class FooMessage {
+    public long getBarField() {
+        // The constant "2" is the field number.  Internally this will look
+        // up the number "2" in the MiniTable and use that to read the value
+        // from the message.
+        upb.glue.getLong(this.ptr, 2);
+    }
+}
+```
+
+One downside of MiniTables is that they cannot support parsing or serializing
+to JSON or TextFormat, because they do now know the field names.  It should be
+possible to generate reflection data "on the side", into separate generated
+code files, so that reflection is only pulled in if it is being used.  However
+APIs to do this do not exist yet.
+
+To use MiniTable-based access:
+
+1. Load and access MiniDescriptors data using the interfaces in google3/third_party/upb/upb/mini_table.h.
+2. Access message data using the interfaces in google3/third_party/upb/upb/mini_table_accessors.h.
+
+## Memory Management
+
+One of the core design challenges when wrapping upb is memory management.  Every
+language runtime will have some memory management system, whether it is
+garbage collection, reference counting, manual memory management, or some hybrid
+of these.  upb is written in C and uses arenas for memory management, but upb is
+designed to integrate with a wide variety of memory management schemes, and it
+provides a number of tools for making this integration as smooth as possible.
+
+### Arenas
+
+upb defines data structures in C to represent messages, arrays (repeated
+fields), and maps.  A protobuf message is a hierarchical tree of these objects.
+For example, a relatively simple protobuf tree might look something like this:
+
+```dot {align="center"}
+digraph G {
+  rankdir=LR;
+  newrank=true;
+  node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=1, ordering=out]
+  upb_msg -> upb_msg2;
+  upb_msg -> upb_array;
+  upb_msg [label="upb Message" fillcolor=1]
+  upb_msg2 [label="upb Message"];
+  upb_array [label="upb Array"]
+}
+```
+
+All upb objects are allocated from an arena.  An arena lets you allocate objects
+individually, but you cannot free individual objects; you can only free the arena
+as a whole.  When the arena is freed, all of the individual objects allocated
+from that arena are freed together.
+
+```dot {align="center"}
+digraph G {
+  rankdir=LR;
+  newrank=true;
+  subgraph cluster_0 {
+    label = "upb Arena"
+    graph[style="rounded,filled" fillcolor=gray]
+    node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=1, ordering=out]
+    upb_msg -> upb_array;
+    upb_msg -> upb_msg2;
+    upb_msg [label="upb Message" fillcolor=1]
+    upb_msg2 [label="upb Message"];
+    upb_array [label="upb Array"];
+  }
+}
+```
+
+In simple cases, the entire tree of objects will all live in a single arena.
+This has the nice property that there cannot be any dangling pointers between
+objects, since all objects are freed at the same time.
+
+However upb allows you to create links between any two objects, whether or
+not they are in the same arena.  The library does not know or care what arenas
+the objects are in when you create links between them.
+
+```dot {align="center"}
+digraph G {
+  rankdir=LR;
+  newrank=true;
+  subgraph cluster_0 {
+    label = "upb Arena 1"
+    graph[style="rounded,filled" fillcolor=gray]
+    node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=1, ordering=out]
+    upb_msg -> upb_array;
+    upb_msg -> upb_msg2;
+    upb_msg [label="upb Message 1" fillcolor=1]
+    upb_msg2 [label="upb Message 2"];
+    upb_array [label="upb Array"];
+  }
+  subgraph cluster_1 {
+    label = "upb Arena 2"
+    graph[style="rounded,filled" fillcolor=gray]
+    node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=1]
+    upb_msg3;
+  }
+  upb_msg2 -> upb_msg3;
+  upb_msg3 [label="upb Message 3"];
+}
+```
+
+When objects are on separate arenas, it is the user's responsibility to ensure
+that there are no dangling pointers.  In the example above, this means Arena 2
+must outlive Message 1 and Message 2.
+
+### Integrating GC with upb
+
+In languages with automatic memory management, the goal is to handle all of the
+arenas behind the scenes, so that the user does not have to manage them manually
+or even know that they exist.
+
+We can achieve this goal if we set up the object graph in a particular way.  The
+general strategy is to create wrapper objects around all of the C objects,
+including the arena.  Our key goal is to make sure the arena wrapper is not
+GC'd until all of the C objects in that arena have become unreachable.
+
+For this example, we will assume we are wrapping upb in Python:
+
+```dot {align="center"}
+digraph G {
+  rankdir=LR;
+  newrank=true;
+  compound=true;
+
+  subgraph cluster_1 {
+    label = "upb Arena"
+    graph[style="rounded,filled" fillcolor=gray]
+    node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=1, ordering=out]
+    upb_msg -> upb_array [style=dashed];
+    upb_msg -> upb_msg2 [style=dashed];
+    upb_msg [label="upb Message" fillcolor=1]
+    upb_msg2 [label="upb Message"];
+    upb_array [label="upb Array"]
+    dummy [style=invis]
+  }
+  subgraph cluster_python {
+    node [style="rounded,filled" shape=box colorscheme=accent8 fillcolor=2]
+    peripheries=0
+    py_upb_msg [label="Python Message"];
+    py_upb_msg2 [label="Python Message"];
+    py_upb_arena [label="Python Arena"];
+  }
+  py_upb_msg -> upb_msg [style=dashed];
+  py_upb_msg2->upb_msg2 [style=dashed];
+  py_upb_msg2 -> py_upb_arena [color=springgreen4];
+  py_upb_msg -> py_upb_arena [color=springgreen4];
+  py_upb_arena -> dummy [lhead=cluster_1, color=red];
+  {
+     rank=same;
+     upb_msg;
+     py_upb_msg;
+  }
+  {
+     rank=same;
+     upb_array;
+     upb_msg2;
+     py_upb_msg2;
+  }
+  {  rank=same;
+     dummy;
+     py_upb_arena;
+  }
+  dummy->upb_array [style=invis];
+  dummy->upb_msg2 [style=invis];
+
+  subgraph cluster_01 {
+    node [shape=plaintext]
+    peripheries=0
+    key [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
+      <tr><td align="right" port="i1">raw ptr</td></tr>
+      <tr><td align="right" port="i2">unique ptr</td></tr>
+      <tr><td align="right" port="i3">shared (GC) ptr</td></tr>
+      </table>>]
+    key2 [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
+      <tr><td port="i1">&nbsp;</td></tr>
+      <tr><td port="i2">&nbsp;</td></tr>
+      <tr><td port="i3">&nbsp;</td></tr>
+      </table>>]
+    key:i1:e -> key2:i1:w [style=dashed]
+    key:i2:e -> key2:i2:w [color=red]
+    key:i3:e -> key2:i3:w [color=springgreen4]
+  }
+    key2:i1:w -> upb_msg [style=invis];
+  {
+    rank=same;
+    key;
+    upb_msg;
+  }
+}
+```
+
+In this example we have three different kinds of pointers:
+
+* **raw ptr**: This is a pointer that carries no ownership.
+* **unique ptr**: This is a pointer has *unique ownership* of the target.  The owner
+  will free the target in its destructor (or finalizer, or cleaner).  There can
+  only be a single unique pointer to a given object.
+* **shared (GC) ptr**: This is a pointer that has *shared ownership* of the
+  target.  Many objects can point to the target, and the target will be deleted
+  only when all such references are gone.  In a runtime with automatic memory
+  management (GC), this is a reference that participates in GC.  In Python such
+  references use reference counting, but in other VMs they may use mark and
+  sweep or some other form of GC instead.
+
+The Python Message wrappers have only raw pointers to the underlying message,
+but they contain a shared pointer to the arena that will ensure that the raw
+pointer remains valid.  Only when all message wrapper objects are destroyed
+will the Python Arena become unreachable, and the upb arena ultimately freed.
+
+### Links between arenas with "Fuse"
+
+The design given above works well for objects that live in a single arena. But
+what if a user wants to create a link between two objects in different arenas?
+
+TODO
+
+## UTF-8 vs. UTF-16
+
+TODO
+
+## Object Cache
+
+TODO

data/ext/ruby_memprofiler_pprof/vendor/upb/python/BUILD

@@ -0,0 +1,216 @@
+# Copyright (c) 2009-2021, Google LLC
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in the
+#       documentation and/or other materials provided with the distribution.
+#     * Neither the name of Google LLC nor the
+#       names of its contributors may be used to endorse or promote products
+#       derived from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL Google LLC BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# begin:github_only
+load("//python:py_extension.bzl", "py_extension")
+# end:github_only
+
+load("@bazel_skylib//lib:selects.bzl", "selects")
+load("@bazel_skylib//rules:common_settings.bzl", "bool_flag", "string_flag")
+load("//bazel:build_defs.bzl", "UPB_DEFAULT_COPTS")
+
+licenses(["notice"])
+
+package(
+    default_visibility = ["//python/dist:__pkg__"],
+)
+
+LIMITED_API_FLAG_SELECT = {
+    ":limited_api_3.7": ["-DPy_LIMITED_API=0x03070000"],
+    ":limited_api_3.10": ["-DPy_LIMITED_API=0x030a0000"],
+    "//conditions:default": [],
+}
+
+bool_flag(
+    name = "limited_api",
+    build_setting_default = False,
+)
+
+string_flag(
+    name = "python_version",
+    build_setting_default = "system",
+    values = [
+        "system",
+        "37",
+        "38",
+        "39",
+        "310",
+    ],
+)
+
+config_setting(
+    name = "limited_api_3.7",
+    flag_values = {
+        ":limited_api": "True",
+        ":python_version": "37",
+    },
+)
+
+config_setting(
+    name = "full_api_3.7_win32",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "37",
+    },
+    values = {"cpu": "win32"},
+)
+
+config_setting(
+    name = "full_api_3.7_win64",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "37",
+    },
+    values = {"cpu": "win64"},
+)
+
+selects.config_setting_group(
+    name = "full_api_3.7",
+    match_any = [
+        ":full_api_3.7_win32",
+        ":full_api_3.7_win64",
+    ],
+)
+
+config_setting(
+    name = "full_api_3.8_win32",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "38",
+    },
+    values = {"cpu": "win32"},
+)
+
+config_setting(
+    name = "full_api_3.8_win64",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "38",
+    },
+    values = {"cpu": "win64"},
+)
+
+selects.config_setting_group(
+    name = "full_api_3.8",
+    match_any = [
+        ":full_api_3.8_win32",
+        ":full_api_3.8_win64",
+    ],
+)
+
+config_setting(
+    name = "full_api_3.9_win32",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "39",
+    },
+    values = {"cpu": "win32"},
+)
+
+config_setting(
+    name = "full_api_3.9_win64",
+    flag_values = {
+        ":limited_api": "False",
+        ":python_version": "39",
+    },
+    values = {"cpu": "win64"},
+)
+
+selects.config_setting_group(
+    name = "full_api_3.9",
+    match_any = [
+        "full_api_3.9_win32",
+        ":full_api_3.9_win64",
+    ],
+)
+
+config_setting(
+    name = "limited_api_3.10_win32",
+    flag_values = {
+        ":limited_api": "True",
+        ":python_version": "310",
+    },
+    values = {"cpu": "win32"},
+)
+
+config_setting(
+    name = "limited_api_3.10_win64",
+    flag_values = {
+        ":limited_api": "True",
+        ":python_version": "310",
+    },
+    values = {"cpu": "win64"},
+)
+
+selects.config_setting_group(
+    name = "limited_api_3.10",
+    match_any = [
+        ":limited_api_3.10_win32",
+        ":limited_api_3.10_win64",
+    ],
+)
+
+py_extension(
+    name = "_message",
+    srcs = [
+        "convert.c",
+        "convert.h",
+        "descriptor.c",
+        "descriptor.h",
+        "descriptor_containers.c",
+        "descriptor_containers.h",
+        "descriptor_pool.c",
+        "descriptor_pool.h",
+        "extension_dict.c",
+        "extension_dict.h",
+        "map.c",
+        "map.h",
+        "message.c",
+        "message.h",
+        "protobuf.c",
+        "protobuf.h",
+        "python_api.h",
+        "repeated.c",
+        "repeated.h",
+        "unknown_fields.c",
+        "unknown_fields.h",
+    ],
+    copts = UPB_DEFAULT_COPTS + select(LIMITED_API_FLAG_SELECT) + [
+        # The Python API requires patterns that are ISO C incompatible, like
+        # casts between function pointers and object pointers.
+        "-Wno-pedantic",
+    ],
+    deps = [
+        "//:collections",
+        "//:descriptor_upb_proto_reflection",
+        "//:reflection",
+        "//:table",
+        "//:textformat",
+        "//:upb",
+        "//upb/util:compare",
+        "//upb/util:def_to_proto",
+        "//upb/util:required_fields",
+    ],
+)