protostar 0.1.0.beta1

data/README.md

@@ -0,0 +1,647 @@
+# protostar
+
+(Slightly unhinged) Protocol buffers in pure Ruby. protostar compiles `.proto` files to Ruby, encodes and
+decodes messages, reads them without decoding, writes them as JSON and as text, and can
+stand in for Google's `google-protobuf` gem so that `grpc` and friends run without it.
+
+No `protoc`, no C extension, no runtime dependencies. No native memory handles, GC problems, no uninspectable
+blobs, no issues with `fork()`, Fibers or Ractors or Threads. Full introspection and control. Ruby 3.2 or newer.
+
+## The north star
+
+Why bother? Of course pure Ruby will be slower, you say... and that's correct.
+
+But: a Ruby app that speaks protobuf usually drags in a native gem that
+has to compile (or come prebuilt for your exact platform), plus a `protoc` binary of a
+matching version that someone has to install on every machine that regenerates code. protoc
+versions change. The DSL it outputs changes. The formatting changes. The proto messages are
+handles to native pointers - they can't be marshaled, they can't be cached, they can't be
+recreated without access to the native gem and the kitchen sink. They are, by any stretch
+of the imagination, _as anti-Ruby spirited as things can be._
+
+This is why. We sacrifice some speed (primarily in encoding and allocations) for the
+luxury of having a readable, predictable, Ruby-spirited protobuf universum.
+
+One that doesn't scream _"We despise you and your ridiculous slow toy language, but we made it
+work because we were told to"_ every time you touch it in your `irb`.
+
+protostar is one gem. It passes protobuf's own conformance suite, reproduces protoc's
+descriptors byte for byte on protoc's own test files, and runs Google's Ruby tests through
+its `google-protobuf` shim.
+
+## Very experimental
+
+This is young code. It passes protobuf's conformance suite and Google's own Ruby tests, but
+it has not yet carried anyone's production traffic, the API will move before 1.0, and the
+`google-protobuf` shim has a documented list of places where it differs from the C gem.
+Try it, kick it, and tell me what breaks.
+
+Ideas are as welcome as bug reports. The `llm/` directory is where this project thinks out
+loud: `llm/todos/` holds feature briefs that are not scheduled yet (lazy readers, Marshal
+round trips for Rails caching, a native codec, arenas, and more), `llm/instructions/` the
+briefs that became workstreams, and `llm/outputs/` what each one reported back. If you have
+a feature in mind, open a PR that adds a brief to `llm/todos/` in the same shape as the ones
+there: what you want, why, what it would take. That is the fastest way to get it built.
+
+## Installation
+
+protostar is in prerelease, so name the version (or `gem install protostar --pre`):
+
+```ruby
+gem "protostar", "0.1.0.beta1"
+```
+
+## Compile a `.proto`
+
+Take a schema:
+
+```proto
+// proto/acme/person.proto
+syntax = "proto3";
+
+package acme;
+
+import "google/protobuf/timestamp.proto";
+
+message Person {
+  string name = 1;
+  int32 id = 2;
+  repeated string emails = 3;
+  map<string, string> labels = 4;
+  Address address = 5;
+  google.protobuf.Timestamp born_at = 6;
+  oneof contact {
+    string phone = 7;
+    string pager = 8;
+  }
+}
+
+message Address {
+  string street = 1;
+  string city = 2;
+}
+
+service Directory {
+  rpc Find(Person) returns (Person);
+}
+```
+
+and compile it:
+
+```console
+$ bundle exec protostar proto/acme/person.proto > person_pb.rb
+```
+
+With one file and no output flags the Ruby goes to stdout, and errors go to stderr.
+The generated file is meant to be read, and committed:
+
+```ruby
+# frozen_string_literal: true
+
+# Generated by protostar from person.proto. Do not edit.
+
+require "protostar"
+require "protostar/google/protobuf/timestamp_pb"
+
+Protostar::DSL.file(
+  "person.proto",
+  package: "acme",
+  syntax: :proto3,
+  dependencies: ["google/protobuf/timestamp.proto"]
+) do |f|
+  f.message "Person" do |m|
+    m.field :name, 1, :string
+    m.field :id, 2, :int32
+    m.field :emails, 3, :string, label: :repeated
+    m.map :labels, 4, :string, :string
+    m.field :address, 5, :message, ".acme.Address"
+    m.field :born_at, 6, :message, ".google.protobuf.Timestamp"
+    m.oneof :contact do |o|
+      o.field :phone, 7, :string
+      o.field :pager, 8, :string
+    end
+  end
+  f.message "Address" do |m|
+    m.field :street, 1, :string
+    m.field :city, 2, :string
+  end
+  f.service "Directory" do |s|
+    s.rpc :Find, ".acme.Person", ".acme.Person"
+  end
+end
+
+module Acme
+  Person = Protostar::DSL.message("acme.Person")
+  Address = Protostar::DSL.message("acme.Address")
+  Directory = Protostar::DSL.service("acme.Directory")
+end
+```
+
+Without `-I`, the directory the file is in is where imports are looked up, and the
+file is named `person.proto`. Once protos import each other by paths like
+`acme/person.proto`, compile them like protoc, with protoc's flags:
+
+```console
+$ bundle exec protostar compile -I proto --ruby_out=lib proto/acme/person.proto
+```
+
+That writes `lib/acme/person_pb.rb`, for the file named `acme/person.proto` under the
+include path `proto`. `--ruby_out=DIR` also takes several files at once.
+
+The well-known types (`google/protobuf/timestamp.proto` and the rest) are bundled, so
+importing them needs no `-I`. Other outputs:
+
+- `--grpc_out=DIR` writes gRPC service stubs (`*_services_pb.rb`)
+- `--rbs_out=DIR` writes RBS signatures for the classes
+- `--google_out=DIR` writes exactly what `protoc --ruby_out` writes, for the google-protobuf API
+- `--descriptor_set_out=FILE` writes a `FileDescriptorSet`, with `--include_imports` and `--include_source_info`
+
+`protostar --help` lists them all. Errors look like protoc's, file and line first.
+
+### From Rake
+
+Let Rake recompile a proto when it changes:
+
+```ruby
+rule "_pb.rb" => ->(target) { target.sub(/_pb\.rb\z/, ".proto") } do |t|
+  sh "protostar", "compile", "-I", ".", "--ruby_out=.", t.source
+end
+
+desc "Compile every .proto here into a _pb.rb next to it"
+task protos: FileList["*.proto"].pathmap("%X_pb.rb")
+```
+
+### As a protoc plugin
+
+If a build already runs protoc, protostar plugs into it as `protoc-gen-protostar`:
+
+```console
+$ bundle exec protoc -I proto --protostar_out=lib proto/acme/person.proto
+$ bundle exec protoc -I proto --protostar_out=ruby,grpc:lib proto/acme/person.proto
+```
+
+The parameter picks the outputs: `ruby` (the default), `google`, `rbs`, `grpc`.
+
+### Without generated files
+
+Compile at runtime if you would rather not commit generated code:
+
+```ruby
+files = Protostar::Compiler.new(include_paths: ["proto"]).compile(["acme/person.proto"])
+registry = Protostar::Types::Registry.new
+files.each { |f| registry.define(f) } # imports first, as compile returns them
+person_class = registry.find("acme.Person") # => #<Class acme.Person>
+person_class.new(name: "Ada")               # => #<acme.Person name: "Ada">
+```
+
+Classes defined this way are anonymous; they live in the registry, not in constants.
+
+## Lint a `.proto`
+
+`protostar lint` compiles files (or every `*.proto` under a directory) and explains
+what is wrong the way `rustc` does, then checks what compiled against best-practice
+rules, under `buf lint`'s names:
+
+```console
+$ bundle exec protostar lint proto/
+warning[W0214]: `Person` is the response of `Find` and already the request of `Find`
+  --> proto/acme/person.proto:27:29
+   |
+27 |   rpc Find(Person) returns (Person);
+   |            ------           ^^^^^^ used again
+   |            |
+   |            first used here
+   |
+   = help: give `Find` a `FindResponse` of its own
+   = note: rule RPC_REQUEST_RESPONSE_UNIQUE (default set)
+```
+
+Compiler errors get the same treatment, with a code to look up (here with `id = 1`):
+
+```console
+error[L0301]: field number 1 is already used in `acme.Person`
+  --> proto/acme/person.proto:10:14
+   |
+ 9 |   string name = 1;
+   |                 - first used by `name`
+10 |   int32 id = 1;
+   |              ^ reused here
+   |
+   = help: the next available field number is 2
+```
+
+`protostar lint --explain L0301` (or a rule, `--explain FIELD_NOT_REQUIRED`) says why
+and how to fix it; `--list-rules` lists the rules. They come in three sets, each
+including the one before:
+
+- `minimal` (11 rules): the longevity ones, led by "`required` is forever"
+  (`FIELD_NOT_REQUIRED`), plus groups, enum zero values, unreserved gaps in field
+  numbers, imports and a missing `syntax` line
+- `default` (28, what runs unless told otherwise): naming and package layout
+- `strict` (37): comments on everything, version suffixes, field numbers 1 to 15, editions
+
+Pick with `--rules default,documentation` (sets, categories or rule ids) and
+`--except FIELD_NOT_REQUIRED`, or in a `.protostar.yml` next to the inputs or in the
+current directory:
+
+```yaml
+lint:
+  rules: [default]
+  except: [PACKAGE_VERSION_SUFFIX]
+  ignore: ["vendor/**"]
+  allow_comment_directives: true
+```
+
+A comment switches a rule off for the next line, or for a stretch of the file:
+
+```proto
+// protostar-lint: disable-next-line FIELD_NOT_REQUIRED
+required string id = 1;
+```
+
+`// protostar-lint: disable COMMENT_FIELD` ... `// protostar-lint: enable COMMENT_FIELD`
+does the same for everything between. `--fix` applies the fixes that are mechanical
+(renaming a field, oneof or enum value nothing else in the file mentions, adding the
+missing `syntax` line) and prints the diff; `--diff` prints it without writing.
+`--format json` prints one object per finding, `--format github` workflow annotations
+for pull requests. The exit status is 0 when clean, 1 with warnings (unless
+`--warnings-ok`) and 2 with errors.
+
+`protostar compile --error_format=rich` prints compile errors this way too; without it,
+`compile` prints protoc's one-liners, as before.
+
+## Use the classes
+
+```ruby
+require "acme/person_pb"
+
+person = Acme::Person.new(name: "Ada", id: 7, emails: ["ada@example.com"])
+person.address = Acme::Address.new(city: "London")
+person.labels["team"] = "engines"
+person.born_at = Time.utc(1815, 12, 10)
+person.phone = "+44 20 7946 0000"
+
+person.name          # => "Ada"
+person.which_contact # => :phone
+person
+# => #<acme.Person name: "Ada", id: 7, emails: ["ada@example.com"], labels: {"team" => "engines"}, address: #<acme.Address city: "London">, born_at: #<google.protobuf.Timestamp seconds: -4861728000>, phone: "+44 20 7946 0000">
+```
+
+Fields are keyword arguments or a Hash, repeated fields are Arrays, maps are Hashes, enums
+are Symbols. Values are checked when you assign them:
+
+```ruby
+person.id = "seven"
+# raises Protostar::TypeError: acme.Person.id expects an Integer in int32 range, got "seven"
+```
+
+`Protostar::TypeError` is a Ruby `TypeError`. Fields with presence get `has_name?` and
+`clear_name`; oneofs get `which_contact`; every field is also `person[:name]`.
+
+### Binary
+
+```ruby
+bytes = person.encode         # also Acme::Person.encode(person)
+bytes.bytesize                # => 82
+Acme::Person.decode(bytes) == person # => true
+```
+
+Fields the schema does not know (sent by a newer version of it) are kept and written back
+on `encode`; `unknown_fields` lists them. A broken input raises `Protostar::DecodeError`
+with the offset:
+
+```ruby
+Acme::Person.decode("\x0a\x05ab")
+# raises Protostar::DecodeError: length 5 runs past the end of the input at byte 2
+```
+
+For a stream of messages, `encode_delimited(io)` and `Klass.decode_delimited(io)`, which
+returns nil at the end. `msg.encode_to(io)` writes to an IO without building the String.
+
+### JSON
+
+The proto3 JSON mapping, as google-protobuf writes it:
+
+```ruby
+person.to_json
+# => "{\"name\":\"Ada\",\"id\":7,\"emails\":[\"ada@example.com\"],\"labels\":{\"team\":\"engines\"},\"address\":{\"city\":\"London\"},\"bornAt\":\"1815-12-10T00:00:00Z\",\"phone\":\"+44 20 7946 0000\"}"
+Acme::Person.decode_json(person.to_json) == person # => true
+```
+
+`to_json` and `Klass.encode_json` take `preserve_proto_field_names:`, `emit_defaults:` and
+`format_enums_as_integers:`; `decode_json` takes `ignore_unknown_fields:`.
+`JSON.pretty_generate(message)` works:
+
+```ruby
+puts JSON.pretty_generate(Acme::Address.new(city: "Delft", street: "Oude Delft 1"))
+```
+
+```
+{
+  "street": "Oude Delft 1",
+  "city": "Delft"
+}
+```
+
+### Text format
+
+protoc's text format, for logs and fixtures:
+
+```ruby
+puts person.to_text
+```
+
+```
+name: "Ada"
+id: 7
+emails: "ada@example.com"
+labels {
+  key: "team"
+  value: "engines"
+}
+address {
+  city: "London"
+}
+born_at {
+  seconds: -4861728000
+}
+phone: "+44 20 7946 0000"
+```
+
+```ruby
+Protostar::Text.parse(Acme::Person, 'name: "Ada" address { city: "London" }')
+# => #<acme.Person name: "Ada", address: #<acme.Address city: "London">>
+```
+
+### Views: read without decoding
+
+A view reads fields straight from the bytes. It looks at the top level once, and decodes
+a field only when you read it; nested messages are skipped until you open them. On a
+1 MiB message, reading two fields through a view allocates 8 objects where `decode`
+allocates 2,522.
+
+```ruby
+view = Acme::Person.view(bytes)   # a String, or an IO::Buffer (a mapped file, say)
+view.address.city                 # => "London"
+view.emails.to_a                  # => ["ada@example.com"]
+view.to_message                   # the decoded message, when you want all of it
+```
+
+Views are read-only, equal to the message they hold (`view == message`), and raise the
+same errors as `decode`.
+
+### Ractors
+
+Classes and descriptors are frozen and shareable, so messages can be decoded, encoded and
+read in any Ractor. Decode with `frozen: true` to get a message you can pass between
+them:
+
+```ruby
+Ractor.shareable?(Acme::Person) # => true
+frozen = Acme::Person.decode(bytes, frozen: true)
+Ractor.shareable?(frozen)       # => true
+Ractor.new(frozen) { |m| m.name.upcase }.take # => "ADA" (Ractor#value on Ruby 4.0)
+```
+
+Load your `_pb.rb` files in the main Ractor before starting others. `require "protostar"`
+loads the binary codec and autoloads the rest on first use, and on Ruby 3.2 and 3.3
+nothing can be loaded from another Ractor, not even by `autoload`, so any part of
+protostar a Ractor will use has to be touched in the main Ractor first. Encoding,
+decoding and views need nothing beyond the classes. A first `to_json` or `decode_json`
+loads `Protostar::JSON`, `to_text` loads `Protostar::Text`, `build` loads
+`Protostar::Builder`, a `to_proto` or `to_h` on a descriptor loads `descriptor.proto`'s
+classes, and `Protostar::Timestamp` and the other well-known types load theirs. Ruby 3.4
+lifts this.
+
+### Reflection
+
+Each class answers `descriptor` (a `Protostar::Schema::Message`, with every field, option
+and resolved feature of the schema) and `reflection`, for code that handles messages it
+does not know in advance:
+
+```ruby
+Acme::Person.reflection.fields.map(&:name)
+# => ["name", "id", "emails", "labels", "address", "born_at", "phone", "pager"]
+Acme::Person.reflection.get(person, :name) # => "Ada"
+Acme::Person.descriptor.field("labels").map? # => true
+```
+
+[docs/schema-api.md](docs/schema-api.md) describes the schema objects, options and
+reflection.
+
+## Write bytes as you go: `Protostar::Builder`
+
+When a response is large, building message objects only to encode them is waste. The
+Builder writes the wire bytes as the block runs; nested messages and packed fields are
+the only things it holds back.
+
+```ruby
+bytes = Protostar::Builder.build(Acme::Person) do |pb|
+  pb.name "Team"
+  pb.labels "kind" => "group"
+  pb.address do
+    pb.city "London"
+  end
+end
+Acme::Person.decode(bytes)
+# => #<acme.Person name: "Team", labels: {"kind" => "group"}, address: #<acme.Address city: "London">>
+```
+
+Inside a nested block the same `pb` writes the nested message. A repeated message field
+takes a collection and a block per element (`pb.friends(users) { |u| pb.name u.name }`),
+an Array of Hashes, or messages. `Builder.write(io, Klass)`
+streams to an IO, `Klass.build { |pb| ... }` returns a message, and `pb.cache!(key) { }`
+caches a fragment's bytes in a Rails-style cache store. A singular field written twice
+raises unless you pass `overwrites: true`; then the later value wins when read, which
+relies on the reader applying protobuf's merge rules (every conforming one does).
+Building 1,000 nested elements this way takes a third (YJIT) to a half of the time and a fifth of the
+objects of `new` and `encode` (see the [benchmarks](docs/benchmarks.md)).
+
+`Builder.write` takes any sink: an IO or socket, a Rack or `ActionController::Live` stream,
+an Array, a Proc. Top-level fields go out through a 64 KiB buffer (`buffer: 0` writes each
+one as it completes), and only one nested message per level is held, until it closes. For a
+nested message too large to hold, `chunk_nested_at: 64 * 1024` writes it in pieces that
+every reader merges back into one; that relies on the default `overwrites: false`, the
+promise that no field is written twice. `Builder.each(io, Klass, rows) { |pb, row| ... }`
+writes a length-prefixed sequence for `decode_delimited`, and `Builder.each_chunk(Klass) {
+|pb| ... }` is an Enumerator of chunks for a Rack body.
+[docs/concepts/streaming.md](docs/concepts/streaming.md) measures what streams and what has
+to be held.
+
+## The DSL, by hand
+
+The generated files are plain Ruby. You can write one yourself:
+
+```ruby
+Protostar::DSL.file("shop/item.proto", package: "shop", syntax: :proto3) do |f|
+  f.message "Item" do |m|
+    m.field :sku, 1, :string
+    m.field :price_cents, 2, :int64, proto3_optional: true
+    m.field :tags, 3, :string, label: :repeated
+    m.map :stock, 4, :string, :int32
+  end
+end
+
+module Shop
+  Item = Protostar::DSL.message("shop.Item")
+end
+
+item = Shop::Item.new(sku: "lamp", stock: {"ams" => 3})
+item.has_price_cents?          # => false
+Shop::Item.decode(item.encode) # => #<shop.Item sku: "lamp", stock: {"ams" => 3}>
+```
+
+Options are keywords (`deprecated: true`, `packed: false`, `features: {...}`), custom
+options go under `extensions: {"acme.my_option" => value}`.
+
+## Well-known types
+
+`Timestamp`, `Duration`, `Any`, `Struct`, `Value`, `ListValue`, `Empty`, `FieldMask` and
+the wrappers have short names under `Protostar`, loaded on first use, and the helpers
+google-protobuf has:
+
+```ruby
+Protostar::Timestamp.from_time(Time.utc(2024, 3, 1))
+# => #<google.protobuf.Timestamp seconds: 1709251200>
+Protostar::Duration.new(seconds: 90).to_f # => 90.0
+any = Protostar::Any.pack(item)
+any.type_url                              # => "type.googleapis.com/shop.Item"
+any.unpack(Shop::Item) == item            # => true
+Protostar::Struct.from_hash({"a" => 1, "b" => [true, nil]}).to_h
+# => {"a" => 1.0, "b" => [true, nil]}
+```
+
+A Timestamp field takes a `Time` (as above, `person.born_at = Time.utc(...)`), a Duration
+field takes a number of seconds.
+
+## gRPC
+
+`--grpc_out` writes the stubs grpc-tools would, and they work with protostar's classes:
+
+```console
+$ bundle exec protostar compile -I proto --ruby_out=lib --grpc_out=lib proto/acme/person.proto
+```
+
+```ruby
+require "acme/person_services_pb"
+
+class Directory < Acme::Directory::Service
+  def find(person, _call)
+    Acme::Person.new(name: person.name, id: 42)
+  end
+end
+
+server = GRPC::RpcServer.new
+port = server.add_http2_port("127.0.0.1:0", :this_port_is_insecure)
+server.handle(Directory)
+Thread.new { server.run }
+server.wait_till_running
+
+stub = Acme::Directory::Stub.new("127.0.0.1:#{port}", :this_channel_is_insecure)
+stub.find(Acme::Person.new(name: "Ada")) # => #<acme.Person name: "Ada", id: 42>
+defined?(Google::Protobuf)               # => nil
+```
+
+The `grpc` gem depends on `google-protobuf`, so Bundler installs it, but with protostar's
+classes it is never loaded. Install the shim as that gem (below) and the C extension is
+gone from the bundle too.
+
+## Replacing google-protobuf
+
+protostar ships a shim that provides Google's `Google::Protobuf` API on protostar, so
+`grpc`, `googleapis-common-protos-types` and the `_pb.rb` files protoc generated run
+unchanged. It is off until you ask for it. Require it before anything requires
+`google/protobuf`:
+
+```ruby
+require "protostar/google_protobuf_shim"
+require "grpc"
+Google::Protobuf::IMPLEMENTATION # => :PROTOSTAR
+```
+
+From then on every `require "google/protobuf"` in the process loads the shim, even with
+Google's gem installed alongside (grpc's dependency makes Bundler install it). If Google's
+gem was loaded first, the require raises `LoadError`.
+
+To keep Google's gem out of the bundle entirely, use the shim as a gem named
+`google-protobuf`, from git. Every gem that depends on `google-protobuf` then gets it
+instead of Google's:
+
+```ruby
+gem "protostar", github: "julik/protostar"
+gem "google-protobuf", github: "julik/protostar", glob: "shims/google-protobuf/*.gemspec"
+```
+
+Its [README](shims/google-protobuf/README.md) has the other ways to install it, what it
+covers, and how to check it is the one loaded. [docs/migration.md](docs/migration.md) walks
+through moving an app or a gem from google-protobuf and protoc, either way.
+
+## What is not supported
+
+- **Extensions as fields.** Extension values are kept, byte for byte, as unknown fields,
+  and the text format, JSON and the shim's `FieldDescriptor#get`/`#set` read and write them
+  by name. There is no `msg.extension(...)` on protostar's own classes. This is also the one
+  conformance test protostar fails: UTF-8 checking of a string extension.
+- **The legacy DSL.** `_pb.rb` files from protoc before 3.21 (`add_file`, `add_message`)
+  do not load, natively or through the shim. Regenerate them.
+- **JSPB**, protobuf's JavaScript array format.
+- The C++ and Java options (`cc_enable_arenas`, `java_package`, ...) are kept in the
+  descriptors and otherwise ignored, as protoc's Ruby generator ignores them.
+
+## Performance
+
+Pure Ruby is slower than C, and there is no hiding it. On the [benchmarks](docs/benchmarks.md)
+(Ruby 3.4, Apple M1 Pro), decoding a populated 413-byte message takes 42 µs, 30 µs with
+YJIT, where google-protobuf's C extension takes 3 µs; encoding it, 30 µs and 12 µs against
+0.9 µs. A 50-byte message decodes in 2.5 µs and encodes in 3.6 µs with YJIT. Where
+protostar does well: with YJIT, a view reads two fields of a 1 MiB message in 53 µs, five
+times faster than the C gem decodes it; the Builder writes large responses without
+building objects; and nothing has to compile at install time. Defining classes costs
+about 1.1 ms per message class when a `_pb.rb` is loaded.
+
+If protobuf encoding is your hot path, measure with your own messages
+(`rake bench` shows how) before you switch.
+
+## Contributing
+
+```console
+$ bundle install
+$ bundle exec rake              # the tests and standard
+$ bundle exec rake shim:test    # the shim's suite, with its own bundle
+```
+
+The tests check protostar against protoc's own output, not against what we think protoc
+does:
+
+- `test/corpus/protobuf/` is a slice of the protobuf repository at v36.2 (see `PIN`), BSD
+  licensed. `test/corpus/goldens/` holds, for each of its 107 `.proto` files, the
+  `FileDescriptorSet` protoc 36.2 wrote (`.binpb`, with source info) and the same as JSON.
+  The compiler must reproduce them exactly. Never edit either by hand: `rake corpus:fetch`
+  and `rake corpus:goldens` regenerate them (the latter needs protoc, which
+  `rake protoc:install` downloads into `~/.cache/protostar`). The tests themselves never
+  need protoc.
+- `BUNDLE_WITH=oracle bundle install` and `PROTOSTAR_ORACLE=1 bundle exec rake test`
+  cross-check the codec and JSON against the C-extension gem. Without it those tests skip.
+- `rake conformance:build` builds protobuf's conformance runner (CMake and a C++ compiler);
+  `rake conformance:run` runs it.
+- `rake test:warnings` fails on any Ruby warning from `lib/`; `rake rbs:validate` checks
+  `sig/`.
+- Generated files are committed and checked: `rake bootstrap:check`, `rake examples:check`,
+  `rake conformance:check`. After changing an emitter, run the matching `:regenerate`.
+- `test/release/` builds both gems, installs them into an empty GEM_HOME and uses them
+  from a subprocess with no bundle.
+- `rake test` runs the test files in a forked worker per core (`test/parallel.rb`);
+  `PARALLEL=0`, or `TEST=` naming one file, runs them in one process. Tests make temp
+  dirs with `tmpdir`, ports with `free_port`, and keep generated classes out of the
+  process-wide registry; `test/parallel/isolation_test.rb` checks that they do.
+
+[docs/design.md](docs/design.md) is the map of the code; [docs/compiler-internals.md](docs/compiler-internals.md)
+follows the compiler stage by stage. [examples/](examples/README.md) has four runnable
+playbooks.
+
+## Licence
+
+MIT (`LICENSE.txt`). The Google `.proto` files bundled under
+`lib/protostar/compiler/include/`, the code generated from them, and the vendored protobuf
+sources under `test/corpus/` are Google's, under the BSD 3-Clause licence;
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) lists them and carries that licence.