nmspec 1.5.0.pre → 1.5.0.pre2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c31069c7f7e84ff54eefaf56ce4d329b57ded1c7399e5115ffeb98e47f076ffa
4
- data.tar.gz: 000c7f21a97ec6057b325ee353b651be100381d7b5c1b2a02ff3946c5890d9ba
3
+ metadata.gz: 66f0ebb3eb93e07ad986a38e902277cc7987593d43dcdc19bbb54d346bdc224e
4
+ data.tar.gz: b877b1962c64ba409141d7b202bd674e189189799dbd6b3534af5a462f90a55d
5
5
  SHA512:
6
- metadata.gz: beacc64d359e823249e488b00eb4ecae72591e7b97ab9a451dfbb78d35e5d6b12739252173e6ef4dd45eb699cf25f6ea8d6df13278bdad100554c2a2674e7b0d
7
- data.tar.gz: c044170035c3145a626d729eac1aa78c1cea1d9ef70b1240f0364933da831b03811a31a3fafc39850af713b84210a43be0e81454ec253e3b4fe3a285f23af1b7
6
+ metadata.gz: dfa77fccbe36e774f5bd50c3e362afc577a5a7a125dea9e42f1fc3b8712ff7bf2f6d8ba5089d74c749e60726c5a3da2d7b6f6f6437614e1dd5fd744a9ce28c59
7
+ data.tar.gz: e25c2d97af1d9c7aea5c5109638127d0923a9e19512d7181fd8febc55a8dfde8023f693f5d4d89043b305335b56ce9439994b7336b9dbbbb3aa322cbe195be36
data/LICENSE ADDED
@@ -0,0 +1,19 @@
1
+ Copyright 2022 Jeffrey Lunt
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining a copy of
4
+ this software and associated documentation files (the "Software"), to deal in
5
+ the Software without restriction, including without limitation the rights to
6
+ use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
7
+ of the Software, and to permit persons to whom the Software is furnished to do
8
+ so, subject to the following conditions:
9
+
10
+ The above copyright notice and this permission notice shall be included in all
11
+ copies or substantial portions of the Software.
12
+
13
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,301 @@
1
+ # `nmspec`
2
+
3
+ `nmspec` (network message specification) is a combination of binary
4
+ serialization and network communication (two things that are usually treated
5
+ separately), designed to make creating TCP protocols between two ends of a
6
+ network connection easier to specify and keep consistent..
7
+
8
+ A centralized YAML file is used to describe the types and messages passed
9
+ between network peers, and from that description TCP peer code (a "messenger")
10
+ is generated in any supported output programming language. The messages
11
+ described in `nmspec` are used to create both the reading and writing sides
12
+ of the connection, so that a single source code file contains everything you
13
+ need for a network peer, regardless of if it's the client or server.
14
+
15
+ ## Motivation
16
+
17
+ `nmspec` was specifically created to help with with a problem I was facing,
18
+ where I was designing some network peers in two different programming languages
19
+ that needed to talk to each other, and I found that keeping the two sides in
20
+ sync generated a lot of bugs that I thought might be avoided by centralizing the
21
+ description of their communication protocols. Without this, what was happening
22
+ regularly was:
23
+
24
+ 1. I would change something on the server side in one programming language
25
+ 2. I would change the same thing on the client side in a different programming
26
+ language
27
+ 3. The serialization of data on one side of the network would get out of sync
28
+ with the deserialization on the other side
29
+
30
+ By describing the wire protocol in one place it was my hope that I would reduce
31
+ the amount of time I spent on synchronization issues.
32
+
33
+ ## Results
34
+
35
+ My approach to making this constantly-shifting communication easier to develop
36
+ and debug was to come up with a language-agnostic representation of the network
37
+ protocols within the game, specifically in some kind of easily editable
38
+ configuration language. YAML fits that description. I then integrated this
39
+ tightly with TCP (a decent starting point).
40
+
41
+ The code generators are written in Ruby, which is reasonably expressive for this
42
+ purpose.
43
+
44
+ # Output language support
45
+
46
+ As a starting point this gem supports network messengers in these two languages:
47
+
48
+ * [Ruby 3.0.x][ruby-lang]
49
+ * [GDScript 3.4.stable][gdscript]
50
+
51
+ `nmspec` came out of a online game project where the backend was written in
52
+ Ruby, and the frontend build with the Godot game engine, which includes the
53
+ embedded scripting language, GDSCript.
54
+
55
+ # Sample usage
56
+
57
+ ```ruby
58
+ # add 'nmspec' to your Gemfile
59
+
60
+ $ irb
61
+
62
+ > require 'nmspec'
63
+ => true
64
+ > pp Nmspec::V1.gen({
65
+ 'spec' => IO.read('generals.io.nmspec'),
66
+ 'langs' => ['ruby', 'gdscript']
67
+ })
68
+ => {
69
+ "valid"=>true,
70
+ "errors"=>[],
71
+ "warnings"=>[],
72
+ "code"=> {
73
+ "ruby"=> "< a string of generated Ruby code that you can save to a file>",
74
+ "gdscript"=> "< a string of generated GDSCript code that you can save to a file>",
75
+ }
76
+ }
77
+ ```
78
+
79
+ # Main concepts
80
+
81
+ ## Messenger
82
+
83
+ A `messenger` is the thing you're descripting in an .nmspec file. A `messenger`
84
+ has default support for reading and writing a number of numeric, string, and
85
+ array types.
86
+
87
+ ## Built-in types
88
+
89
+ The following built-in types are supported by `nmspec`
90
+
91
+ ```plaintext
92
+ bool # boolean true/false
93
+ i8 u8 i8_list u8_list # signed/unsigned 8-bit ints, and lists of the same
94
+ i16 u16 i16_list u16_list # signed/unsigned 16-bit ints, and lists of the same
95
+ i32 u32 i32_list u32_list # signed/unsigned 32-bit ints, and lists of the same
96
+ i64 u64 i64_list u64_list # signed/unsigned 64-bit ints, and lists of the same
97
+ float float_list # signed single-precision 32-bit floating point numbers, and a list of the same
98
+ double double_list # signed double-precision 64-bit floating point numbers, and a list of the same
99
+ str str_list # strings (arrays of bytes)
100
+ ```
101
+
102
+ As of this writing, all types are sent with big-endian encoding.
103
+
104
+ `*_list` types are ordered lists of elements (i.e. arrays).
105
+
106
+ There is no support for mixed-type list, mostly because socket libraries seem to
107
+ be centered around efficiently encoding/decoding streams of bytes with known bit
108
+ widths. If you want to send multiple data types one after the other, place them
109
+ into separate messages (see examples below).
110
+
111
+ ## Custom types
112
+
113
+ Custom types are a way for you to give a more domain-relevant name to the
114
+ built-in types. Custom types are not structs, nor are they similar to classes
115
+ from object-oriented programming. You could, however, write your own structs or
116
+ object classes to wrap the reading/writing of protocols, if you like, but that
117
+ would be extra work that you would need to do in your own program code.
118
+
119
+ A `Messenger` has many types.
120
+
121
+ ## Protocols
122
+
123
+ A `protocol` is a list of `messages` that pass between two `Messenger` peers. A
124
+ `Messenger` has many protocols.
125
+
126
+ ## Messages
127
+
128
+ Messages are either read (`r`) or writes (`w`) of types over a network
129
+ connection. `Messages` also define logical names for parameters and returned data.
130
+
131
+ # `nmspec` format
132
+
133
+ `nmspec` is a subset of YAML. So, first and foremost, if your `.nmspec` file is
134
+ not valid YAML, then it's definitely not valid `nmspec`.
135
+
136
+ ## Required keys:
137
+
138
+ A minimal `messenger`, with only a name and default types supported must include:
139
+
140
+ * `version` - which currently must be set to `1`
141
+ * `msgr` - the top-level key for naming and describing the messenger
142
+ * `name` - the name of the messenger
143
+ * `desc` - a description of the messenger
144
+ * `bigendian` - (optional, defaults to `true`)
145
+ * if `true`, communication uses big-endian byte order
146
+ * if `false`, communication uses little-endian
147
+ * `nodelay` - (optional, defaults to `false`)
148
+ * if `true`, disables Nagle's algorithm, which prioritizes low-latency over
149
+ throughput efficiency
150
+ * if `false`, leaves Nagle's algorithm enabled
151
+
152
+ ## Optional keys:
153
+
154
+ * `types` - if your `.nmspec` file creates custom sub-types, then this is where
155
+ you declare them
156
+ * `protos` - the top-level key for the list of messaging protocols
157
+ * for each protocol:
158
+ * `name` - the name of the protocol (converted to function/method name)
159
+ * `desc` - a description of the protocol
160
+ * `msgs` - a list of messages in the protocol
161
+
162
+ ## Sample `.nmspec` file
163
+
164
+ `demo/minimal.nmspec` shows the absolute minimum amount of information needed
165
+ to get a basic messenger working.
166
+
167
+ ```yaml
168
+ version: 1
169
+
170
+ msgr:
171
+ name: minimal
172
+ desc: this messenger only supports the built-in types, and has no custom protocols
173
+ ```
174
+
175
+ `demo/base_types.nmspec` shows an example of a one-protocol messenger that is
176
+ used to ensure that all base types can be read and written correctly.
177
+
178
+ ```yaml
179
+ version: 1
180
+
181
+ msgr:
182
+ name: base types
183
+ desc: this messenger supports the built-in types, and is mainly used for testing code generators
184
+ nodelay: true
185
+ bigendian: false
186
+
187
+ protos:
188
+ - name: all_base_types
189
+ desc: write all base types
190
+ msgs:
191
+ # type var name
192
+ # ----------------------------------------------------
193
+ - bool bool
194
+ - i8 i8
195
+ - u8 u8
196
+ - i8_list i8_list
197
+ - u8_list u8_list
198
+ - i16 i16
199
+ - u16 u16
200
+ - i16_list i16_list
201
+ - u16_list u16_list
202
+ - i32 i32
203
+ - u32 u32
204
+ - i32_list i32_list
205
+ - u32_list u32_list
206
+ - i64 i64
207
+ - u64 u64
208
+ - i64_list i64_list
209
+ - u64_list u64_list
210
+ - float float
211
+ - float_list float_list
212
+ - double double
213
+ - double_list double_list
214
+ - str str
215
+ - str_list str_list
216
+ ```
217
+
218
+ `demo/generals.io.nmspec` contains a theoretical implementation of a messenger
219
+ for the game, [generals.io][generals.io]:
220
+
221
+ ```yaml
222
+ version: 1
223
+
224
+ msgr:
225
+ name: generals.io
226
+ desc: demo nmspec file for generals.io
227
+
228
+ types:
229
+ - u8 player_id
230
+ - u8 serv_code
231
+ - str serv_msg
232
+ - u16 tile_id
233
+ - u8 terrain
234
+
235
+ protos:
236
+ - name: set_player_name
237
+ desc: client message sets the player name for a given player
238
+ msgs:
239
+ - str player_name
240
+ - name: resp_player_name
241
+ desc: server message to accept or reject the player name
242
+ msgs:
243
+ - serv_code resp_code
244
+ - serv_msg resp_msg
245
+ - name: set_player_id
246
+ desc: server message to client to set player id/color
247
+ msgs:
248
+ - player_id pid
249
+ - name: player_move
250
+ desc: client message to server to make a player move
251
+ msgs:
252
+ - tile_id from
253
+ - tile_id to
254
+ - u16 armies
255
+ - name: set_tile
256
+ desc: server message to client to set state of a tile
257
+ msgs:
258
+ - tile_id tid
259
+ - terrain ttype # 0 = hidden, 1 = blank, 2 = mountains, 3 = fort, 4 = home base
260
+ - player_id owner # 0 = no owner, 1 = player 1, 2 = player, etc.
261
+ - u16 armies
262
+ ```
263
+
264
+ ## How code is generated
265
+
266
+ Output program code is generated in the following manner:
267
+
268
+ 1. `nmspec` file is read - the source YAML file is read
269
+ 2. validity check - the YAML is checked to make sure it conforms to the `nmspec`
270
+ subset; useful errors and warnings in formatting may be added if mistakes are
271
+ found
272
+ 3. If all is well, then the parsed YAML is converted into a data structure that
273
+ is designed to be easy for code generators to interpret
274
+ 4. The data structure is passed on to one code generator per requested output
275
+ language
276
+ 5. The resulting output code in all requested languages is gathered together and
277
+ returned to the user
278
+
279
+ ## Preliminary research, and comparison to other methods
280
+
281
+ I started with researching how other people had designed network protocol
282
+ description languages/tools in the past, beginning with [Prolac][prolac]. This
283
+ lead me to other network messaging tools, binary serialization in general,
284
+ finally [Google's protocol buffers][protobuffs]. Protocol buffers were probably
285
+ the closest thing to what I wanted, and took care of binary
286
+ serialization/deserialization, but weren't packaaged with the networking layer,
287
+ which introduces additional considerations such as byte ordering, efficient
288
+ packet construction, TCP stack options, and communication retries and graceful
289
+ failover. While protocol buffers are a good design, and I think do a good job of
290
+ solving binary serialization as its own problem (becoming reusable for file I/O
291
+ as well as networks), I really wanted something that packaged serialization,
292
+ cross-language support, and TCP communication all in one package from a single
293
+ config file, so that a programmer needs only to write a single artifact (a
294
+ `.nmspec` file), and get the code for their target programming language(s)
295
+ generated automatically.
296
+
297
+ [ruby-lang]: https://www.ruby-lang.org/
298
+ [gdscript]: https://docs.godotengine.org/en/stable/getting_started/scripting/gdscript/gdscript_basics.html
299
+ [generals.io]: https://generals.io/
300
+ [prolac]: https://pdos.csail.mit.edu/archive/prolac/prolac-the.pdf
301
+ [protobuffs]: https://developers.google.com/protocol-buffers