Given an overall systems architecture or infrastructure which gets the “IoT” box ticket, the will probably be a place where data transfer and size will come into account, for instance if constrained devices are using a potentially unreliable or expensive connection, such as a cellular data connection. For instance, an enbedded monitoring device which serves the purpose of delivering real-time telemetry to the core system of a car manufacturer will quickly come to a point where JSON-Encoding might exceed the computing power required for the actual job.
In those cases, I often prefer to take a step back from fancy human-readable protocols to the size-aware binary protocols, but on the other hand, even if I implemented countless proprietary binary protocol providers and consumers, I would not implement my own today, especially when this would mean writing the same thing for multiple language in the very common event of polyglot architectures, such as Java on the server and Golang on the devices side.
How Data Serialization Frameworks work
At this point, data serialization frameworks enter the stage. What they have in common is that they provide APIs for multiple languages (usually C/C++, Java, Python, Golang) and sometimes an own (optional and out of scope) TCP/IP client/server implementation.
Schema or Schemaless
During the evaluation of several alternatives, three different types will be encountered:
- Interface definition driven: Client-and Server code, including data structures and de/serialization is derived from a definition written in a meta-language, the “Interface definition language”.
- Schemaless: The framework just provides a source and sink for information, client and server need to be aware of sequence and meaning of a certain transmission
- Extended Schemaless: A schemaless framework is extended with custom functionality to make it schema/structure-aware again, for instance if it is used as a workhorse for a given library such as Jackson
As always, there is no silver bullet here, the correct choice depends on the technical environment. When the aim simply is to save some bytes on an existing (de)serialization with Jackson, exchanging the JSON backend against its Protobuf- or CBOR counterpart provides a quick improvement for almost no effort. When different provider/consumer versions with complicated schema evolutions are expected, it might make sense to use the schema and migration support of an IDL-driven framework, and when it is preferred to handle marshalling in own code or simply work without a schema to save library overhead, schemaless might be a compelling option.
Expected reduction of transmitted data
Depending on the payload transcived, binary serialization saves between 40% and 70% compared with XML or JSON due to the missing overhead and formating. This article will draw a comparism between several options, including JSON, later.
A walkthrough
Given a system which reports telemetry from a new engine family of your friendly local automobile manufacturer via i.E. Google Protocol Buffers, a typical “hello world” example would consist of:
- Schema/IDL design
- Client code generation
- Message composition and serialization
- Message transfer
- Deserialization
Schema design
In most frameworks, the IDL formats look pretty much similar to common programming languages, i.e. Protocol Buffers has certain similarities to C-Style languages:
1syntax = "proto3";
2import "Common.proto";
3import "google/protobuf/timestamp.proto";
4
5package jcon.telemetry;
6option java_package = "de.m9d.telemetry.engine";
7option go_package = "m9d.de/telemetry/engine“;
8
9message Telegram {
10 string engineId = 1;
11 google.protobuf.Timestamp timeFrom = 2;
12 google.protobuf.Timestamp timeTo = 3;
13 double totalWork = 4;
14 Curve engineSpeed = 5;
15 Curve temperature = 6;
16 Curve oilPressure = 7;
17 Curve fuelConsumption = 8;
18}Client code generation
To use the data structures definied on our interface definition in actual applications, code for the corresponding target platform (i.E. Java) has to be generated. This purpose is served by a commandline application, but in production, this is usually integrated into the build process, for example by using a Maven Plugin, a stage in a Makefile or at least a go-generate declaration. Depending on the overall project setup, the task of generating client code could also be performed on a CI server and pulled as a standard dependency later.
This is the first place to mention that data serialization frameworks are violently non-opiniated with regards on how to integrate then in any given stucture, which provides flexibility and, on the other hand, requires some thoughts regarding the architecture.
For Protocol Buffers, the most basic CLI call would be:
1protoc
2 -I=../idl
3 —cpp_out=api/
4 —wrap=*.protoAnother more comfortable way would be including the code generation into a maven build by using the generator plugin:
1<dependency>
2 <groupId>com.google.protobuf</groupId>
3 <artifactId>protobuf-java</artifactId>
4 <version>${protoc-version}</version>
5</dependency>
6...
7<plugin>
8<groupId>com.github.os72</groupId>
9<artifactId>protoc-jar-maven-plugin</artifactId>
10<version>3.5.0</version>
11<executions>
12 <execution>
13 <phase>generate-sources</phase>
14 <goals>
15 <goal>run</goal>
16 </goals>
17 <configuration>
18 <protocVersion>3.5.0</protocVersion>
19 <includeStdTypes>true</includeStdTypes>
20 <includeDirectories>
21 </includeDirectories>
22 <inputDirectories>
23 <include>../idl</include>
24 </inputDirectories>
25 </configuration>
26 </execution>
27</executions>
28</plugin>Message Composition and Serialization
The Java code generated usually includes Builder patterns, so in many cases it is not required to use libraries such as immutables or lombok anymore. Initializing a Java version of the Telemetry class definied in the IDL above could be performed as following:
1Common.Curve.Builder sampleCurve = Common.Curve.newBuilder();
2
3for (int x = 0 ; x < 32 ; x++) {
4 sampleCurve.addPoints(
5 Common.Point.newBuilder()
6 .setX(x)
7 .setY(x)
8 .build());
9}
10
11EngineTelemetry.Telegram message= EngineTelemetry.Telegram
12 .newBuilder()
13 .setTimeFrom(Timestamp.newBuilder()
14 .setSeconds(fromTimestamp)
15 .setTimeTo(Timestamp.newBuilder()
16 .setSeconds(toTimestamp)
17 .setTotalWork(42.23)
18 .setEngineId(engineUid.toString())
19 .setFuelConsumption(sampleCurve)
20 .setOilPressure(sampleCurve)
21 .build();
22
23byte[] proto = message.toByteArray();The “proto” byte array includes the fully serialized and ready-to-transfer class instance:
10 |24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4
210 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e
320 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc
430 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@:
540 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ?
650 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @
760 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @
870 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @
980 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @
1090 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A
11A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A
12B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A
13C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A
14D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA
15E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A Compared to the same information represented by JSON, capped to the same message size, it becomes obvious that there is a certain difference in message sizes:
10 |22 65 6E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077
210 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4-
320 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f
430 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": {
540 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389
650 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo":
760 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153
870 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW
980 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil
1090 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi
11A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x":
12B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{"
13C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0
14D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y":
15E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0,While the binary version above already includes almost all 32 curve points defined in the loop above, the JSON version stops after the 4th point. Using a more efficient way of serializing the UUID in the beginning of the telegram would have increased the difference even further.
Message Transfer and Deserialization
After transfer, the byte array can be converted back into the original object on the same or any other language and architecture, for example in Java:
1EngineTelemetry.Telegram deserialized =
2 EngineTelemetry.Telegram.parseFrom(wire);Integration Considerations
A common ground between all known serialization frameworks is that they are not opiniated in any aspect which is not covered by the core aspect of (de)serializing a data structure to binary, which means that they are a few things to address during the selection and integration phase.
In exchange, it is trivial to integrate them into any messaging system, such as:
- TCP/IP raw sockets
- Messaging: MQTT, AMQP, Kafka, …
- Encapsulated in machine protocols (i.E. OPC/ua)
- XMPP, RSS, …
- Shared memory
Size optimization versus access costs
When a field is declared as a uint32, it is usually expected to end up in the serialized data exactly as defined. Depending on the original use case (or configuration) of the serializer, certain optimizations apply, such as:
- Reducing the encoded size of a field if it contains a value which can be expressed with less bytes
- Padding a field to an expected size to allow cheap an random access.
In Protocol buffers, the output message is automatically reduced if the original value can be represented in a shorter message:
1telegram.
2 .setMaximumTorque(32)
3 .setTotalWork(32)
4 .build(); // 2 8 20 21 0 0 0 0 0 0 40 40 (13 Bytes)
5
6telegram.
7 .setMaximumTorque(65535)
8 .setTotalWork(65535)
9 .build(); // 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40 (15 bytes)
In the default setting of Captain Proto, fields are padded to their maximum size, so a device which is just interested in a subset of a message could simply fetch a subset of the data received and save computation power on decoding.
Error detection
By default, there is no concept of error correction, such as checksumming or signing. If it is required, the developer has to take care of it after serializing the message.
1// Generate a message
2// Output: 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40
3wire = EngineTelemetry.Telegram.newBuilder()
4 .setTimeFrom(Timestamp.now())
5 .setTotalWork(65535)
6 .build();
7
8// Simulate a minor transmission error
9wire[10] = 0x40;
10
11// Deserialize
12// totalwork: 65535.0078125
13deserialized = EngineTelemetry.Telegram.parseFrom(wire);As long as the byte stream contains valid data which can be converted into the given structure, no error would be raised. If the transport is potentially inreliable, measures such attaching a simple checksum or message digest should be taken.
No type announcements
The serialized messages don’t contain any information regarding their data type, so if multiple messages types are transferred and the framwork does not provide a substitute, such as the one_of feature of protobuf, this has to be dealt with, too.
API comparism
Lets show some code. In this example, a telemetry telegram unit will be composed and serialized into a byte array using the Java API of the given framework. The results will be compared in terms of message size and processing time, using JBH Java Microbenchmarking.
To have a non-binary format to compare against, we use Jackson to serialize the testdata to JSON.
Google Protocol Buffers
Protocol buffers uses an interface definition language, for the demo-use case a representation could be:
1syntax = "proto3";
2import "Common.proto";
3import "google/protobuf/timestamp.proto";
4
5package jcon.telemetry;
6option java_package = "de.m9d.telemetry.engine";
7option go_package = "de.m9d/telemetry/engine";
8
9message Telegram {
10 string engineId = 1;
11 google.protobuf.Timestamp timeFrom = 2;
12 google.protobuf.Timestamp timeTo = 3;
13 double totalWork = 4;
14 Curve engineSpeed = 5;
15 Curve temperature = 6;
16 Curve oilPressure = 7;
17 Curve fuelConsumption = 8;
18}
19
20// Common.proto
21message Point {
22 float x = 1;
23 float y = 2;
24}
25
26message Curve {
27 repeated Point points = 1;
28}Encoding:
1EngineTelemetry.Telegram message= EngineTelemetry.Telegram.newBuilder()
2 .setTimeFrom(Timestamp.newBuilder().setSeconds(NOW).build())
3 .setTimeTo(Timestamp.newBuilder().setSeconds(NOW-3600).build())
4 .setTotalWork(42.23)
5 .setEngineId(UUID.randomUUID().toString())
6 .setFuelConsumption(sampleCurve)
7 .setOilPressure(sampleCurve)
8 .build();
9
10byte[] proto = message.toByteArray();Captain Proto
Captain Proto and Google Protocol Buffers have a lot of similarities, for the simple reason that they have been designed by the same developer. Captain Proto was designed to be a faster and tidier alternative and successor to Protocol Buffers, which does apply in certain combinations and scenarios.
From the design, Captain Proto also relies on Interface Definitions which look just a little bit different than its Protobuf counterparts:
1using Java = import "/java.capnp";
2
3$Java.package("de.m9d.telemetry.engine");
4$Java.outerClassname("Telemetry");
5
6struct Telegram
7{
8 code @0 :Text;
9 engineId @1 :Text;
10 timeFrom @2 :UInt32;
11 timeTo @3 :UInt32;
12 totalWork @4 :Float32;
13 engineSpeed @5: Curve;
14 temperature @6: Curve;
15 oilPressure @7: Curve;
16 fuelConsumption @8: Curve;
17}
18
19struct Curve
20{
21 points @0: List(Point);
22}
23
24struct Point
25{
26 x @0: Float32;
27 y @1: Float32;
28}Compiling the IDLs to Java code is possible either by using the commandline or integrating a maven dependency:
1<dependency>
2 <groupId>org.capnproto</groupId>
3 <artifactId>runtime</artifactId>
4 <version>0.1.1</version>
5</dependency>
6
7// Build
8<plugin>
9 <groupId>org.expretio.maven.plugins</groupId>
10 <artifactId>capnp-maven-plugin</artifactId>
11 <executions>
12 <execution>
13 <goals>
14 <goal>generate</goal>
15 </goals>
16 </execution>
17 </executions>
18</plugin>Building and serializing an object in Java, however, is more complicated, which appears to
1org.capnproto.MessageBuilder message =
2 new org.capnproto.MessageBuilder();
3
4de.m9d.telemetry.engine.Telemetry.Telegram.Builder builder = message.initRoot(de.m9d.telemetry.engine.Telemetry.Telegram.factory);
5builder.setEngineId(engineId);
6builder.setTimeFrom((int) Instant.now().getEpochSecond());
7builder.setTimeTo((int) Instant.now().getEpochSecond());
8builder.setTotalWork((float)42.561);
9
10builder.initEngineSpeed();
11de.m9d.telemetry.engine.Telemetry.Curve.Builder engineSpeedBuilder = builder.getEngineSpeed();
12engineSpeedBuilder.initPoints(32);
13
14for (int idx = 0 ; idx < 32 ; idx ++) {
15 engineSpeedBuilder.getPoints().get(idx).setX(idx);
16 engineSpeedBuilder.getPoints().get(idx).setY(idx);
17}
18
19builder.initOilPressure();
20de.m9d.telemetry.engine.Telemetry.Curve.Builder oilPressureBuilder = builder.getOilPressure();
21oilPressureBuilder.initPoints(32);
22
23for (int idx = 0 ; idx < 32 ; idx ++) {
24 oilPressureBuilder.getPoints().get(idx).setX(idx);
25 oilPressureBuilder.getPoints().get(idx).setY(idx);
26}
27
28ByteArrayOutputStream bos = new ByteArrayOutputStream();
29WritableByteChannel wbc = Channels.newChannel(bos);
30
31org.capnproto.Serialize.write(wbc, message);
32
33byte[] proto = bos.toByteArray();Apache AVRO
Apache Avro can operate both in schemaless and schema-driven mode, for a better comparism this report will focus on the schema-driven way. Like in Protobuf or Captain Proto, an IDL is compiled into platform-specific stubs, with the comfort of a maven plugin taking care of the Java side.
1[
2{
3 "namespace": "de.m9d.telemetry.telemetryserver.domain",
4 "type": "record",
5 "name": "Curve",
6 "fields": [
7 {"name": "points", "type": {"type": "array", "items": {
8 "name":"Point",
9 "type":"record",
10 "fields":[
11 {"name":"x", "type":"float"},
12 {"name":"y", "type":"float"}
13 ]
14 }}}
15 ]
16},
17{
18 "namespace": "de.m9d.telemetry.telemetryserver.domain",
19 "type": "record",
20 "name": "Telegram",
21 "fields": [
22 {"name": "engineId", "type": "string"},
23 {"name": "timeFrom", "type": {"type": "long", "logicalType": "timestamp-millis"}},
24 {"name": "timeTo", "type": {"type": "long", "logicalType": "timestamp-millis"}},
25 {"name": "totalWork", "type": "double"},
26 {"name": "engineSpeed", "type": ["null","de.m9d.telemetry.telemetryserver.domain.Curve"], "default": null },
27 {"name": "temperature", "type": ["null","de.m9d.telemetry.telemetryserver.domain.Curve"], "default": null },
28 {"name": "oilPressure", "type": ["null","de.m9d.telemetry.telemetryserver.domain.Curve"], "default": null },
29 {"name": "fuelConsumption", "type": ["null","de.m9d.telemetry.telemetryserver.domain.Curve"], "default": null }
30 ]
31}
32]Maven plugin:
1<dependency>
2 <groupId>org.apache.avro</groupId>
3 <artifactId>avro</artifactId>
4 <version>1.8.2</version>
5</dependency>
6
7<plugin>
8 <groupId>org.apache.avro</groupId>
9 <artifactId>avro-maven-plugin</artifactId>
10 <version>1.8.2</version>
11 <executions>
12 <execution>
13 <phase>generate-sources</phase>
14 <goals>
15 <goal>schema</goal>
16 </goals>
17 <configuration>
18 <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
19 <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
20 </configuration>
21 </execution>
22 </executions>
23</plugin> 1ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
2DatumWriter<Telegram> writer = new SpecificDatumWriter<>(Telegram.getClassSchema());
3BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null);
4
5List<Point> points = new ArrayList<>();
6for (float x = 0 ; x < 32 ; x++) {
7 points.add(Point.newBuilder().setX(x).setY(x).build());
8}
9Curve curve = Curve.newBuilder()
10 .setPoints(points)
11 .build();
12
13Telegram telegram = Telegram.newBuilder()
14 .setEngineId(UUID.randomUUID().toString())
15 .setTimeFrom(DateTime.now())
16 .setTimeTo(DateTime.now())
17 .setTotalWork(42.23)
18 .setEngineSpeed(curve)
19 .setFuelConsumption(curve)
20 .build();
21
22writer.write(telegram, encoder);
23encoder.flush();
24
25byte[] wire = outputStream.toByteArray();Common Binary Object Representation (CBOR)
CBOR is often neglected, but a competitive option because of its low memory footprint and wide platform support. CBOR itself does not provide schema-driven operation, but it is often used as a simple serialization workhorse in frameworks such as Jackson.
Serializing an object, such as the engine telemetry example used here, without a schema works straightforward:
1ByteArrayOutputStream baos = new ByteArrayOutputStream();
2
3CborBuilder di = new CborBuilder();
4ArrayBuilder ab = di.addArray();
5
6for (float x = 0 ; x < 32 ; x++) {
7 ab.addArray()
8 .add(x)
9 .add(x)
10 .end();
11}
12List<DataItem> curves = di.build();
13
14new CborEncoder(baos).encode(new CborBuilder()
15 .add(UUID.randomUUID().toString())
16 .add(Instant.now().getEpochSecond())
17 .add(Instant.now().getEpochSecond())
18 .add(42.23)
19 .add(curves.get(0))
20 .add(curves.get(0))
21 .build());
22byte[] encodedBytes = baos.toByteArray();Decoding the message requires knowledge of the schema used for encoding, and works right the way around.
Performance and Message Size comparism
While message sizes are easily comparable, performance measurements have a limited applicablity due to the mass of combinations which could occour. A certain binary serialization framework may have superior performance when serializing with the Java implementation and poor performance while deserializing the message i.E. with Micropython on an ESP. To have a common ground here, the observed unit is the serialization time of the Engine Telemetry above with Java, measured with Java Benchmarking Harness (JBH).
| Tech | #(message) | t(encode) |
|---|---|---|
| JSON | 1537 | 69000ns |
| Java Ser | 1022 | 5000ns |
| Protobuf | 817 | 2700ns |
| CaptainPro | 664 | 5280ns |
| CBOR | 509 | 14000ns |
| AVRO | 577 | 5700ns |
JSON, obviously, is very large and slow due to the high effort on string processing, which is even higher when deserializing JSON. Pure Java Binary serialization was acceptable and did not require any additional library, but lacks interoperability.
The multilanguage BSFs delivered compareable performance in terms of computing time, but AVRO stands out for its pre-optimization size.
Which one to use?
The easiest and most correct answer is: “It depends”. When talking to Java clients with considerable processing power only, the choice may highly be driven by the features and comfort a certain solution can deliver, such as the lightweight Client/Servers with gRPC. It may also be an option not to bother with any of them if no change in communication peers is expected during the entire product lifetime.
When it is foreseeable that any constrained device (let’s call them IoT devices) is involved and size and speed is something worth considering, I would recommend playing around with the options above and check how they integrate in the overall architecture. Personally, I mostly ended up with protobuf, which for a current project which involves messaging between a Java server and Golang Workers via RabbotMQ, AVRO was a better option.
