Using Google Protocol Buffers as Glue between Java and Golang

This is a companion article for my first talk at the Go User Group “Gophers” Aachen about Protocol Buffers and coupling Go services with Java.

In a distributed system it is required to go beyond communication by shared memory, and even cross technology borders. This is nothing uncommon, it is more or less standard for a contemporary system to be designed as a distributed, interlinked collection of software modules which are not necessarily implemented in the same technology or even computer architecture.

Serving the task of establishing a communication between multiple processes of any kind basically does not take much more than a REST API, a technology which is supported perfectly in frameworks such as Spring. But, regardless of the sense a REST service makes when providing an open API, Protobuf has everything to be superior to REST in backend communications, because:

Low Overhead and size. Protobuf is a binary protocol with minimum payload and verbosity. For instance, a type- and architecture-safe pair of Double values takes from 4 to 8 bytes, * with no further overhead
Not opinionated regarding the transport. Protobuf messages can safely be transported over Busses and 1:1 communication architectures.
No excuse for missing API documentation. It is literally not possible to build any messages without providing a corresponding Interface-Definition Language fileset.
Auto-generated Protocol Buffer Objects. Protobuf supports en- and decoding messages in any mayor programming language, even across language or architecture borders.

For demonstration purposes, I built a simple Java SpringBoot service that periodically generates Protobuf-encoded messages and uses MQTT to deliver them to a tiny service implemented in Go, serving the really important purpose to compute a moving average. MQTT Broker Setup

If your lab does not provide an MQTT broker, just use the simple docker-compose recipe from the repository at the end of this article to bring up a working broker:

version: '2'

services:

  mosquitto:
    hostname: mosquitto
    image: eclipse-mosquitto
    ports:
      - 9001:9001

Protocol buffer Setup

Protobuf uses Interface Definition Language files, short IDL’s as a meta-definition of the message format. From this format, the client- and server code for all languages, such as C/C++, Java, C#, Go etc, are derived. For the democase used, a simple protobuf file suffices:

syntax = "proto3";

package gophersaah.measurements;
option java_package = "ac.gophers.domain.measurements";
option go_package = "gophers.ac/domain/measurements";

message Measurement {
  uint64 timestamp = 1;
  repeated SensorReading readings = 2;
}

message SensorReading {
  string sensorName = 1;
  double value = 2;

}

Obviously, it just defines a message named “SensorReading”, containing a String-representation of the sensors name and Double with the temperature. The explicit package names are a courtesy of namespace alignment preferences in the various languages.

To generate the actual language-specific client code, the tool protoc comes in place. Nevertheless, I very quickly refrained from using it directly, in favor of using an auto-generation toolchain or a step in my builds. For Java and Go, we use a Maven Plugin and a go-generate macro. Protobuf and Java

Generating the protobuf Java stubs is not much more than introducing a Maven Plugin:

<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.5.0</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocVersion>3.5.0</protocVersion>
<includeStdTypes>true</includeStdTypes>
<includeDirectories>
<!--<include>src/main/more_proto_imports</include>-->
</includeDirectories>
<inputDirectories>
<include>../idl</include>
</inputDirectories>
</configuration>
</execution>
</executions>
</plugin>

After the next Maven run, the protobuf generated code will be available at target/generated-sources, the IDE should be able to use this code for auto-completion right away. Generating a Protobuf Object Instance ready for serialization then does not differ much from using libraries like Immutables.io or Lombok:

Measurement m = Measurement.newBuilder()
.setTimestamp(Instant.now().getEpochSecond())
.addReadings(
SensorReading.newBuilder()
.setSensorName("Temperature-1")
.setValue(23.23)
.build()
)
.addReadings(
SensorReading.newBuilder()
.setSensorName("Temperature-2")
.setValue(42.42)
.build()
)
.build();

Defining a Publisher for the message.. public interface MeasurementPublisher { public void publish(Measurement m); }

.. and implementing it: @Service public class MqttMeasurementPublisher implements MeasurementPublisher {

MqttClient client;
MqttConnectOptions connectOptions;

@Value("${mqtt.topic}")
String topic;

public MqttMeasurementPublisher(
@Value("${mqtt.broker.url}") String brokerUrl,
@Value("${mqtt.client.id}") String clientId)
throws MqttException {
  client = new MqttClient(brokerUrl, clientId);
  connectOptions = new MqttConnectOptions();
  client.connect();
}

@Override
public void publish(Measurement m) {
try { 
  client.publish(topic, new MqttMessage(m.toByteArray()));
} catch (MqttException ex) {
  Logger.getLogger(MqttMeasurementPublisher.class.getName()).log(Level.SEVERE, null, ex);
}
}

}

After starting the service, it periodically generates messages on the MQTT bus, using the topic “gophers/measurements”. Without a consumer though, the setup does not yet make much sense. The Consumer

In Go, consuming a protobuf message from MQTT is, as usual, straightforward. With Paho, Eclipse provides a very proven MQTT client, and there is an official Protobuf implementation for Go. After importing the Paho MQTT library, it is required to specify which topics to listen on and which callbacks to invoke after a message was received. client.Subscribe(“gophers/measurements”, 0, measurementHandler) 1

client.Subscribe(“gophers/measurements”, 0, measurementHandler)

The callback has to implement the interface func(MQTT.Client, MQTT.Message). Here, it unmarshals the Protobuf message, prints its contents to stdout and performs some basic computations with it.

func measurementHandler(client MQTT.Client, message MQTT.Message) {
  measurement := &measurements.Measurement{}
  err := proto.Unmarshal(message.Payload(), measurement)

  if err != nil {
    log.Fatal("MHANDLER error=Could not decode message")
  } else {
    for _, reading := range measurement.GetReadings() {
      log.Println("sensor name=", reading.GetSensorName(), " value=", reading.GetValue())
      if value, exists := averageTemperatures[reading.GetSensorName()]; exists {
        averageTemperatures[reading.GetSensorName()] = (value + reading.GetValue()) / 2
      } else {
        averageTemperatures[reading.GetSensorName()] = reading.GetValue()
      }
    }
  }

  for sensor, value := range averageTemperatures {
    log.Println("averageTemp", " sensor="+sensor, " value=", value)
  }
}

Message Size Deathmatch

In the Java Example, there is a test which illustrates the size advantages of messages serialized with Protocol Buffers, compared to JSON.

@Test
public void serializeToJson_shouldBeMoreThan0Bytes() {
  String json = new JsonFormat().printToString(measurement); 
  int length = json.length();
  System.out.println("message.json=" + json);
  System.out.println("size(message.json)=" + length);
  assertTrue(length > 0);
}

@Test
public void serializeToProtobuf_shouldBeMoreThan0Bytes() { 
  byte[] wire = measurement.toByteArray();
  int length = wire.length; 
  System.out.println("message.protobuf=" + new Hex().encodeHexString(wire));
  System.out.println("size(message.protobuf)=" + length);
  assertTrue(length > 0);
}

Given the tests above, the output from jUnit is:

Running ac.gophers.sensorServer.SerializationSizeComparism
message.protobuf=byte[58]
size(message.protobuf)=58
message.json={"timestamp": 1518356051,"readings": [{"sensorName": "Temperature-1","value": -32.15938602925038},{"sensorName": "Temperature-2","value": 94.68777054203808}]}
size(message.json)=158
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec - in ac.gophers.sensorServer.SerializationSizeComparism

This means that a message that takes 158 bytes in JSON only requires 58 bytes in protobuf. Speaking from experience, the ratio of 1:3 easily increases to 1:8 when the messages become more complex. Alternatives and Conclusion

Protobuf is not the only framework serving the purpose of flexible exchange of binary-encoded information. Written from the same feather with a lot of simularities, there is Thrift. An alternative without the need to provide IDLs in the first place is Avro, and some people also use CBOR. I uses Protobuf for this example because of its stability and perfect Java and Go support. I very much like it for backend communication and use it since the early 2000s. But, because of the tooling and the “long live the standards” maxime, I would always favor REST for APIs exposed to 3rd parties.

You can find the code and all details in my Github profile:

https://github.com/codecyclist/gophersAachen-golang-protobuf

6 Minutes

java
avro

development

Message Size Deathmatch

Recent Posts