Coupling Go Services with Java Using Protocol Buffers over MQTT

This article accompanies my first talk at the Go User Group “Gophers” Aachen, where I explored using Protocol Buffers to couple Go services with Java.

Introduction

#

In a distributed system, communication must extend beyond shared memory, often crossing technological boundaries. This is commonplace in modern architectures, where systems are typically designed as interconnected modules that may not share the same technology stack or hardware architecture.

Facilitating communication between disparate processes can be as straightforward as using a REST API, which is well-supported in frameworks like Spring. However, while REST is an excellent choice for open APIs, Protocol Buffers (Protobuf) offer several advantages for backend communication:

• Low Overhead and Size: Protobuf is a binary protocol with minimal payload and verbosity. For example, a type-safe pair of double values can be represented in just 4 to 8 bytes with no additional overhead.
• Transport-Agnostic: Protobuf messages can be transmitted over various communication architectures, including buses and one-to-one channels.
• Built-In API Documentation: It’s impossible to construct messages without the corresponding Interface Definition Language (IDL) files, ensuring consistent API documentation.
• Auto-Generated Objects: Protobuf supports encoding and decoding messages across major programming languages, facilitating cross-language and cross-architecture communication.

To demonstrate these benefits, I developed a simple Java Spring Boot service that periodically generates Protobuf-encoded messages and delivers them via MQTT to a Go service, which calculates a moving average.

MQTT Broker Setup

#

If you don’t have an MQTT broker in your setup, you can use the following Docker Compose configuration to quickly spin up a broker:

version: '2'

services:

  mosquitto:
    hostname: mosquitto
    image: eclipse-mosquitto
    ports:
      - 9001:9001

Protocol Buffer Setup

#

Protobuf uses Interface Definition Language (IDL) files to define message formats. From these definitions, client and server code can be generated for various languages, including C++, Java, C#, and Go. For this demo, the following simple Protobuf file is used:

syntax = "proto3";

package gophersaah.measurements;
option java_package = "ac.gophers.domain.measurements";
option go_package = "gophers.ac/domain/measurements";

message Measurement {
  uint64 timestamp = 1;
  repeated SensorReading readings = 2;
}

message SensorReading {
  string sensorName = 1;
  double value = 2;
}

This file defines a SensorReading message containing a sensor name and a temperature value. The explicit package names ensure consistent namespace alignment across languages.

To generate language-specific client code, the protoc tool is used. However, I opted for automated generation using Maven in Java and a go-generate macro in Go.

Protobuf and Java

#

Generating Java stubs from Protobuf is straightforward using a Maven plugin:

<plugin>
  <groupId>com.github.os72</groupId>
  <artifactId>protoc-jar-maven-plugin</artifactId>
  <version>3.5.0</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>run</goal>
      </goals>
      <configuration>
        <protocVersion>3.5.0</protocVersion>
        <includeStdTypes>true</includeStdTypes>
        <inputDirectories>
          <include>../idl</include>
        </inputDirectories>
      </configuration>
    </execution>
  </executions>
</plugin>

After running Maven, the generated code will be available in target/generated-sources, ready for use in your IDE with auto-completion support.

The Consumer

#

In Go, consuming a Protobuf message from MQTT is similarly straightforward. Using Eclipse’s Paho MQTT client and the official Protobuf implementation for Go, you can easily subscribe to a topic and define a callback function:

client.Subscribe("gophers/measurements", 0, measurementHandler)

The callback function unmarshals the Protobuf message, prints its contents, and calculates the moving average:

func measurementHandler(client MQTT.Client, message MQTT.Message) {
  measurement := &measurements.Measurement{}
  err := proto.Unmarshal(message.Payload(), measurement)

  if err != nil {
    log.Fatal("MHANDLER error=Could not decode message")
  } else {
    for _, reading := range measurement.GetReadings() {
      log.Println("sensor name=", reading.GetSensorName(), " value=", reading.GetValue())
      if value, exists := averageTemperatures[reading.GetSensorName()]; exists {
        averageTemperatures[reading.GetSensorName()] = (value + reading.GetValue()) / 2
      } else {
        averageTemperatures[reading.GetSensorName()] = reading.GetValue()
      }
    }
  }

  for sensor, value := range averageTemperatures {
    log.Println("averageTemp", " sensor="+sensor, " value=", value)
  }
}

Message Size Comparison

#

A Java test demonstrates the size efficiency of Protobuf compared to JSON serialization:

@Test
public void serializeToJson_shouldBeMoreThan0Bytes() {
  String json = new JsonFormat().printToString(measurement); 
  int length = json.length();
  System.out.println("size(message.json)=" + length);
  assertTrue(length > 0);
}

@Test
public void serializeToProtobuf_shouldBeMoreThan0Bytes() { 
  byte[] wire = measurement.toByteArray();
  int length = wire.length; 
  System.out.println("size(message.protobuf)=" + length);
  assertTrue(length > 0);
}

This results in the following output:

message.protobuf=byte[58]
size(message.protobuf)=58
message.json={"timestamp":1518356051,"readings":[{"sensorName":"Temperature-1","value":-32.159},{"sensorName":"Temperature-2","value":94.688}]}
size(message.json)=158

In this example, the Protobuf message is only 58 bytes compared to 158 bytes for the JSON equivalent. In more complex scenarios, the size difference can increase to a ratio of 1:8.

Alternatives and Conclusion

#

While Protobuf excels at efficiently encoding binary data, it’s not the only option. Apache Thrift offers similar features, while Avro eliminates the need for predefined IDL files. CBOR is another alternative.

I chose Protobuf for this demonstration due to its maturity and excellent support for both Java and Go. It’s my preferred choice for backend communication, and I’ve used it since the early 2000s. However, for public APIs, I still recommend REST, as standardization and tooling are crucial for third-party integrations.

Source Code

#

You can find the complete code and setup instructions on my GitHub profile.

This version retains the technical complexity and depth of your original article while enhancing readability and narrative flow. If you need further adjustments, feel free to ask!