JavaLand 2017

Performance Testing with Apache jMeter

Designing and implementing distributed systems, both customer-faced or just datacrunching farms, one is soon required to determine performance impacts and possible bottlenecks.

In this specific case, I wanted to gain information regarding the limits and scalability aspects of a customer-facing web application which also manages a high number of connected devices.

Why Apache jMeter

The performance testing case I was working on made me opt for jMeter in the end, for the following reasons:

  • Developed in Java, supporting plugins in Java or Beanshell. It is unlikely to have a metering case not which cannot be met with Java. In this case, Java became a killer feature, as most modules were implemented in Java, so it was possible to integrate jMeter into the given scenario without writing gluecode.
  • Distributed by design. It is unlikely for a single machine to stress the system under test (SUT) enough to gain any useable information. Testing and stressing a distributed system requires controlled distributed load generators.
  • Easy to get load generators on demand. jMeter is used by many engineers, there are a lot of services that accept jMeter recipies and play them on their machines for cash.
  • jMeter is able to take jUnit tests and their performance metrics into account, too. This makes it possible to share tests between jMeter and the main testbench.
  • jMeter brings a sophisticated GUI for testplan generation and debugging and also supports headless operation.
  • Very flexible to configure.
  • It is easy to package a jMeter node inside a docker image, so it can also run on a cloud computing provider which allows the execution of docker.
  • Many built-in graph and reporting components, data export to 3rd party analysis tools is available.
  • Open-Source, Apache Project
  • Finally: A wise man once said to me: “Long live the standards!”. jMeter can be considered as a defacto-standard swiss army knife for performance testing.

Functions and Vocabulary

jMeter uses a set of functional units for performing tests. After learning the vocabulary, the system is quite straightforward. The table below gives an overview on the jMeter modules and their purpose.

Aspect jMeter Components
Control Structures, Grouping and behavior Threads, Logic Controllers
Controlling iteration speeds and timing Timers -> Constant Timer, Random Timer, Constant Throughput timer, …
Storing Configuration and State Data Variables
Creating Load and performing actions on Systems under Test Samplers -> HTTP, Websocket, FTP, HTTP, Beanshell, Java Code, JDBC, WS, SMTP,..
Altering or extracting sampled Data before sampler execution Pre-Processors -> Regex, XPath, JSON Path, …
Altering or extracting sampled Data  after sampler execution Post-Processors -> Regex, XPath, JSON Path, …
Verifying and asserting sampled Data Assertions -> Regex, Compare, XPath,…
Obtaining Information and reporting Listeners -> Graph, File, Table, …

For more details on elements of a test plan, jMeter provides a concise documentation.

Designing a Test Plan

jMeter manages its test plan in a tree structure, which favours the XML data format used on the filesystem. So, the whole testplan meets a structure of ordered nodes with 0<n<inf children. For example, a parallel execution of a certain set of nodes would be represented by a parent node with the functionality of a thread controller, same applies on loop controllers or conditional controllers.

As an example, a login on a standard user/password form would be represented in jMeter as follows:

Analysis and Reporting

After running the testplan and listening for the metrics delivered by the samplers, jMeter compiles a set of prebuilt reports which gather a lot of information, in most cases every information required to derive the next actions. For instance, it is possible to graph the respone times, the error ratio and the response times in relation to the quantity of parallel accesses.
It is also possible to export the data into csv/xml files or use the generated reportfiles for further analysis. An interesting approach is to pass the data into R and use R’s graphing and reporting tools for further analysis.


Even though jMeter brings a really impressive GUI, it can be fully operated from the commandline. So, it is no problem to script it and, for example, integrate it into a CI/CD pipeline and let a build fail if it does not meet the performance expectations.


In a distributed jMeter installation, workers are called “servers” and masters “clients”. Both are connected via old-fashioned Java RMI, so, after setting up an appropiate communication foundation between servers and client(s), triggering a load/performance testing job on the master suffices to start the slaves and collect their metrics.

Test plan creation

The jMeter files (.jmx) are pure XML, so it is theoretically possible to write them manually or, more probably, generate programatically. In most cases, one would use the GUI to click a test and customize it with config files, environment variables or own tiny scripting, depending on the system under test.

Plugin development

If jMeter does not deliver a functionality out of the box, it is possible to add the functionality by scripting or plugins. This means, it is possible to divide any implementation of a Sampler, Pre/Postprocessor, Assertion or Listener into three classes:

Class A: Available out of the box
Class B: Possible by jMeter/Beanshell scripting
Class C: Only possible by developing an own plugin

Developing a Java Sampler

A test-case classified as C needs to be implemented as a plugin. Basically, every aspect of jMeter can be delegated to a Java plugin, so it would also be possible to use a Java class to implement a custom assertion. Nevertheless, I think that the most common case is implementing a custom sampler to wrap a test around a functionality which either is not available through a public API or has asynchronity/concurrency requirements jMeter itself cannot meet.

An easy way of implementing a custom sampler is fetching the dependencies via maven and providing an implementation to the jMeter Sampler API.

A very minimal Maven POM just includes a dependency expression to the ApacheJMeter_java artifact, in a real use-case one might want to add maven-assembly to create a fat bundle including all further dependencies, so a downloadable package can be built on a buildserver.


The JavaRequest sampler main class needs to extend the AbstractJavaSamplerClient class and provide a tree (1 <= n <= inf) of SampleResults:

After deploying the build artifact to $JMETER/lib/ext and restarting jMeter, it is available and can be integrated into a testplan using the GUI.

Dockerizing jMeter

An easy way to deploy a distributed jMeter installation is providing a docker set which consists of a master and n slaves. In this setup, it is advisable to create a base image and derive both master and server. If the setup is supposed to run in distributed environments which do not neccessarily provide a registry, it is advisable to use an own lightweight mechanism, such as a list in Redis which is populated by the slaves as soon as they get invoked. In bigger setups, it surely is advisable to switch to another discovery mechanism or integrate the jMeter Appliance into a Docker Swarm/Kubernetes cluster.

The base image (docker-built to “m9d/jmeter-base” here)


The master:

The server:

Performance Testing (or: Turrican II vs Github)

This is an intruductionary post on a series of articles on application performance.

In the not-so-old times of computing, dealing with performance was, in most cases, trivial in terms of the primary approach. A developer was required to optimize an application for acceptable or competitive performance on the given hardware or a small local network for one or a small number of users. This time gave us a lot of digital art that pushed the limits of popular home computers so far that it was unbelievable how far they could go with such limited hardware power.

Regardless of the glory, the scenario above is unlikely to be applicable on todays mainstream application scenarios and could even be considered as less challenging as todays hardware-abstracted systems. Nowadays, hardcore hardware programming is limited to the embedded or special applications field, performance tweaking primarily focuses on gaining the most from a limited hardware, building the solid foundation for distributed systems. In as good as no accounts, however, performance observations are limited to one single machine only. Google or Github are running on a large array of machines and services with a lot of parameters and their permutations to take into account – we are talking about distributed systems.

Looking back on the 90s, there also was another concept of scaling – one additional player of Turrican II meant buying another C64 or Amiga and a new copy of the game – literally the relation between users and “servers” was 1:1, what is straight in opposition to the “cloud” and visualization virtue we are pursuing today.

Another relevant parameter to take into consideration is agile development. Back in 1992, every piece of software was built as a single artifact and shipped on one floppy disc which remained unchanged throughout the complete shipment (not considering various unofficial “forks” which were much more liberate in terms of copy protection) and the whole lifespan of the product, which, thanks to emulators and lovers of computer game history, still lasts on. Again, this does not relate on todays work, which is built in multiple small artifacts, distributed on multiple machines, and likely to change and possibly evolve many times a day.

So, comparing a computer gaming masterpiece of the 90s with todays distributed high-scale applications reveals that both of them still are masterpieces, but in a very different way.

Performance matters

As described above, todays business is seldom centered around developing games for a hardware replicated million times. Consumer systems are heterogeneous, such are servers and applications. This raises the following questions in terms of performance:

  • Does the update we just promoted to the beta/acceptance/productive system impact the overall performance in any way?
  • Does the new version cause side-effects on other components of the system?
  • I know Amdahl and Gustavson take no prisoners, but how far can I scale with my current architecture? Can I meet my performance peak goals by showing my credit card to the hoster of my choice or is my architecture likely to get stuck in the tar at a certain point?
  • Will I be able to react to load changes, such as a black friday sale or my hobby project mentioned on a mayor news site?
  • Can I fulfill a business case on final approach or have I got to take additional measures to deliver a working system?
  • Anyway, what is performance? Which metric determines whether my application is acceptable and delivers value or not?

In the further articles in the performance testing series, I will deal with various aspects of performance and related tools.


Turning a spare Dreamplug mini-PC into a remote-controllable audio player

My company gave away a couple of abandoned spare devices previously used for evaluation purposes. So, I got hands on a dreamplug, a socket-plugin sized mini computer (ARM based) with decent power, low energy consumption, integrated dual Ethernet, WiFi, Bluetooth and, most important, an optical audio output – in other words: a Raspberry PI with a housing and the IO I was missing. The digital output was the primary reason for using it as an addon to my stereo.

My stereo was lacking the capability to play mp3/mp4/aac/lossless files without hassle, which means playing music without having to perform any manual action after importing a new CD I bought. I was not satisfied with my current solution, which consisted of a TerraTec NOXON2audio with an attached USB pen drive, as unfortunately it never ran perfectly stable with USB drives with a capacity greater than 64GB. When we developed the device at TerraTec, even 8GB USB drives were beyond “are enough for everybody/anybody wants to pay” capacity.

My other option was using an empty C.A.R. 4000 housing from the 2000s, fit a Raspberry PI and a USB DAC (or PI Audio) inside, hook up the buttons and the display to the GPIOs and build a beautiful HiFi size player. Even if this approach was very tempting, a couple of circumstances made the expected bill of material too high to take the retrofit project into consideration. First, the 3″ display. The original CAR4000 uses a parallel powertip display, which takes a lot of GPIO space and requires me to develop a new driver from scratch. New 3″ displays are too expensive or take too much GPIOs aswell, so I would have to use shift registers for connecting the control buttons, which also are originally designed to be connected to the controller via ADC – the Raspberry PI does not have an ADC, so I would have had to add an ADC, modify the board or use fancy timing code. None of them was a satisfying option, so I suspended the project.

Simply exchanging any of the gear in my living room was the least acceptable version. My oldschool AVR, built in France in 1999, may not have any retail value anymore, but such a great performance that it simply would not make sense to replace it.

So, using the dreamplug as a hidden addon perfectly was the most viable and promising option in the situation described above.

Use case

The amplifier offers both switched and unswitched power outlets on the rear, I plugged the Dreamplug to the unswitched one and connected the digital output to the digital in of the amp, the 3.5mm jack should go to my multiroom speaker system.  This makes the entire solution feel like an upgrade for the unit, not an additional device, and I came to the opinion that reducing the numbers of units in my stereo is not a bad thing.

Software stack

Originally, I wanted to build a new Hifi size unit with a display, buttons, incremental encoders and more fancy things. As I have a lot of old tablet computers and mobile phones, it became more attractive to relocate the user interface to an external unit.

Regarding the usecase, music player daemon (MPD) was the fitting solution: It offers a network interface for remote control, a media library, and various input and output plugins. Unlike common clones of platforms such as Google Music, such as Ampache or Subsonic, MPDs primary concern is using the audio output facilities of the machine it is installed on. This results into the following software stack on the device:

  • Debian Jessie Linux
  • Libertas WiFi drivers for the Marvell WiFI chipset
  • Pulseaudio
  • MPD
  • Alsatools for mixer controls
  • sshd for maintainance and file transfer
  • rsync for music library synchronization
  • udev for automounting
  • Golang runtime for a web service that switches the 3.5mm jack on and off to control multiroom

The dreamplug

Unlike the Raspberry pi, the unit does not provide a HDMI output, the only way to get a console is using a UART port. The device has a 4-pin mini connector, the pinout is:

1     GND
2     RX
3     TX
4     3.3V

I used a fan connector from an old VGA card, connected it to a 5V->3.3V serial level converter and used a terminal with the settings 115k/8N1. From there, it was possible to interact with the u-boot bootloader and write a Debian Jessie image to the flash.

Music Player Deaemon (MPd)

Just a few changes to the configurations are neccessary:

To mount the pen drive automatically, udev comes into play:

Mobile Phone control

Among many options, I choose M.A.L.P (for Android) because of its featureset, the decent UI and the fact that it is open source and free of ads. M.A.L.P requires an initial configuration consisting of the server address and the password. Afterwards, an old spare tablet computer became the control unit for the media player.


My home has got ceiling-mounted speakers in all rooms, with a central terminal which I can connect to an external audio source. My amplifier offers a Zone2 functionality, unfortunately it does not work when the main speakers are connected via bi-wiring/bi-amping, which is the case in my setup. So, I needed another solution. In case of the dreamplug, two facts made the decision on the solution very easy:

  • The analogue 3.5mm jack is capable of driving passive speakers with an acceptable performance
  • The analogue output uses a separate Pulseaudio mixer control, so I can use the digital out to the amp and the analogue multiroom out independently

The only challenge left was turning the analogue out on and off. Simply pulling the cable was not acceptable. So I decided on writing a slim golang webservice which provided an endpoint for setting the analog output volume and an Angular2 app to provide a simple user interface.

The golang webservice

Thanks to systemd, the inclusion of the service into the system start process is easy:

The Angular2 application

The Angular2 application just provides a frontend for the webservice, which consists of two simple on/off buttons.




After installing the Angular2 application, serving it with Nginx, proxy_passing the /api/v1 url segments to the golang service, the device presented a welcome page with the option to turn multiroom on and off.


Podcasting with MPD

Although MPD can archive much, it has no native support for Podcasts, but this functionality can easily by added by taking advantage of the playlisting feature. Here, I created a simple Golang task that retrieves a podcast (using the awesome Podfeed Go library) and converts it into a m3u playlist. This service is triggered by cron.hourly and piped to a file named according to the podcast title inside the MPD playlist folder.

Please find the source for podcast2m3u on Github: [podcast2m3u]

Somewhere in Northern Germany

Message Queue Telemetry Transport (MQTT)

After various encounters and two projects which employed MQTT, I thought it was a good idea to write a summary about my experiences and my point of view. As my new job is very close to the IoT and Industry 4.0 sector, it did not take much time to encounter the hype.

Once upon a time

MQTT is, like AMQP, a messaging protocol for distributed systems, meaning that a node can pass a message to one or multiple other nodes. Having used RabbitMQ and AMQP for years, this principle is nothing seriously new to me, and especially as a RMQ user, a core point of MQTT does not sound very exciting at the first glance – its simplicity. In AMQP, I have a whole bunch of functionality to get messages from A to B, including advanced routing and the, pardon me, awesome exchange functionality. MQTT lacks all of them – in exchange for a very little basic footprint, which makes it very tempting for embedded systems and little devices (such as sensors…) which do not possess much memory.

In the basic version of MQTT, developed in the late 1990s, the functionality was just publishing a Message M from a client C to a Broker B, which would dispatch it to all Clients S0..Sn which expressed interest in the type of message. A message M was defined as {Topic T, Payload P, QOS Q}, both parts of the tuple were represented as String. A broker B would wait for a set of Subscriptions S0..Sn, do just do a simple strcmp on any new message and dispatch the message to all clients which subscribed to the topic of a new message in the first place. Using a simple QOS mechanism, the publisher of a message was able to specify the way the Broker should deal with messages on a per-message basis: fire-and-forget (and do not care), deliver to at least one client, deliver to >1 clients.

On the transport side, MQTT could be sent through plain wire (such as a RS-232 / RS-485 line or even teletype), or via TCP/IP, of course unencryted and unauthorized.

So, the protocol was plain simple and a real minimalistic approach to setting up a message exchange, by no ways applicable to our needs in the year 2016 and should be forgotten, right?


My first applications with MQTT

My first experience with MQTT was in 2010, when I used it as a transport protocol for sensor data over a RS-485 line.

The project was a greenfield project, but the controller in the sensor was very limited, and my Atmel AtTiny had much more to care about than parsing, verifying and deserializing fancy XML data even for the transport layer. Using full-blown frameworks such as Apache Thrift was not an option aswell, but MQTT was deployed very fast. Using topic names “config.sensitivity”, “config.sensorType”, “sensor[0].value” and so on made the evaluation of data very easy, not to say trivial. On a 2-pin cable inside a machine cabinet, I also did not care very much about security, so it just made my live easier and saved me some hours of work to implement my own protocol. Cool, job done.

Then came the second, which is mainly used for telemetry and event exchange with other devices, using more advanced features of todays MQTT implementations:

  • Server and client certificates, automatically rolled out during device production, respectively orchestration of the broker server – which means: Authentication by certificates. Of course MQTT also supports plain username/passwords.
  • Hierarchical topics. Topics are not only strings anymore – following the convention of naming them in url-pattern style, such as devices/<id>/events/updates/caseTemperature, it is possible to perform registrations on a hierarchical basis. For example, a subscriber registered for devices/<id>/* would also receive messages published to sub-topics such as the above.
  • Which brings us to Access control lists, supported by a subset of the mainstream brokers. For example, why should a sensor be able to subscribe to its own readings? Or, more relevant, why shouldn’t one share a MQTT broker with other clients, as long as they are not likely to interfere?
  • Distributed, scalable brokers.

Which means, one could even use MQTT on the internet nowadays without being fired.



On the client side, I mainly used Eclipse Paho. After writing a Autocloseable Fluent-API facade for it, it was quite pleasant to work with.

Examples (Java)

Connect to a Broker

Or with SSL:

Publishing a message:

Simple publish-receive test:



On the broker side, I used Moquette (easily embeddable into own Java applications) and Mosquitto. As there are excellent documentations on both, I will not go into details here.


We have been through many approaches to make computers talk to each other. We have been through the SOAP-Ceremony, the CORBA-Hell, and now we even manage to write beautiful RESTful webservices. We also laughed about Windows NT messaging. Personally, I like to model communication architectures using RabbitMQ, but I also have to confess that often a simple messaging protocol without any advanced features would have done the job.

On the other hand, I do not think much of hypes and consultants who try to make a fortune on selling the next best thing. So, I will just add MQTT to my toolbox, be happy with the current exciting project which employs MQTT for M2M and telemetry, and, as usual, choose the right tool for the job, with or without a fancy title, next time again. As MQTT will gain more use-cases just because of simplicity, I do not think this will be our last encounter.

Etretat revisited


Test-Driving the Go Language

When I completed my compiler design studies, I felt it was not appropriate for me to give my exercise projects any fancy names and put them on Github. With this in mind, I was used to question any new language which came out of nowhere and wait for a number of readings on my technology radar before considering to deal with it.

With a number of significant readings from Google Go, I decided to give it a pleasant test drive.

The programming language Go, originally a 20%-Project initiated by a Google employee, quickly found a position in my mind – a slick alternative to C/C++ for smaller systems and fast backend services.  Being used to Java/Spring Boot and .net/Nancy, Go could be interesting to me if there is any gap it could fill – maybe a gap I was yet to discover.

To find out, I built a simple echo service which covers some aspects I would usually face on backend services:

  • HTTP Server
  • JSON
  • Commandline Arguments or Environment Variables
  • Logging (ELK friendly, please)
  • Easy to integrate in CI/CD environments

Setting up Go

Setting up go takes 5 minutes on UMTS or 4 hours on a crappy hotel WiFi. Download the package, install, create your workspace and start coding. I used a text editor for writing Go code, afterwards I noticed that there is an awesome set of plugins for Atom.

Right after creating the workspace one thing becomes obvious – Go loves conventions, convention over configuration and implicit declarations. The workspace is organized in a way that seems comfortable to Java developers, because it works perfectly if you organize all your projects and external dependencies in the form of Java namespaces.

Java developers will, on the other hand, not be comfortable with the missing concept of classes and the implicit private/public declarations: function names in camelCase are private, while their PascalCased counterparts are public.

The Echo Service

Lets build a simple Echo Service, which just listens to HTTP requests and echoes them back, while writing comprehensive logs and following a set of practices which appear a good idea in Java.

Enter Go..

Main file


Service Handler



or, to build to GoHome/bin:

Go generates fat binaries which include all their dependencies, which means that deployment is a little bit easier, regardless of the bigger files.

Test Run



I needed about 3 hours to learn the basic concepts of go, get my workspace running and develop the trivial service above. Not too far away from Spring Boot when leaving the Java language out of consideration. To me, Go looks more than promising. I will definitively consider it as an alternative to C on my next project which requires a small and slick microservice backend for tasks I can quickly replace with an alternative implementation.