From a Mathematician’s Point of View: JMeter – Beloved Crap Tool

My company, codecentric AG, has a strong disposition towards open source tools and solutions. Therefore, it is quite natural that for me as a Java performance specialist, the load testing tool of choice is Apache’s JMeter .

Let me just state something right at the beginnig of this blog post: This is no comprehensive overview over JMeter, its features and limitations. Instead, this is a very personal view on this tool. A settlement of an overdue account, if you want to call it so.

After some years in which I was mainly involved in performance optimizing and performance monitoring, I was working with JMeter in my latest customer project again. This time, JMeters JUnit-sampler was the main interface for load generation whereas before I was mainly working with the HTTP sampler.

In principle, to write and execute load tests is an easy task with JMeter. When – as in our recent project – you want to use already existing functional JUnit tests to generate load, you just have to provide these JUnit tests as a jar-file to JMeter. After this you use JMeter’s JUnit sampler to choose the corresponding test class and test method. A JMeter thread group serves as bracket around the JUnit samplers. Within the thread group you configure the load (number of parallel threads) and the duration of the test. By and large, this is it. Press “run” and execute your test, called a “testplan” in JMeter. If you want more details, the user manual will provide plenty of them in easily understandable form. It is lacking some explicit examples especially in the regex section, but all in all it is an excellent documentation.

Somehow I had forgotten about the typical feeling of working with JMeter. But JMeter brought back this feeling very, very fast. What I had forgotten was that JMeter – as any other human being depending on its form of the day – can and does behave really moody, sometimes even bitchy.

Want some examples?

• The error-messages of the JUnit sampler are quite expressive – normally. But beware of any discrepancy to what JMeter expects of the constructors of the JUnit test class. You just get a scant “failed to create an instance of the class” message, then JMeter snaps shut and leaves you alone in your misery. Neither JMeter (nor Google in this case) will help you at all. Well, maybe this can help: JMeter seems to expect the standard constructor as well as a constructor with a single String argument. Additionally, if any libraries are used within the constructors (e.g. jlog), JMeter needs them, too.
• Completely incomprehensible behavior happened when I used JMeter’s “Regular Expression Extractor” to extract variables from the response of the last sampler. The content of these variables was as fleeting as a bubble in a thunderstorm. I was able to use it in the very next sampler. The content was already gone with the wind in the next sampler thereafter, It also was not transferable to other variables or structures. I tried a whole bunch of different workarounds, none was working.
• If a JMeter testplan contains several JUnit samplers, JMeter adds suspense to your work by sometimes changing the test classes and test methods of the other samplers to the class and method of the current JUnit sampler. I still have no idea what triggers this particular bug.

Pretty features like these help to keep your awareness up while working with JMeter. In other words: I developed some sort of paranoia, never trusting JMeter to do what it ought to do. They also have the side effect of making JMeter the default suspect if something goes wrong.

Some other points are merely impractical, e.g. directory paths. JMeter has to read and write many files: testplans, test parameters, result files. The default directories for these files are different, though. Sometimes it is the same directory as the testplan, sometimes the JMeter-bin directory, with no apparent logic behind it. Also, when running tests through the JMeter GUI instead of scripts or the command line, one often encounters the urge to restart the GUI for every test run. This is necessary to ensure that JMeter does not remember anything from the last test and also to restart the JVM in a typical JUnit test scenario, where everything is running in just the JMeter-JVM: JMeter, the JUnit test classes and sometimes even the Java system under test.

O.k. – so JMeter is a crap tool. But why do I love it, nevertheless? Well, JMeter is dependent on next to nothing, is installed easily, open source and free of charge. And on top of all that it offers an astounding flexibility. The HTTP sampler is probably the most widely used, as it is best suited to test web applications from the user’s point of view. Other samplers like the JUnit sampler described in this blog operate at a much lower, technical level. At this time, there are about twenty different samplers in JMeter for a lot of various protocols.

Another useful little bonus is the so called performance plugin for Jenkins, which is de facto a JMeter plugin. At codecentric, we want to automate as much as possible, performance tests not excepted. With the performance plugin, JMeter is suitable even for that.

And if all this is not enough for your taste, you can directly hack the JMeter testplan (which is just an xml-file) – but beware of JMeter’s moods: not all seemingly obvious modifications are accepted – or even write your own add-on code.

JMeter reminds me of something a friend told me about his first car – a little French one. “I always wanted a car with a character. And this car has a character: A mean one, indeed, but it has character!” He loved his car.

All in all, will I use JMeter again? Most definitively. JMeter helps you to get the job done, trains your stress resistance potential and finally: it teaches humbleness.