Automating JMeter tests with Maven and Jenkins

In this post, I will show how to integrate the popular load testing tool JMeter in a Maven build. The goal is to allow every developer to easily develop and execute JMeter tests on his local machine. There will be no need to learn a new commandline utility or to install separate tools. Everything will be possible via some simple maven goals. With this setup, it is also easy to create a Jenkins job, that allows execution of JMeter tests against a testing environment with a single click.

Integrating JMeter with Maven

There are two plugins, that allow integration of JMeter tests in a Maven build: jmeter-maven-plugin and chronos-maven-plugin. Which plugin to choose, depends on the requirements. For me, the following points were important:

Points 1) to 3) are satisfied by both plugins. The reporting is always a matter of taste. The chronos plugin comes with an additional reporting plugin, the chronos-report-maven-plugin, that embeds test results in a Maven report. There is also a jmeter-analysis-maven-plugin , which creates a simple html report with graphs showing response times over time. What finally motivated my decision to go for the jmeter-maven-plugin, was the last point. With the jmeter-maven-plugin, a simple additional dependency to kg.apc:jmeter-plugins allowed using the JMeter Plugins in GUI as well as in headless mode. I couldn’t find a similar possibility in the jmeter-chronos-maven-plugin.

The following code- and configuration-snippets are taken from the example project jmeter-maven-example , which is available on GitHub. There is also a public jenkins with a simple job, that allows executing JMeter tests against a real application (hosted on CloudBees).

Because the public jenkins hibernates after 15 days, I also added a docker build for a preconfigured jenkins to the github repo. Just see the README.md for an explanation how to build and start the jenkins docker image.

The configuration of this jenkins job is explained in the following sections. But first, the necessary Maven configuration:

1<plugin>
2  <groupId>com.lazerycode.jmeter</groupId>
3  <artifactId>jmeter-maven-plugin</artifactId>
4  <version>1.8.1</version>
5  <configuration>
6    <!--
7       By default the test results are saved in a file  
8       /target/jmeter/results/<testname>-<timestamp>.jtl
9       Further processing is easier without timestamp though.
10    -->
11    <testResultsTimestamp>false</testResultsTimestamp>
12 
13    <!--
14       To simplify debugging, it is advisable to adapt the loglevel.
15       The jmeter logs are written to the file jmeter.log.
16    -->
17    <overrideRootLogLevel>DEBUG</overrideRootLogLevel>
18 
19    <!--
20       By default, the console output during a jmeter test run is suppressed.
21       We want to display the progress using the listener "Generate Summary Results" 
22       (which periodically prints stats to stdout). Therefore we 
23have to make sure,
24       that the jmeter output is not suppressed.
25    -->
26    <suppressJMeterOutput>false</suppressJMeterOutput>
27 
28    <!--
29       If tests fail (e.g. a http-request running into a timeout), the corresponding maven
30       goal also fails (and subsequent goals aren't executed anymore). We want to create graphs
31       from test-results, no matter if some requests failed or not, so we ignore jmeter failures.
32    -->
33    <ignoreResultFailures>true</ignoreResultFailures>
34  </configuration>
35  <dependencies>
36    <dependency>
37      <groupId>kg.apc</groupId>
38      <artifactId>jmeter-plugins</artifactId>
39      <version>1.0.0</version>
40      <exclusions>
41         <!--
42            Unfortunately some transitive dependencies cannot be found on mvncentral
43
44   and we have to explicitly exclude them.
45            For a complete list, see https://github.com/mlex/jmeter-maven-example/
46        -->
47        <exclusion>
48            <groupId>kg.apc</groupId>
49            <artifactId>perfmon</artifactId>
50        </exclusion>
51        <!-- ... -->
52 
53        <!--
54            Because of a bug in the jmeter-maven-plugin (see 
55            https://github.com/Ronnie76er/jmeter-maven-plugin/issues/77) we have to
56            exclude jmeter dependencies here, too.
57        -->
58        <exclusion>
59            <groupId>org.apache.jmeter</groupId>
60            <artifactId>jorphan</artifactId>
61        </exclusion>
62        <!-- ... -->
63      </exclusions>
64    </dependency>
65  </dependencies>
66</plugin>

The tests must be placed in the folder /src/test/jmeter. The JMeter GUI can be started with mvn jmeter:gui. There you can edit and execute tests. With mvn jmeter:jmeter is it possible to execute tests without GUI. In headless mode the JMeter listener Generate Summary Results is great to display progress information (via simple console output).

The tests are most likely executed in various envorinments. It is good practice to have a central point, where you can store the configurations for the different environments. The easiest way is to use Maven properties and profiles. The example project defines two different Maven profiles, one for local execution and one for execution from jenkins. The Maven properties are passed to JMeter via the userProperties option. Inside a JMeter test, you can then access the properties using the function ${__P(propertyName)}.

1<profiles>
2  <profile>
3    <id>local</id>
4    <properties>
5      <performancetest.webservice.host>localhost</performancetest.webservice.host>
6      <performancetest.webservice.port>8080</performancetest.webservice.port>
7    </properties>
8  </profile>
9  <profile>
10    <id>jenkins</id>
11    <properties>
12      <performancetest.webservice.host>my.test.system</performancetest.webservice.host>
13      <performancetest.webservice.port>80</performancetest.webservice.port>
14    </properties>
15  </profile>
16  <build>
17    <plugins>
18      <plugin>
19  
20      <groupId>com.lazerycode.jmeter</groupId>
21        <artifactId>jmeter-maven-plugin</artifactId>
22        <version>1.8.1</version>
23        <configuration>
24          <propertiesUser>
25            <webservice.host>${performancetest.webservice.host}</webservice.host>
26            <webservice.port>${performancetest.webservice.port}</webservice.port>
27          </propertiesUser>
28        </configuration>
29      </plugin>
30    </plugins>
31  </build>
32</plugin>

Executung JMeter tests with Jenkins-CI

Now that we have different Maven profiles for the various environments, the configuration of an appropriate Jenkins job is just a small step. The parameterized builds are another nice feature of Jenkins, that is very useful for JMeter tests. Parameters can be used for example, to allow users to define the size of the load test (that is: the number of threads and iterations).

Reporting

What meaningful reporting looks like, always depends on the concrete requirements. Sometimes it makes sense to regularly run equally sized load tests, so that you get instant feedback whenever a change in the system considerably effects performance. In this case, the Jenkins Performance Plugin is a good choice. However, I was more interested in graphs that show the system behaviour over time. Those graphs can be easily created from command line using the CMDRunner tool of JMeter Plugins. With the help of a small Maven plugin, the jmeter-graph-maven-plugin , it is possible to easily integrate this graph-generation in a Maven build.

Result

Just let the images speak for themselves. Starting the job with one click, …

… defining test properties (like number of threads), …

… running the test …

… and finally displaying the test results couldn’t be much easier.