There are plenty of open-source tools available that are supporting the automation of acceptance tests. This also results in a lot of different concepts to do this. For example the Robot Framework is using a keyword-driven approach and offers a lot of tooling and ready-made test libraries. FitNesse on the other hand is organising the tests in an own Wiki and offers the possibility to start tests from this Wiki right away. Other tools like JBehave are going into a different direction and focus very much on a lean approach and are using BDD-format very consequently for describing the tests. The test implementation is then done entirely in Java. We have taken a closer look to this approach already here .
When discussing agile testing tools Concordion is almost always mentioned sooner or later. And often it is mentioned in a very positive manner. On the other hand I have not yet found anyone using it productively in any project. Thus I thought it is time to have a closer look to know what people are talking about :-).
The test implementation for Concordion happens typically in Java. But functionality has already been ported to .NET, Ruby, und Python. To make my life a little bit easier we will take a look at the Java version in the following.
As this article is quite long here comes a short overview on the topics presented in the following:
- Download and “Hello World”-Example: Explains the installation and how to execute the sample project provided with the download.
- Test description and test implementation: After executing the example let’s try to really understand Concordion and implement some tests of our own.
- Reporting: How well are the testing results presented by Concordion?
- Integration into a CI environment: How to integrate tests written with Concordion into some environment for Continuous Integration.
- Conclusion: How suitable is Concordion for automating Acceptance Tests? And for whom?
Download and “Hello World”-Example
Obviously the first step is to download Concordion :-). this can be either done from the download page or from the Getting Started page . Concordion comes along as a ZIP-file that contains an Eclipse-Project. All required libraries are included as well as some small sample project to demonstrate the basic concepts. This article is based on concordion-kickstart-1.4.2.zip.
To import the project in Eclipse I create a new workspace into which I then import the Concordion directory from the ZIP-file as a new project. Afterwards you have the following result in Eclipse:
On first sight I notice the Ant and Gradle build scripts. I am for sure not a big Maven -fan, but somehow it is a standard, isn’t it? But let’s put that aside for the time being, as the first test can be simply executed from Eclipse as a Junit test. To do this just excecute the “HelloWorldTest.java” located under “com.example.specs.greeting” in the “specs”-directory as a JUnit test. That works without any problems and it is printed to the console where to find the report with the testing results.
C:\DOKUME~1\tja\LOKALE~1\Temp\concordion\com\example\specs\greeting\HelloWorld.html
Successes: 1, Failures: 0
The generated report that looks as follows:
This looks very good so far. But how about implementing some tests of our own?
Test description and test implementation
All agile testing tools are seperating the test description from the test implementation. Concordion is using XHTML as the input format for the test description. In writing this test description the user has a lot of freedom and is not bound to any one format like tabular or BDD-style. So let’s try to write a test for the Java Stack implementation . This way we gain very good comparability with our article on JBehave, where the same example was used .
The syntax for describing the tests is easy to learn and the following test description is quickly created. There are some commands that are embedded into the HTML-code to set variables, execute methods and assert expected results. In Concordion this is called instrumentation. As the class we want to test here is a Java API class and thus already existing we only have to create a new package “com.example.specs.stack” below “specs”. Inside this package we now create our test description called “Stack.html” as well as the corresponding test implementation in the class “StackTest.java”. Please notice the coupling of these two files based on some naming conventions. The HTML file must have the same name as the Java class, but without “Test” and of course with “.html” instead of “.java”.
To be able to compare this example very well with the one used in the corresponding JBehave article, the test description is written in BDD-syntax. Please note that this is not required and that the syntax to be used offers a lot of freedom. In the end what is relevant for the test execution are the commands that are embedded. This results in the following test description:
1<html xmlns:concordion="http://www.concordion.org/2007/concordion"> 2<link href="../concordion.css" rel="stylesheet" type="text/css" /> 3<body> 4 5 <h1>Stack Test</h1> 6 7 <p>Testing the basic functionality of the Java Stack implementation.</p> 8 9 <div class="example"> 10 <h3>Basic functionality of a Stack</h3> 11 <p> 12 Given an empty Stack<br/> 13 <span concordion:execute="initializeStack()" /> 14 When the string <b concordion:set="#firstString">Java</b> is added<br/> 15 <span concordion:execute="addElement(#firstString)" /> 16 And the string <b concordion:set="#secondString">C++</b> is added<br/> 17 <span concordion:execute="addElement(#secondString)" /> 18 And the last element is removed again<br/> 19 <span concordion:execute="removeLastElement()" /> 20 Then the resulting element should be 21 <b concordion:assertEquals="stackResult()">Java</b> 22 </p> 23 </div> 24 25 <div class="example"> 26 <h3>Stack search</h3> 27 <p> 28 Given an empty Stack<br/> 29 <span concordion:execute="initializeStack()" /> 30 When the string <b concordion:set="#firstString">Java</b> is added<br/> 31 <span concordion:execute="addElement(#firstString)" /> 32 And the string <b concordion:set="#secondString">C++</b> is added<br/> 33 <span concordion:execute="addElement(#secondString)" /> 34 And the string <b concordion:set="#thirdString">PHP</b> is added<br/> 35 <span concordion:execute="addElement(#secondString)" /> 36 And the element Java is searched for<br/> 37 Then the resulting element should be 38 <b concordion:assertEquals="stackSearch(#firstString)">3</b> 39 </p> 40 </div> 41 42</body> 43</html>
The corresponding Java class is trivial. It has to implement the instrumentalisation done in the HTML file, which are defined by the expressions: concordion:execute and concordion:assertEquals. This results in the following class:
1package com.example.specs.stack;
2
3import java.util.Stack;
4import org.concordion.integration.junit3.ConcordionTestCase;
5
6public class StackTest extends ConcordionTestCase {
7
8 private Stack<String> testStack;
9
10 public void initializeStack() {
11 testStack = new Stack<String>();
12 }
13
14 public void addElement(String element) {
15 testStack.push(element);
16 }
17
18 public void removeLastElement() {
19 testStack.pop();
20 }
21
22 public String stackResult() {
23 return testStack.pop();
24 }
25
26 public int stackSearch(String element) {
27 return testStack.search(element);
28 }
29}
It is very nice that no further classes are required here to glue things together. All required functionality to run the tests is inherited from the class “ConcordionTestCase” and is thus transparent for the developer.
Reporting
The generated report for a test run looks very much like the test description itself. Thus it is up to the developer of the test description how well this report looks like and how easy it is to read the tests. On the other hand there is really not much additional functionality or information offered. Our example results in the following report:
Those values that are checked using assertions are marked green in case of success. Otherwise the – not expected – values are marked red.
Integration into a CI environment
Maybe I have overlooked something here, but on the Concordion web pages I was not really able to find any explanation how to execute the tests from the command line. I guess that this can be done using the Ant or Gradle scripts mentioned above.
As my experience using Gradle is even smaller than my experiences using Ant I try the latter one. And luckily everything works out without any problems. After opening the build.xml build file you can use the Eclipse Outline-View to select the “run tests” task. This one is then executing the tests and writes the results to the workspace under “build/concordion-output”.
This is the output that is generated by Ant. To implement a corresponding Maven configuration is something to keep in mind for a lter blog post :-).
1Buildfile: C:\workspace\concordion\concordion-kickstart\build.xml 2compile: 3 [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\classes 4 [javac] Compiling 3 source files to C:\workspace\concordion\concordion-kickstart\build\classes 5 [copy] Copying 6 files to C:\workspace\concordion\concordion-kickstart\build\classes 6run.tests: 7 [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\test-output 8 [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\concordion-output 9 [junit] Running com.example.specs.greeting.HelloWorldTest 10 [junit] C:\workspace\concordion\concordion-kickstart\build\concordion-output\com\example\specs\greeting\HelloWorld.html 11 [junit] Successes: 1, Failures: 0 12 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0,234 sec 13 [junit] Running com.example.specs.stack.StackTest 14 [junit] C:\workspace\concordion\concordion-kickstart\build\concordion-output\com\example\specs\stack\Stack.html 15 [junit] Successes: 2, Failures: 0 16 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0,031 sec 17BUILD SUCCESSFUL 18Total time: 3 seconds
Conclusion
Getting started with Concordion is quite easy for a new user. The example from the download packages works out-of-the-box and the underlying concept is easy to understand. Of course it helps here knowing already some other agile testing tools :-). It will be more difficult to really follow the concept of Concordion to write specifications and not scripts. The possibility to do this is definitly a strength of this tool.
When writing the specification the user has a lot of freedom and instrumenting the specification is straightforward. Overall Concordion is a lean tool and to some extend quite comparable with JBehave. Executing tests in some Continuous Integration environment is also possible without problems. The reporting on the other hand is minimalistic. But as usual everything depends on ones requirements here.
Of course there is more to explore, but hopefully this article offers a good starting point to start test automation using Concordion. Getting started is easy and there are hardly any frustrating moments working with it. Everything works as expected. Thus if you are evaluating agile testing tool Concordion is worth a glance.
Was this post helpful?
More articles
fromThomas Jaspers
22.2.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Thomas Jaspers
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.