Selecting the “Opponents”
From the rather huge amount of – commercially and freely available – tools supporting agile testing I would like to have a closer look at FitNesse and the Robot Framework . Both tools are available for free and by looking at the corresponding project pages it can be seen that both frameworks are constantly enhanced. Another positive aspect is that there are mailing-lists available that can be used to get support. This is definitly helpful when using one of those tools in some real-life projects. Of course selecting these tools for a review might feel a bit random, especially because there are lot of other tools in this area. For example Concordion as a freely available alternative or Liquid as a commercial one to name only two. But as time is – unfortunetly – a limited resource I had to do some preselection here. Hopefully there will be the possibility to have a look at some of the alternatives in some later article still.
The Installation
At first glance the installation of FitNesse seems to be extremely simple. As the tool is entirely implemented in Java no additional prerequisites are needed. But the installation does not run as smoothly as expected as the proposed installation command java -jar fitnesse.jar was throwing a lot of errors (at least on my machine):
Unpacking new version of FitNesse resources. Please be patient.
java.io.FileNotFoundException: .\FitNesseRoot\files\images\executionStatus\Thumbs.db (Das System kann den angegebenen Pfad nicht finden)
Thus I decided to use an older release for the evaluation of the tool. Here the installation comes as a ZIP-file and after unpacking this one FitNesse could be taken into use without any problems.
The Robot Framework seems to be more demanding here as it requires the installation of Python and Jython first. But due to the fact that all the installers are working very well there are also no problems in installing locally under Windows XP nor in installation on a Linux server.
Attention: To be able to use the graphical Editor of the Robot Framework Jython version 2.5.0 (or higher) needs to be installed. Jython versions are installed into parallel directories and if you have installed an older version first – like I have done – you must edit the file C:\Python26\Scripts\jybot.bat and set the corresponding line to set jython=”C:\jython2.5.0\jython.bat. The paths used here are the ones of the default installation under Windows.
Thus the effort for installation – from the download to the first successful startup – is very low for both tool. It is in the range of 15 to 30 minutes. And even though there have been some problems in the installation of FitNesse first I would rate both tools as very mature with respect to this topic.
The Philosophy
On first sight both tools are following quite different approaches. FitNesse is getting started as a Web-Server running its own Wiki .
New testcases can be added by editing the main page. Those pages can then be worked on as usual in a Wiki. It is possible to execute individual tests directly from the web frontend. Of course tests can also be executed using a scripting approach.
In comparison the Robot Framework looks a bit more frugal. Testcases are written using tables in HTML-format and there is also no possibility to execute individual tests from some kind of GUI. The test execution is always started using the command line.
But with the Robot IDE there is a very good graphical editor (even though still Alpha) that eases the creation and editing of the HTML-pages representing the tests extremely.
But there exists also a very fundamental commonness of both tools. Tests are implemented as a collection of so called keywords. Both frameworks support Java as a language to implement such keywords. The Robot Framework is supporting in addition Python “natively”, but the focus of this article is writing keywords in Java. Thus the common aspects are:
- Keywords are written in Java and included as libraries by creating JAR-files for example.
- Several keywords can be combined to create an individual testcase.
- Parameters can be given to those keywords to test different scenarios.
- Tests can be executed by a script and some kind of report is generated based on the execution.
Of course both tools have different focus areas as well when it comes to the definition of testcases. For example FitNesse has its own concept to pass testdata in tabular form to the testing class. In addition it is here to some extend possible to define keywords in a more “natural language'”, by mixing method names with parameters. The Robot Framework has here a more classical approach but due to the fact that method names can have spaces in the HTML definition this is also very well readable. It also supports the easy definition of new keywords by combining already existing ones. Another focus area is the possibility to extensively document testcases and to tag those, which is then used in the generated report later on.
Testing with Selenium
To evaluate both tools using a simple example that requires no additional source code I decided to use a simple Selenium-Test on some public web pages. Thus I will search for a specific site on Google and then navigate to a specific page on that site and check for the existence of some text element there. Both frameworks bring their own libraries with them to support Selenium. (FitNesse Selenium and Robot Selenium ). But those have not really convinced me as there are too many problems and limitations and in the end it is simply much more complicated to use those libraries in comparison to the excelent Selenium IDE , which can be installed as a plug-in to Firefox.
Thus in this test I will utilize the Selenium IDE to generate a Java class with the relevant keywords (=testing methods). This is at the same time a good test to see how easy (or complicated) it is to use such Java libraries with these frameworks. The following class has been generated out of the Selenium IDE (as Java export). First the term “codecentric gmbh” is searched on Google. The codecentric homepage must then be among the results on the first page, as the corresponding link is clicked. Then we navigate to the imprint and check for the proper value of the trade register. (Of course this is a rather simple test, but should be enough for demonstrating purposes.)
1package de.codecentric.agiletesting.demo;
2
3import com.thoughtworks.selenium.*;
4import java.util.regex.Pattern;
5
6public class MyDemo extends SeleneseTestCase {
7 public void setUp() throws Exception {
8 setUp("http://www.google.de/", "*chrome");
9 }
10 public void testKeyword() throws Exception {
11 selenium.open("/");
12 selenium.type("q", "codecentric gmbh");
13 selenium.click("btnG");
14 selenium.waitForPageToLoad("30000");
15 selenium.click("//div[@id='res']/div/ol/li[1]/h3/a/em");
16 selenium.waitForPageToLoad("30000");
17 selenium.click("//div[@id='footer_innen']/p/a[1]/span");
18 selenium.waitForPageToLoad("30000");
19 verifyTrue(selenium.isTextPresent("HRB 19214"));
20 }
21}
After some small adjustments to some names and the package name this class can be used for our tests right away. This has the additional advantage that those tests can be in priciple still be executed as a JUnit test from within Eclipse. This is a bit hidden by deriving from the class SeleneseTestCase.
Preparation
This class will now be included to a small Eclipse project and then exported as SeleniumKeywordDemo.jar. To do so we need in addition the selenium-java-client-driver.jar, which can be downloaded from the Selenium homepage. Then we need the junit.jar, which is part of every Eclipse installation. I am collecting all those JARs into one directory C:\projects\codecentric\SeleniumKeywordDemo\lib and the tools can utilize these JARs from there. Of course we need to start a local Selenium-Server in addition to be able to execute those tests.
Robot Framework – Integrating and executing the tests
A good starting point writing tests with the Robot Framework is to launch the graphical editor. Using a default Windows installation this can be done as:
C:\>cd C:\Python26\Scripts
C:\Python26\Scripts>ride.py
Afterwards we are creating a new testsuite using Suite -> New Suite. Then the required libraries must be added under the top-level folder using -> New Library. It must be noted that here the full qualified class names (de.codecentric.agiletesting.demo.MyDemo) are given and not the name of any JAR-files. Those must be included to the classpath later on in the script used for the execution of the tests. In the Robot IDE one can then simply add a new test where the keywords from our Java class are getting called.
As the setUp()-method does not really implement a testcase it should be defined as the Setup-method for the testsuite instead. A click onto the following screenshot will lead to the HTML sources for the final Robot test.
Now the only thing left is the batch-file that is required to start the tests. The following script will do this job for a Windows system:
@echo off
set CLASSPATH=C:\projects\codecentric\SeleniumKeywordDemo\lib\junit.jar;C:\projects\codecentric\SeleniumKeywordDemo\lib\selenium-java-client-driver.jar;C:\projects\codecentric\SeleniumKeywordDemo\lib\SeleniumKeywordDemo.jar
C:\Python26\Scripts\jybot C:\projects\codecentric\SeleniumKeywordDemo\SeleniumDemoTest.html
The test is then started by executing this batch-file and is running through smoothly. The Firefix- and the SeleniumRC-windows are staying open when the test is finished and must be closed manually. In a real test closing those windows could be easily added to a teardown-method in the Java class.
The Robot Framework is creating a XML output file from which a very nice and complete report can be generated in HTML format. An additional HTML file is generated that contains corresponding log-information that can be used for troubleshooting.
FitNesse – Integrating and executing the tests
The usual way to add a new test in FitNesse is using the web frontend. Thus it is required that the FitNesse server is running. Choosing Edit from the homepage it is possible to add a new test there called SeleniumDemoTest and save the page again. Now this page is displayed with a “?” as a link. By clicking on that link the new page is opened for editing. Also here the testcase is implemented quite quickly even though there is one trap. The Java class must be derived from the class DoFixture that comes with FitNesse as otherwise the following error is shown “Class de.codecentric.agiletesting.demo.MyDemo is not a fixture.”. To find this out really required a bit of research work. Thus we have the following source code for the test:
!path C:\fitnesse\fitnesse.jar
!path C:\fitnesse\lib\fitlibrary.jar
!path C:\projects\codecentric\SeleniumKeywordDemo\lib\junit.jar
!path C:\projects\codecentric\SeleniumKeywordDemo\lib\SeleniumKeywordDemo.jar
!path C:\projects\codecentric\SeleniumKeywordDemo\lib\selenium-java-client-driver.jar
!|de.codecentric.agiletesting.demo.MyDemo|
|setUp|
|testKeyword|
In the Java class the fitlibrary.DoFixture must be imported and the class must be derived from DoFixture. Afterwards the test can be executed directly from the web frontend and results in the following output:
Usually tests will be executed as a part of a continuous integration environment and thus it must be possible to start them using a scripting approach. For FitNesse the following command can be used for this:
C:\fitnesse>java -cp C:\fitnesse\fitnesse.jar fitnesse.runner.TestRunner -xml report.xml localhost 8070 SeleniumDemoTest
This will also generate – a rather short – report in XML format.
Forgetting to include the fitnesse.jar JAR file into the test definition the execution of the test is hanging without any further error message. This happens when starting the test from the web and when starting it from a command line and is not too nice.
Summary
In my opinion it is not really possible to assess these tools in a “short” blog article in such a way that this results in a general recommendation. Due to the complexitiy of the tools and the whole topic “Agile Testing” it would be easily possible to write an own book for each of the frameworks. In addition I have been gained some experience with the Robot Framework in earlier projects before this evaluation which would make such a judgement a bit unfair in addition. But nevertheless I would like to name a few pros and cons I found from evaluating the tools.
One strength of FitNesse is for sure the specific format to enter testdata in a tabular format and the possibility to describe tests to some extend using “colloquial phrases”. The Robot Framework is not having that many possibilities here but is on the other hand not too far away from this. Starting tests from the web frontend is on first sight another advantage for FitNesse. But one has to check in detail here if this is really meaningful and possible in the concrete use case. Potential problems here are dependencies between tests and also the problem that some parameters must be passed to the testcase to execute them locally or in the ci-environment. Looking at the collaborative work on testcases the underlying SCM-system is probably more important than the question whether to use a Wiki or a graphical editor. In both cases the test documents must be stored under version control.
Wirting tests with the Robot Framework is very easy thanks to the graphical editor. This is especially true, because the underlying concepts of testsuites, testcases and keywords are nicely reflected here. When it comes to sharing testplans with a customer I also see some advantages in using the HTML-format as this can be viewed everywhere using some browser. Exchanging test definitions written with FitNesse requires that the Wiki-server is available as the source code of those is not too readable. Also the very strong reporting and the possibility to use any Java class (even JUnit tests) as libraries and keywords are on the positive side for the Robot Framework. But especially die-hard Java developers will have to overcome some mistrust here that Java testcases can really be executed without problems using Jython.
FitNesse, Robot or a completely different testing framework? It remains a questions of concrete project needs, available experiences within the team and personal preferences. Both tools are defintily worth a closer look when evaluating frameworks for agile testing.
Was this post helpful?
More articles
fromThomas Jaspers
22.2.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Thomas Jaspers
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.