Ever since I started working with the Robot Framework – already back in the days when I saw it evolving in the Nokia R & D – I am totally convinced of the power of its keyword-driven approach. It is an excellent example of the fact that the best solutions are the simple ones. This is the backbone of the Robot Framework functionality and on top of this a lot of other valuable features have been developed. To name the ones that are most impotant to me:
- The possibility to develop new test functuionality (keywords) in Java.
- A lot of ready-made Test Libraries for different purposes.
- Support to implement Test Libraries that can be run on Remote Servers.
- An IDE for writing new testcases and high-level keywords (RIDE).
- A very good documentation and active community.
It should be noticed that we have to differ carefully in the following between the term Test and Testcase. It is assumed that the Testcases are known and already implemented as working Robot Framework keywords. The part we are talking about in this post is adding (a lot of) different variations of the actual parameters which derive a concrete Test from a Testcase.
Ok, but what is then the but? What is the gap that is mentioned in the title of this blog post? To make it short: It is the possibility to allow non-technical persons to write – potentially a lot of – new tests.
- Even though the Robot Framework has a quite nice IDE with RIDE it is not realistic to assume that non-technical people are using it to write new tests on their own; even if the basic testcase implementation is already done by the development team (and that this is the case is our basic assumption throughout this post).
- A huge amount of tests are hard to maintain and this is not only true for less technically oriented people.
- Both of the above mentioned topics are independent of the Robot Framework also true for all other open source test automation tools I have seen so far.
It might be hard to understand for developers that RIDE is not sufficient, but it is the feedback we are getting in some of our customer projects. My very rough guess is that in half of the cases using RIDE is fine, but the other half of the users/customers are missing something more convenient. Of course this also depends on the amount of tests and persons involved in writing tests. In the end this could lead to the point where the Robot Framework as such is challenged.
Generic Testdata Framework (GTF)
Enough talking on problems, let’s think of some ideas for a solution :-). Of course we need a cool name first and somehow Generic Testdata Framework (GTF) was coming to my mind. This makes it also easier to refer to this solution later on. What are the basic requirements and ideas for the framework which would be put on top of the Robot Framework:
- Definitly some kind of graphical frontend (GUI) is required that can be used to write additional tests comfortably.
- We want to hide as much of the underlying technical stuff from the user as possible.
- In an ideal case the solution should be open to allow switching the used GUI without breaking the whole solution.
- This will not be an out-of-the-box solution, but it will require some initial customizing (by some development team).
- All existing Robot Framework features should be supported by this solution.
The following figure points out the basic ideas for the GTF:
Generic Testdata Framework
The GUI could be basically anything, a Web-Frontend, a full-blown stand-alone GUI or – like it or not 😉 – Excel. The important part is that each GUI-implementation must be capable of writing the – yet to be defined – Common Testdata Format. Together with some Metadata information and Testsuite Templates the Generator-component can then generate a valid Testsuite file. From there onwards the tests can e executed using the usual Robot Framework scripts.
Sketching the Implementation Details
When considering the GTF I was thinking from the very beginning that – with some hopefully clever decisions – the effort for the actual implementation of a first release should not require too much coding. Therefore no fancy XML-format will be used for the Common Testdata Format, but it will be CSV (Comma-Separated Values). The biggest advantage of this format is that it can be directly exported from Excel. And also any custom GUI-implementation should not have any problems producing this format.
Basically any Robot Framework testcase can in the end be considered to be some high-level keyword that is expecting a certain amount of parameters. Some of them are input parameters to the test and some of them might be expected values to be checked in the test execution. Thus each row in CSV-file can be seen as such a set of parameters that must be passed to the corresponding keyword.
The Common Testdata Format does not contain any information as such on the keywords to be executed. If this would be the case at least in the Excel-approach there would be again some – even if it would be a minor – technical part visible to the user.
Now this leads to the required …
Metadata and Testsuite Templates
The Metadata is the glue between the Common Testdata Format (CTF) and the Robot Framework keywords to be used. The basic idea is that one CTF-file will be mapped to one corresponding Testsuite file. For each resulting Testsuite file there are corresponding templates created. The nice thing about those is that they anyway basically do already exist once a first test is written in a testsuite. The testsuite file is therefore split into the following templates:
- Testsuite-Header: Here all import statements and common definitions are made.
- Testsuite-Footer: Closing the Testsuite to make it a valid Robot Framework Testsuite-file.
- Testcase-Template: A template to execute the test once with parameter values at all places where concrete arguments (those are read from one line of the CTF-file) are required.
Now the Metadata-format must describe how to match one CTF-file with such a set of Template-files. On first (and second) sight the easiest way is probably using (parts) of the CTF-file filename to perform this kind of matching. This would only require a minimal amount of metadata that only needs to be defined once for any new kind of testcase, but therefore anyway new implementation is required by the development team. The Testsuite-file would be as well already be available in the “traditional” approach, thus the additional effort here is to cut it into the different template-pieces and potentially there is a need to ensure that there exists really one top-level keyword that can be used to execute a concrete test.
This leads quite straightforward to the …
Generator-Component
First of all this is a command-line script and probably it will be written in Java, even though I could be tempted to use Perl for it, but that is another story ;-).
The script has to read in CTF-files stored to some directory. Then it reads in the Metadata-file and has to match the information from the filenames to the corresponding template-files. It then generated the Testsuite-files by just reading the Header-Template, then in a loop adding the testcases by filling and adding the Testcase-Template and then closing with the Footer-Template. Once all Testsuites are generated it could even execute the Robot Framework directly on those or keep this to some wrapper-script. Passing the Robot Framework specific parameters in could be easily achieved using argument files .
The nasty Details
Of course – as usual in software projects – there are a lot of nasty details to be considered. Some of them coming to my mind right away are:
- What about the tagging feature of the Robot Framework. With this approach individual tagging of tests would not be possible.
- What if I have some complex arguments that potentially break the CSV-format when saved from Excel.
- Non-technical users might now be able to write tests but they will never be able to execute them.
Luckily we have the chance to discuss some of these topics quite in detail with one of our customers and a lot of good ideas are coming from those discussions. It is really very helpful to get a chance to see those problems entirely from the user perspective.
With Excel as such there are of course some additional problems one could think of. Parallel editing of files. Changing the format and thus breaking the generator process. For sure some of those problems cannot be solved in the end using a pure technical approach but probably also some process model is required.
For the problem of executing the tests one could think of one (or more) central Robot Framework installation. On top of this some CI-server (e.g. Jenkins ) could be configured and then the tests can be again executed with some quite nice Web-UI. But this is already an idea – and a blog post – of its own.
So this is for sure not the end of possible improvements to the overall process, but maybe and hopefully it could be a small part of it.
To be continued …
These ideas are spinning around in my head for some time now luckily now I could finally put them down on some (virtual) paper. Of course only some real coding can now prove whether or not this is a valid idea. I am prepared for this 🙂 and I hope to find some time to start some real coding already next week. That will for sure help (me) getting a better understanding of this.
Of course any kind of feedback on this idea would be great and is more than welcome. To be continued …
Was this post helpful?
More articles
fromThomas Jaspers
22.2.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Thomas Jaspers
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.