The comparison of images (screenshots) is one aspect of acceptance-level testing for web applications. It is especially meaningful for relatively static web pages to check for broken image references and layout flaws that are hard to detect otherwise. With Robot-AppEyes there is a ready-made keyword library available that supports this. But as it is using a cloud-based service for the image comparison it might not be suitable for projects due to legal implications. Therefore this blog post describes a different approach to implementing the image comparison using the Robot Framework.
The basic idea is using an external tool for the image comparison. This tool is then executed utilizing the Robot Framework OperatingSystem library. The tool used in this post to perform the image comparison is ImageMagick . It is free and available for all major operating systems. Furthermore it calculates a percentage value for the differences it finds when comparing two images. This can then be used as a result in the Robot Framework. This even enables us to define a threshold value for allowed deviations between a new screenshot from a test run and its reference. The following figure depicts the used components and their interaction.
For the sake of simplicity we will skip the Selenium part in this example and concentrate on executing the image comparison and evaluating the results. Embedding this into a Selenium test should then be straightforward.
Using ImageMagick for the image comparison is of course a bit arbitrary choice. If another command line tool should be used the approach shown in this blog post can be hopefully easily adapted.
Preparation Work – Installing ImageMagick
Installing ImageMagick on Mac OS turned out to be a bit more complicated than initially expected. After downloading the dedicated Mac OS binary from here and starting it (following the given instructions) the following error showed up:
bin ./magick -version
dyld: Library not loaded: /opt/X11/lib/libpng15.15.dylib
Referenced from: /Users/thomasjaspers/Documents/apps/ImageMagick-7.0.5/bin/./magick
Reason: image not found
[1] 21002 trace trap ./magick -version
A bit of googling revealed that this is a rather common problem that has been discussed quite in detail in the ImageMagick forum. Luckily the solution to this problem was rather simple in the end: Install ImageMagick using Brew . This often seems to be the best solution for installations anyway. To be on the safe side it makes sense to update brew first and then triggering the installation:
brew update
brew install imagemagick
The installation using brew works like a charm and Google is helping to find the proper command to compare two images using the command line interface of ImageMagick. As expected, there is no difference when comparing an image with itself.
workshop cd /usr/local/Cellar/imagemagick/7.0.7-3/bin
bin ./magick -version
Version: ImageMagick 7.0.7-3 Q16 x86_64 2017-09-18 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules
Delegates (built-in): bzlib freetype jng jpeg ltdl lzma png tiff xml zlib
bin ./convert image-1.png image-1.png -metric RMSE -compare -format "%[distortion]" info:
0%
With this preparation work done we are well prepared to implement an image comparison keyword for the Robot Framework and taking it into use in a simple example.
Implementing the “Compare Images” Keyword
The implementation of this keyword is based on the possibility to execute a system command and retrieve its output using the “Run And Return Rc And Output” keyword from the Robot Framework OperatingSystem library. Thus we can execute the ImageMagick compare command and retrieve the distortion %-value. That value is then used to decide whether or not the comparison is considered successful or not.
The complete example can be found from GitHub: https://github.com/ThomasJaspers/robot-framework-image-comparison .
The following listing shows the implementation of the keyword. The keyword is implemented in a separate resource file that can be used in one’s own projects right away (aside from adjustments depending on the used operating system).
*** Settings ***
Library String
Library OperatingSystem
*** Variables ***
${IMAGE_COMPARATOR_COMMAND} /usr/local/Cellar/imagemagick/7.0.7-3/bin/convert __REFERENCE__ __TEST__ -metric RMSE -compare -format "%[distortion]" info:
*** Keywords ***
Compare Images
[Arguments] ${Reference_Image_Path} ${Test_Image_Path} ${Allowed_Threshold}
${TEMP}= Replace String ${IMAGE_COMPARATOR_COMMAND} __REFERENCE__ ${Reference_Image_Path}
${COMMAND}= Replace String ${TEMP} __TEST__ ${Test_Image_Path}
Log Executing: ${COMMAND}
${RC} ${OUTPUT}= Run And Return Rc And Output ${COMMAND}
Log Return Code: ${RC}
Log Return Output: ${OUTPUT}
${RESULT} Evaluate ${OUTPUT} < ${Allowed_Threshold}
Should be True ${RESULT}
The variable ${IMAGE_COMPARATOR_COMMAND} defines the command to be executed for the image comparison. It contains placeholders for the concrete path information to the images to be compared. Those are replaced in the command string with the path values given as parameters and the resulting command string is logged out for troubleshooting purposes.
This approach should allow to easily adept this keyword for other operating systems. The idea is that only the command defined in ${IMAGE_COMPARATOR_COMMAND} needs to be changed to fit the underlying operating system.
The command is then executed and the resulting value indicating the differences is compared with the given threshold. That threshold is also given as a parameter and defines the grade of deviation that is considered ok. If that value is exceeded the keyword will fail and thus a corresponding test case. For troubleshooting purposes the return code and output of the system call is logged out to the Robot Framework log.html.
The test cases are defined as shown below. The images used are created in a way that the first test will succeed and the second test will fail (as seen in the screenshot of the log.html above).
*** Settings ***
Resource ./resources/image-comparison-keywords.robot
*** Test Cases ***
Image Comparison Ok
Compare Images ./reference-screenshots/reference-1.png ./test-screenshots/test-1.png 0.1
Image Comparison NOk
Compare Images ./reference-screenshots/reference-2.png ./test-screenshots/test-2.png 0.1
The threshold value is definitely something to play around with in a real-life test scenario. It might make sense to set it to 0.0 and thus completely strict. Of course the reference images must be created the same way (same size, same detail) as the images to be tested. In a scenario using Selenium this would mean to have a run taking the reference screenshots first. Those screenshots must be checked manually and then stored to the reference-screenshots directory. When making screenshots with Selenium it is possible to define the name for those screenshots and thus it should be no problem to trigger the comparison then as shown above in subsequent test runs.
Conclusion
The approach described in this document is hopefully helpful for projects considering image comparison as part of their test automation using the Robot Framework. The quality of the results of course strongly depends on the quality of the image comparison of the used tool – in this case ImageMagick. I do not really have any experience with this, but at least while developing this playground project it worked as expected. Having this example at hand, it might be possible to implement a prototype for these kinds of tests relatively quickly.
Any real-life feedback is of course very welcome :-).
Was this post helpful?
More articles
fromThomas Jaspers
22.2.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Thomas Jaspers
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.