Integration of Microsoft Visual Studio Team Services, Protractor and Sauce Labs

Continuous Integration and Automated Testing in combination with cloud testing platforms can prove as a good practice in the web application development life cycle. As a result, these can improve the product reliability and quality. Cloud testing platforms provide a vast number of possible environments to run tests against, execution in parallel to improve efficiency, execution recording and detailed reporting and overall a very comfortable experience for QA teams. Based on an example, this post will explain how to integrate these concepts. The example will use the following technologies: Microsoft Visual Studio Team Services (VSTS) , the Protractor end-to-end testing framework, and the Sauce Labs cloud testing platform. The web application in this example is based on Angular and created using Angular CLI.

Protractor configuration

To enable test execution forwarding to Sauce Labs, add the following attributes to exports.config object in protractor.conf.js:

1exports.config = {
2  ...
3  sauceUser: /*sauceUser string*/,
4  sauceKey: /*sauceKey string*/,
5  ...
6}

Any specific Sauce Labs options go into the capabilities object inside exports.config.
For example, if there is a need to extend the default Sauce Labs test duration time (30 minutes), then add a property maxDuration with the integer value in seconds.
Also, to enable parallel test execution, add the properties shardTestFiles and maxInstances to capabilities. The property shardTestFiles ensures that tests run in parallel, while maxInstances represents the number of tests running in parallel.

1exports.config = {
2  ...
3  capabilities: {
4    ...
5    maxDuration: /*integer value*/,
6    shardTestFiles: true,
7    maxInstances: /*integer value*/,
8    ...
9  }
10  ...
11}

To have a test result report generated, add Jasmine JUnit XML Reporter in onPrepare function in protractor.conf.js:

1onPrepare: function () {
2  ...
3  jasmine.getEnv().addReporter(new JasmineReporters.JUnitXmlReporter(
4    {
5      consolidateAll: true,
6      savePath: 'e2e/testresults',
7      filePrefix: 'testresult-' + Date.now()
8    }
9  ))
10  ...
11}

The VSTS build definition will use the resulting file of the XML reporter to publish test results.

Microsoft VSTS build definition

Build definition can be created from existing templates or from an empty process with custom tasks.
Microsoft VSTS offers various build definition tasks and tools regarding build, test, package, deploy, utility, etc.
This example will use npm build tasks to run scripts.

The first script is npm install. It will install a project on the build server and fetch dependent npm modules.

Next is a custom script npm run webdriver-update. It will download and install Selenium server and drivers required for end-to-end tests to run.

Next is also a custom script npm run e2e. It actually calls the Angular CLI command ng e2e. This command will serve the application and run end-to-end tests in accordance with the configuration in protractor.conf.js. It is using the Sauce Labs username and key and is forwarding test execution to Sauce Labs.

Therefore, add following two scripts in package.json file:

1...
2"scripts": {
3  ...
4  "webdriver-update": "webdriver-manager update",
5  "e2e": "ng e2e",
6  ...
7}
8...

In addition, the build definition should have a task to publish test results. There is a built-in task for this purpose and it requires the test result format and test result file path. This task will use the resulting file from the Jasmine JUnit XML Reporter, so as a test result format choose JUnit and as test result file path input **\testresult*.xml.

Sauce Labs integration plugin

The command npm run e2e requires the Sauce Labs tunnel to be open to forward the test execution to Sauce Labs.

To integrate VSTS and Sauce Labs, install the additional plugin Sauce for Visual Studio Team Services in VSTS. This plugin can be found in the Visual Studio Marketplace .

After installing this plugin in VSTS, two new build definition tasks will be available: Sauce Labs Configuration (Preview) and Sauce Labs Stop Sauce Connect (Preview).
These two tasks provide VSTS with the actions of establishing and closing a tunnel to Sauce Labs, and they should surround the test execution task (npm run e2e) in the build definition.
Furthermore, VSTS requires Sauce Labs credentials in order to establish a tunnel. To do this, add a new Sauce Labs Credentials service endpoint in VSTS. After that, use the newly created Sauce connection in the Sauce Labs Configuration (Preview) build task.

It is good practice to pass the Sauce Labs username and key as build variables. VSTS offers public variables which will be used for Sauce Labs username. VSTS also offers secret variables which are useful for passwords, keys, etc because VSTS encrypts these variables. Sauce Labs key will be set as a secret variable. Build tasks access secret variables through command arguments (for example $(SAUCE_ACCESS_KEY)).
To access public variables in the Protractor configuration, use process.env and for secret variables use process.argv. So in this example it will be:

1exports.config = {
2  ...
3  sauceUser: process.env.SAUCE_USERNAME,
4  sauceKey: process.argv[3],
5  ...
6}

Conclusion

Finally, the build definition should contain these tasks:

• npm install
• npm run webdriver-update (Command: custom, Command and arguments: run webdriver-update)
• Sauce Labs Configuration (Preview)
• npm run e2e (Command: custom, Command and arguments: run e2e — $(SAUCE_ACCESS_KEY))
• Sauce Labs Stop Sauce Connect (Preview)
• Publish Test Results (Test result format; JUnit, Test results files: **\testresult*.xml, Search folder: $(System.DefaultWorkingDirectory))

After the build run is complete, a test report will be available in VSTS. The report will contain detailed information on the test run, pass/fail status, and error messages and stack trace in case of failed tests.

After the tests execution is complete, a report will also be available in Sauce Labs. This report will contain test results, logs, metadata and test report media (screenshots and execution video). The QA team can easily retrieve and use test report media in the further QA process.