Testing JavaScript on various platforms with Karma and SauceLabs

The web can be a brutal environment with various combinations of browsers and operating systems (platforms). It is quite likely that your continuous integration setup only covers a small portion of your users’ platforms. The unfortunate truth is that testing on these platforms is necessary to accommodate for compliance differences and partial support of standards and technologies like HTML, JavaScript, CSS and network protocols.

SauceLabs is a service that can be used to test web applications by simulating user behaviour or executing JavaScript unit tests. SauceLabs eases testing by supporting 442 different platform combinations and additionally recording the test execution. This means that you can actually see what a user might have seen and therefore trace errors easily.

For the purpose of this blog post we will cover the following setup:

• Sources and tests written using CommonJS modules. Modules are transpiled for the web using Browserify .
• Mocha as our test framework.
• Local test execution in the headless PhantomJS browser.
• Testing against the following platforms in our CI environment:
  • Chrome 35 on Windows 7,
  • Firefox 30 on an arbitrary operating system,
  • iPhone 7.1 on OS X 10.9 and
  • Internet Explorer on Windows 8.1.

You can follow along this blog post or check out a demo setup that you can find on GitHub . The
repository contains a list of all dependencies, the various configuration files, commonJS based tests and a Travis CI setup.

Step 1: Adding required Dependencies

In order to get started we need to install Karma’s command line utility and add some dependencies to our project as the following listing shows.

1npm install -g karma-cli
2 
3npm install --save-dev \
4            mocha \
5            karma \
6            karma-mocha \
7            karma-phantomjs-launcher \
8            karma-spec-reporter \
9            karma-bro

This is a formidable amount of dependencies. Let’s go through them one by one

• karma: Karma is a Node.js based utility that originated in the AngularJS project. Karma itself is a test runner dedicated to running tests in a browser environment. It is test framework agnostic. This means that you can use it with Mocha, Jasmine or any other preferred test framework.
• karma-cli: Karma’s command line interface (CLI) is a slim wrapper which is responsible for invoking the locally installed Karma version. By differentiating between global CLIs and actual implementations the same CLI can be used for different implementation versions.
• mocha: Mocha is our test framework of choice.
• karma-mocha: Karma has a concept of test frameworks, but it doesn’t know the specifics about Mocha or Jasmine. karma-mocha is an adapter that enables Mocha test execution.
• karma-phantomjs-launcher: This teaches Karma how to execute tests in PhantomJS. A variety of launchers exist, e.g. for Internet Explorer, Firefox and Chrome.
• karma-spec-reporter: Karma’s output is configurable and since we really like Mocha’s spec-reporter output, we also want to have it for our Karma based tests.
• karma-bro: CommonJS modules need to be transpiled in order to be usable inside a browser. karma-bro is responsible for the transpilation. Under the hood it uses Browserify.

Step 2: Creating a Karma Configuration File

Karma needs a configuration file which informs it about the specifics of our tests. The configuration file is similar to Grunt- and Gulp config files in that it is a simple commonJS module. This module needs to expose a function that is responsible for the configuration of Karma. The following listing shows a basic configuration which is typically stored in a file called local.karma.conf.js or similar and is checked into your version control system.

1module.exports = function(config) {
2  config.set({
3    logLevel: 'LOG_DEBUG',          // 1
4 
5    reporters: ['spec'],            // 2
6 
7    singleRun : true,               // 3
8    autoWatch : false,
9 
10    frameworks: [                   // 4
11      'mocha',
12      'browserify'
13    ],
14 
15    files: [                        // 5
16      'test/*.js'
17    ],
18 
19    preprocessors: {                // 6
20      'test/*.js': ['browserify']
21    },
22 
23    browserify: {                   // 7
24      debug: true
25    },
26 
27    browsers: ['PhantomJS']         // 8
28 
29  });
30};

Let us step through this from top to bottom. You can always refer to the reference documentation to get more information about the options and additional options that are not used.

1. The log level configuration defines, well, the log level. Nothing special here. Change the log level as necessary when issues occur.
2. Karma can use various reporters. This reporter resembles mocha’s spec reporter. As mentioned above this option is available because we installed the karma-spec-reporter dependency.
3. Karma can watch for file changes and efficiently re-execute the whole test suite. We are disabling the feature for now.
4. As mentioned previously Karma does not natively know how to execute Mocha based tests. It is possible to add support for various testing frameworks and transpilers through NPM dependencies like karma-mocha and karma-bro.
5. These files need to be loaded in the browser. Please note that the sources do not need to be explicitly loaded as this handled by the CommonJS require(...) statements and the Browserify transpilation step.
6. Karma has a concept of preprocessors that can be used for the Browserify transpilation.
7. Transpilers commonly have configuration switches that can be used to configure the output. When executing tests it is handy to have source maps enabled for easier debugging. This is exactly what the debug: tue option is doing.
8. This defines the browsers in which the tests should be executed. PhantomJS can be used because karma-phantomjs-launcher was added to the project. Add additional launchers and change this configuration options in order to use Firefox, Chrome and other browsers.

Step 3: Launching Tests

Karma has its own command line interface (CLI) that we installed in step 1 of
this blog post. The CLI makes it easy to execute our tests now that the test
execution is configured in the local.karma.conf.js file. The start command
exists to execute tests. The demo project it will look like this.

1# Debug log level output elided
2$ karma start local.karma.conf.js
3 
4INFO [karma]: Karma v0.12.23 server started at http://localhost:9876/
5INFO [launcher]: Starting browser PhantomJS
6INFO [PhantomJS 1.9.7 (Mac OS X)]: Connected on socket oyUKE1C1UaYDZ49N8Hpn with id 32075000
7 
8  root module
9    ✓ should welcome users with `Hello World`
10 
11PhantomJS 1.9.7 (Mac OS X): Executed 1 of 1 SUCCESS (0.006 secs / 0 secs)

Step 4: Running Tests against SauceLabs

Before we can execute tests on SauceLabs you will need to prepare yourself. For one you will need a SauceLabs account. Getting one is easy, but there are different processes for open source and private projects. For private projects go to the SauceLabs website and create an account. The free plan is sufficient for the start, but to test the application or website against various platforms a larger plan is necessary. Open source projects can use the OpenSauce plan and thereby get unlimited testing minutes and three parallel VMs. As an open source project maintainer just go to their dedicated website for open source projects and register for free. To progress further you will need your account name and access key. The access key can be found in the lower left hand corner of the SauceLabs website once logged in.

Next up is the installation of the karma-sauce-launcher dependency. This is necessary in order to teach Karma how to execute tests against SauceLabs. The dependency can be installed and saved to the package.json file via npm install --save karma-sauce-launcher.

Lastly a Karma configuration is necessary. We recommend that you create a separate file for the SauceLabs configuration. This has the benefit that tests can still be executed locally. We are using the file name saucelabs.karma.conf.js for the Karma SauceLabs configuration.

1var browsers = {                                 // 1
2  sl_chrome: {
3    base: 'SauceLabs',
4    browserName: 'chrome',
5    platform: 'Windows 7',
6    version: '35'
7  },
8  sl_firefox: {
9    base: 'SauceLabs',
10    browserName: 'firefox',
11    version: '30'
12  },
13  sl_ios_safari: {
14    base: 'SauceLabs',
15    browserName: 'iphone',
16    platform: 'OS X 10.9',
17    version: '7.1'
18  },
19  sl_ie_11: {
20    base: 'SauceLabs',
21    browserName: 'internet explorer',
22    platform: 'Windows 8.1',
23    version: '11'
24  }
25};
26 
27module.exports = function(config) {
28  config.set({
29    // all other options that are defined in
30    // local.karma.conf.js were elided for the
31    // purpose of this blog post.
32    reporters: ['saucelabs', 'spec'],            // 2
33    browsers: Object.keys(browsers),             // 3
34    customLaunchers: browsers                    // 4
35  });
36};

Please note that some configuration options were elided. Please see the demo project or the configuration file for the local test execution for these options. It is possible to share these configuration options between both files. If you are interested in this, check out the blog post’s accompanying GitHub repository . From top to bottom:

1. SauceLabs needs to be informed about the platforms that the tests should run on. In this case the tests will be running against four platforms, but more can be chosen. Check out SauceLabs’ list of platforms to read about possibilities. The object’s keys can be chosen freely. We will be using this object for options three and four.
2. A SauceLabs reporter is necessary when using the SauceLabs launcher. It informs SauceLabs about the test executions, i.e. whether they were successful or not.
3. As before we need to specify the set of browsers that we want to test against. Note that we only have to specify the browser names at this point.
4. SauceLabs needs more information than just a browser name. The specifics are handled by the customLaunchers option.

Now we can execute the tests against SauceLabs! You do it the following way:

1export SAUCE_USERNAME={your user name}
2export SAUCE_ACCESS_KEY={the access key that you see once logged in}
3karma start saucelabs.karma.conf.js

Conclusion

Having the possibility to execute tests against SauceLabs is very nice because it is hard to setup a reliable test infrastructure for a large amount of platforms. You additionally gain the possibility to see what happened in your application via screenshots and screencasts. If you have never seen this, you can check out one of our training project’s test executions on SauceLabs . This makes it way easier to trace and repair build failures.

Karma, with its transpiler and launcher concepts, is very extensible and a great choice for JavaScript projets. Use it to cope with browser differences and to improve your project’s quality.

Pro Tip: Did you know that you can test web applications that are running inside a private network, behind a firewall and that are not accessible via the internet? Use Sauce Connect to establish a secure connection between SauceLabs and the internal enterprise applications!