Traverson – the Hypermedia API Client For JavaScript

A lot of REST APIs offer links between their individual resources. A well constructed client then only needs to know the URI of the root resource and can find its way from there to every other resource by following links with well defined relations – without hardcoding any URI except one, thus decoupling client and server. This pattern is commonly known as HATEOAS – Hypertext As The Engine Of Application State. APIs that do this are also referred to as Hypermedia APIs recently, although according to Mr REST himself, this it not even optional but mandatory for any decent REST API.

Traverson is a JavaScript module for Node.js and the browser that makes working with such APIs much easier.

GitHub’s API at https://api.github.com/ is an example for the HATEOAS pattern. Let’s see what a walk from resource to resource could look like with this API:

Starting at https://api.github.com/,

1{
2  "current_user_url": "https://api.github.com/user",
3  "gists_url": "https://api.github.com/gists{/gist_id}",
4  "repository_url": "https://api.github.com/repos/{owner}/{repo}",
5  ...
6}

… then following the link "repository_url",
using the parameters { owner: "basti1302", repo: "traverson" },
thus going to https://api.github.com/repos/basti1302/traverson

1{
2  "id": 13181035,
3  "name": "traverson",
4  "full_name": "basti1302/traverson",
5  "owner": {
6    "login": "basti1302",
7    ...
8  },
9  "url": "https://api.github.com/repos/basti1302/traverson",
10  "commits_url": "https://api.github.com/repos/basti1302/traverson/commits{/sha}",
11  "issues_url": "https://api.github.com/repos/basti1302/traverson/issues{/number}",
12  ...
13}

… then following the link "commits_url",
using the parameter { sha: 5c82c74583ee67eae727466179dd66c91592dd4a },
thus going to https://api.github.com/repos/basti1302/traverson/commits/5c82c74583ee67eae727466179dd66c91592dd4a

1{
2  "sha": "5c82c74583ee67eae727466179dd66c91592dd4a",
3  "commit": {
4    "author": {
5      "name": "Bastian Krol",
6      ...
7    },
8    "committer": {
9      "name": "Bastian Krol",
10      "date": "2013-10-25T11:41:09Z",
11      ...
12    },
13    "message": "test hal support against @mikekelly's haltalk server"
14    ...
15  },
16  "url": "https://api.github.com/repos/basti1302/traverson/commits/5c82c74583ee67eae727466179dd66c91592dd4a",
17  "comments_url": "https://api.github.com/repos/basti1302/traverson/commits/5c82c74583ee67eae727466179dd66c91592dd4a/comments",
18  ...
19}

… then, at last, following the link "comments_url",
thus finally arriving at
https://api.github.com/repos/basti1302/traverson/commits/5c82c74583ee67eae727466179dd66c91592dd4a/comments

1[
2  {
3    "id": 4432531,
4    "user": {
5      "login": "basti1302",
6      ...
7    },
8    "created_at": "2013-10-25T22:50:59Z",
9    "body": "This commit is a commit is a commit.",
10    ...
11  }
12]

So we followed three consecutive links, hopping from resource to resource, to reach the one resource we were actually interested in (the commit comments, in this contrived example).

Introducing Traverson

There a lot of useful modules and libraries that make working with REST APIs easier. Most of them focus on working with one resource or one request. But, as far as I know, there is no help with the typical pattern of following multiple links, walking from resource to resource, as outlined above – at least not in Node.js, or, more generally in JavaScript. Say hello to Traverson . Traverson centers around the idea of following a sequence of links or walking along a path of link relations.

Let’s return to the GitHub API example to show how this works. If you wanted to make use of the links this API provides, using, for example, the request module for Node.js directly, you could write something like this:

1var request = require('request')
2var rootUri = 'https://api.github.com/'
3 
4function nextUri(response, link) {
5  var resource = JSON.parse(response.body)
6  return resource[link]
7}
8 
9request.get(rootUri, function(err, response) {
10  if (err) { console.log(err); return; }
11  var uri = nextUri(response, 'repository_url')
12  uri = uri.replace(/{owner}/, 'basti1302')
13  uri = uri.replace(/{repo}/, 'traverson')
14  request.get(uri, function(err, response) {
15    if (err) { console.log(err); return; }
16    uri = nextUri(response, 'commits_url')
17    uri = uri.replace(/{\/sha}/,
18        '/5c82c74583ee67eae727466179dd66c91592dd4a')
19    request.get(uri, function(err, response) {
20      if (err) { console.log(err); return; }
21      uri = nextUri(response, 'comments_url')
22      request.get(uri, function(err, response) {
23        if (err) { console.log(err); return; }
24        var resource = JSON.parse(response.body)
25        console.log(resource)
26      })
27    })
28  })
29})

There it is, the infamous callback pyramid. Also, the code is repetitive. Now, with Traverson, the basic pattern for following a sequence of links to a target resource looks like this:

1var api = require('traverson').json.from('https://api.github.com/')
2api.newRequest()
3  .follow('repository_url', 'commits_url', 'comments_url')
4  .getResource(function(err, resource) {
5    // do something with the resource...
6  })

And the complete, working example for the GitHub API would be

1var api = require('traverson').json.from('https://api.github.com/')
2api.newRequest()
3  .follow('repository_url', 'commits_url', 'comments_url')
4  .withTemplateParameters({
5    owner: 'basti1302',
6    repo: 'traverson',
7    sha: '5c82c74583ee67eae727466179dd66c91592dd4a'
8  }).getResource(function(err, resource) {
9    if (err) { console.log(err); return; }
10    console.log(resource)
11  })

I don’t know about you, but I like the version that uses Traverson better.

So what happens here? Traverson will access the provided root URI given to from(). It will then work its way through the API by following the links identified by the arguments to the follow method. When it has finished the journey specified by the follow arguments, it will hand the last resource back to the caller. Actually, all that only happens when getResource is called, all method calls before that only set things up.

Traverson Features

URI Templates

Besides following links, which is Traverson’s core feature, the example above already used another feature, namely working with URI templates. Often the link to the next resource is not an actual URI but an RFC 6570 URI template . If Traverson encounters such a template on its way along the resources and values for the template parameters have been provided with the withTemplateParameters method, they are automatically substituted.

Besides URI templates, Traverson offers a number of other handy features:

• basic authentication,
• OAuth,
• custom HTTP headers,
• JSONPath,
• HAL (hypertext application language)

Let’s examine some of them in detail.

JSONPath

So far we told Traverson which links to follow by giving it a series of property names. Traverson looks for these properties in each resource to find out which URI to access next. What if the link is not a direct property of the resource? For example, if the resource looks like this:

1{
2  "deeply": {
3    "nested": {
4      "link: "http://api.io/congrats/you/have/found/me"
5    }
6  }
7}

To follow the link to http://api.io/congrats/you/have/found/me you can pass JSONPath expressions to the follow method:

1api.newRequest()
2  .follow('$.deeply.nested.link')
3  .getResource(function(error, document) {
4    ...
5  })

Of course, you can mix JSONPath expressions with plain vanilla property names.

Basic Authentication

Under the hood, Traverson uses the aforementioned request module, at least, when running in Node.js (in the browser, request is shimmed by using superagent ). Therefore, you can use all of request‘s features , like basic authentication, OAuth and custom headers directly with Traverson. For example, basic authentication with the GitHub API would look like this:

1var api = require('traverson').json.from('https://api.github.com/'),
2    ghUser = 'basti1302',
3    ghPass = '...'
4 
5api.newRequest()
6  .follow(...)
7  .withRequestOptions({
8    auth: {
9      user: ghUser,
10      pass: ghPass,
11      sendImmediately: true
12    }
13  }).getResource(function(err, resource) {
14    ...
15  })

Working with HAL

HAL, the Hypertext Application Language, “is a simple format that gives a consistent and easy way to hyperlink between resources in your API”. There is a formal spec for the media type application/hal+json. It does not say anything about how to structure the payload content of your JSON, but it defines where and how links between resources are presented to the client. This makes the life easier for clients because they know exactly where to look for links, even when talking to different APIs. For example, you could follow the path from the root resource via the links ht:users, ht:user and ht:posts to arrive at the list of all posts for a user.

Let’s do that with Traverson. To use HAL, you just use require('traverson').jsonHal instead of require('traverson').json.

1var api = require('traverson').jsonHal.from('http://haltalk.herokuapp.com/')
2 
3api.newRequest()
4  .withRequestOptions({
5    headers: {
6      'Accept': 'application/json',
7      'Content-Type': 'application/json'
8    }
9  }).follow('ht:users', 'ht:user', 'ht:posts', 'ht:post')                                                                      
10  .withTemplateParameters({name: 'mike'})                                   
11  .getResource(function(error, document) {
12     // document will represent the list of posts for user mike
13  })

Or, to post a new entry for a user:

1var body = { content: 'Hello there!' }
2api.newRequest()
3  .withRequestOptions({
4    headers: {
5      'Accept': 'application/json',
6      'Content-Type': 'application/json'
7    },
8    auth: {
9      user: 'traverson',
10      pass: '...', // traverson's password would go here :-)
11      sendImmediately: true
12    }
13  }).follow('ht:me', 'ht:posts')
14  .withTemplateParameters({ name: 'traverson' })
15  .post(body, function(error, respons) {
16    if (err) { console.log(err); return; }
17    if (response.statusCode === 201) {
18      console.log('unexpected http status: ' + response.statusCode)
19    }
20  })

For both examples we needed to set the Accept and the Content-Type header to make the Haltalk server understand our requests. Since Haltalk requires authenication to post stuff, we also needed to pass our user and password along.

In the browser

Working with REST APIs is definitely a thing in server side JavaScript, which is why Traverson started out as a Node.js module. But it can also be useful when working with APIs directly from the browser via AJAX. That’s the reason why Traverson also works in the browser, thanks to browserify . The browser build is a single file with a UMD that can be either included with a script tag or loaded by an AMD loader like RequireJS.

Using Traverson in the browser is not any different from using it in Node.js:

1<html>
2  <head>
3    <meta charset="utf-8">
4    <title>Traverson in the Browser</title>
5    <script src="http://code.jquery.com/jquery-2.0.2.min.js"></script>
6    <script src="traverson.min.js"></script>
7  </head>
8  <body>
9 
10    <div id="response"/>
11 
12    <script>
13      var api = traverson.json.from('https://api.github.com/')
14      api.newRequest()
15        .follow('repository_url', 'commits_url', 'comments_url')
16        .withTemplateParameters({
17          owner: 'basti1302',
18          repo: 'traverson',
19          sha: '5c82c74583ee67eae727466179dd66c91592dd4a'
20        }).getResource(function(err, resource) {
21        if (err) {
22          $('#response').html(JSON.stringify(err))
23          return
24        }
25        $('#response').html('Hooray! The commit comment is <br/>"' +
26            resource[0].body + '"')
27      })
28    </script>
29  </body>
30</html>

Background Story: Shave More Yaks

How did Traverson come about? Well, I have been doing a bit of yak shaving lately. Actually, quite a lot of yak shaving. It all started a few weeks ago, when I set out to write a short blog post about Cucumber.js , the JavaScript port of Cucumber that runs on Node.js and in the browser. What would make a nice little example for using Cucumber.js? I decided to showcase testing a REST API with Cucumber.js and Mikeal Rogers’ request library for Node.js. So I started writing some “tests” for GitHub’s API. While doing so, I noticed that the GitHub API actually offers links between the resources.

As mentioned above, a cleverly constructed client only needs to know the URI of the root resource and needs no other hard coded URIs. Because my tests were not “cleverly constructed” and did not make use of the provided links, I needed to hard code the URI of each endpoint. This was clearly not satisfying. So I started looking around if there is a node module that makes working with link-driven APIs easier. There are a few, but none of them offered the functionality I wanted – making following a sequence of links as easy as a single method call.

So I started to implement a little library for working with HATEOAS APIs – Traverson. When Traverson had come far enough to work for the simple Cucumber.js/GitHub API showcase, I thought it would also be nice to have Traverson support HAL, the hypertext application language, which tries to establish a standard for JSON-based hypertext APIs. There was Halbert , a node module for working with HAL resources, so I thought HAL support for Traverson is probably a matter of hours. Turned out that Halbert had some issues and was not yet fully spec-compliant. So I gave it an overhaul and sent the author some pull requests. He merged them nice and fast and also fixed some of the problems I found himself. Hooray for open source. A while later I wrote my own HAL module Halfred anyway to be able to better align it with Traverson’s goals – or because of not-invented-here-syndrome, who knows. It also occured to me that Traverson would be as useful in the browser as in Node.js, so I started to turn it into a cross platform module, that works in Node.js as well as in the browser. Because that is not entirely trivial, I also wrote a quick project template for such cross platform JavaScript modules based on npm, browserify, Grunt, Mocha and some other good stuff. I’ll probably blog a bit or two about cross platform JavaScript later.

However, at that point I was already four to five yak shaving stack frames deep, starting from the Cucumber.js blog post I originally wanted to do. I solemnly swear to write this blog post sooner or later. The code is already there , I just need to write it up 🙂

Anyway, I in my humble opinion Traverson is actually more useful than the Cucumber.js blog post would have been, so maybe yak shaving is not always a bad thing per se. Plus, I own a very nice, yak hair stuffed pillow now.