GoMock vs. Testify: Mocking frameworks for Go

Summary:
Testify/mock and mockery are the tools of choice, providing an overall better user experience, if you do not need the additional power of the GoMock expectation API.

Testify has the better mock generator and error messages while GoMock has the more powerful expecation API, allowing you to assert call order relations between the mocked calls.

Point-by-point overview:

• GoMock provides a more powerful and more type-safe expectation API.
• Testify provides more helpful output on test failures.
• The mockery CLI is easier to use than mockgen.
• Testify handles mocking embedded imported interfaces better than GoMock.

Introduction

In this post, we’ll compare two popular mocking frameworks for Go:

• golang/mock
• testify/mock

We’ll look specifically at the following criteria:

• Repository activity
• Mock generation
• Argument matchers
• Type safety
• Output for failing tests
• Integration with Go tooling

Both libraries work well with the standard library, and both rely on code generation tools as the alternative to writing out the mock boilerplate yourself.

The goal of this post is to investigate any trade-offs you might be making by choosing one over the other.

We’ll start with a quick recap of my last post about GoMock , have a look at basic usage of testify/mock and then proceed to compare the two frameworks point-by-point.

Contents

• GoMock
• Testify/mock
• Comparison
  • Repository activity
  • Output on test failure
    • Unexpected calls
    • Unexpected parameter values
    • Missing calls
  • Mock generation
    • Manual usage
    • Usage with go:generate
    • Embedded interfaces
  • Mock usage
    • Construction and assertion
    • Expectations
  • Type safety
• Conclusion

GoMock

GoMock, slightly older than testify/mock, was first released in March of 2011 and is part of the official github.com/golang namespace. As of this writing, it has 2758 GitHub stars and a total of 43 contributors. GoMock consists of two components:

• The gomock package github.com/golang/mock/gomock
• The mockgen code generation tool github.com/golang/mock/mockgen

Both can be installed via go get:

go get -u github.com/golang/mock/gomock
go get -u github.com/golang/mock/mockgen

Usage of GoMock follows five basic steps:

1. Define an interface that you wish to mock:
   examples/gomock/doer/doer.go

   package doer
   
   type Doer interface {
       Do(int, string) error
   }
   

   …that is used by a file you wish to test:

   examples/gomock/user/user.go

   package user
   
   import "github.com/sgreben/gomock-vs-testify/examples/gomock/doer"
   
   type User struct {
       Doer doer.Doer
   }
   
   func (u *User) Use() {
       u.Doer.Do(1, "abc")
   }
   

2. Use mockgen to generate a mock from the interface:

   mockgen -source=doer/doer.go -destination=mocks/mock_doer.go -package=mocks
   

3. Define a mock controller inside your test, passing a *testing.T to its constructor, and use it to construct a mock of your interface:

   mockCtrl := gomock.NewController(t)
   defer mockCtrl.Finish()
   
   mockDoer := mocks.NewMockDoer(mockCtrl)
   

4. Use EXPECT() to set up the mock’s expectations in your test:
   examples/gomock/user/user_test.go

   func TestUserWithGoMock(t *testing.T) {
       mockCtrl := gomock.NewController(t)
       defer mockCtrl.Finish()
   
       mockDoer := mocks.NewMockDoer(mockCtrl)
       testUser := &user.User{Doer:mockDoer}
   
       // Expect Do to be called once with 1 and "abc" as parameters, and return nil from the mocked call.
       mockDoer.EXPECT().Do(1, "abc").Return(nil).Times(1)
   
       testUser.Use()
   }
   

5. Assert the mock controller’s associated mocks‘ expectations using Finish():

   mockCtrl.Finish()
   

   It’s idiomatic to defer the call to mockCtrl at the point of declaration of the mock controller.

Testify/mock

Testify/mock is slightly newer than GoMock, seeing its first release in October 2012. As of this writing, the testify repository has 7966 GitHub stars and 135 contributors.

As with GoMock, you’ll likely need two components to use testify/mock:

• the testify/mock library from https://github.com/stretchr/testify/
• the companion code generation tool mockery from https://github.com/vektra/mockery

Both can be installed using go get:

go get github.com/stretchr/testify/mock
go get github.com/vektra/mockery/.../

Usage of the Testify mocking framework usually comprises the following five steps:

1. Define an interface, say Doer in doer/doer.go, that you wish to mock and a user of that interface that you wish to test, in our example user/user.go:
   examples/testify/doer/doer.go

   package doer
   
   type Doer interface {
       Do(int, string) error
   }
   

   …that is used by a file you wish to test:

   examples/testify/user/user.go

   package user
   
   import "github.com/sgreben/gomock-vs-testify/examples/testify/doer"
   
   type User struct {
       Doer doer.Doer
   }
   
   func (u *User) Use() {
       u.Doer.Do(1, "abc")
   }
   

2. Use mockery to generate mocks for your interfaces. In our example, we generate a mock for the interface Doer in doer/:

   mockery -dir doer -name Doer
   

   This will place a file Doer.go into the directory (and package) mocks containing an implementation mocks.Doer of the Doer interface.

3. Construct the mock object by instantiating its struct:

   mockDoer := &mocks.Doer{}
   

4. Set up expectations using the .On() method of your mock:
   examples/testify/user/user_test.go

   func TestUserWithTestifyMock(t *testing.T) {
       mockDoer := &mocks.Doer{}
   
       testUser := &user.User{Doer:mockDoer}
   
       // Expect Do to be called once with 1 and "abc" as parameters, and return nil from the mocked call.
       mockDoer.On("Do", 1, "abc").Return(nil).Once()
   
       testUser.Use()
   
       mockDoer.AssertExpectations(t)
   }
   

5. Assert each of your mock’s expectations using .AssertExpectations(t), passing the *testing.T of your test as a parameter:

   mockDoer.AssertExpectations(t)
   

Comparison

Now that we have briefly introduced both frameworks, let’s have a look at their features and requirements side-by-side.

Repository activity

Summary: Testify is both more popular and more active on GitHub. It is also used by more packages according to GoDoc.

The testify repo is more popular (judging by GitHub stars) and more active (judging by number of contributors). Testify is also imported by more packages (3348 as of this writing) than GoMock (2664 as of this writing) according to GoDoc stats.

Output on test failure

Summary: Testify provides better error messages on unexpected calls and calls with unexpected parameter values, including argument types, a helpful stack trace, and closest matching calls. GoMock provides more helpful reports on missing calls.

When tests succeed, we don’t particularly care about the mock framework’s output – we know that the mock was used as expected and that’s about it. However, when things go wrong, we need to pinpoint the cause of the failure as quickly and comfortably as possible. This is where the mocking framework can support us. We’ll distinguish three classes of failures:

1. Unexpected calls
2. Missing calls (expected, but not occurred)
3. Expected calls with unexpected parameter values

For each of these cases, we’ll construct a minimal test and compare the frameworks‘ output.

Unexpected calls

In our experience, this is the most frequent cause of mock assertion failure. Thus, it is particularly important that the frameworks provide adequate reporting here.

For the test case, we’ll reuse our minimal Doer/User example from the introduction:

comparison/user/user_unexpected_call_test.go

package user_test

import (
    "testing"

    "github.com/golang/mock/gomock"
    "github.com/sgreben/gomock-vs-testify/doer"
    "github.com/sgreben/gomock-vs-testify/mocks"
    "github.com/sgreben/gomock-vs-testify/user"
)

func TestUser_GoMock_UnexpectedCall(t *testing.T) {
    mockCtrl := gomock.NewController(t)
    defer mockCtrl.Finish()

    mockDoer := doer.NewMockDoer(mockCtrl)
    testUser := &user.User{Doer: mockDoer}

    testUser.Use()
}

func TestUser_Testify_UnexpectedCall(t *testing.T) {
    mockDoer := &mocks.Doer{}
    testUser := &user.User{Doer: mockDoer}

    testUser.Use()
    mockDoer.AssertExpectations(t)
}

We execute the test using go test -v:

$ go test -v user/user_gomock_unexpected_call_test.go

From GoMock, we obtain the following output:

=== RUN   TestUser_GoMock_UnexpectedCall
--- FAIL: TestUser_GoMock_UnexpectedCall (0.00s)
        controller.go:113: no matching expected call: *doer.MockDoer.Do([1 abc])

We do get the actual arguments ([1 abc]), but not their type (is that 1 an int32 or an int64?). Furthermore, we do not get the source location of the unexpected call. The report from Testify is more helpful:

=== RUN   TestUser_Testify_UnexpectedCall
--- FAIL: TestUser_Testify_UnexpectedCall (0.00s)
panic:
assert: mock: I don't know what to return because the method call was unexpected.
        Either do Mock.On("Do").Return(...) first, or remove the Do() call.
        This method was unexpected:
                Do(int,string)
                0: 1
                1: "abc"
        at: [Doer.go:13 user.go:10 user_unexpected_call_test.go:26] [recovered]
        panic:
(repeated error message omitted)
(stack trace omitted)

In particular, by including a compact stack trace, the Testify output allows us to pin-point the exact call that was not expected. Moreover, both values as well as types of the unexpected call’s arguments are included.

Unexpected parameter values

Another frequent failure scenario is that our code calls the mocked object with different parameters than expected. The mocking framework can help us here by telling us how the actual situation differed from the expected one.

We use our Doer/User example to set up the test:

comparison/user/user_unexpected_args_test.go

package user_test

import (
    "testing"

    "github.com/golang/mock/gomock"
    "github.com/sgreben/gomock-vs-testify/comparison/mocks"
    "github.com/sgreben/gomock-vs-testify/comparison/user"
)

func TestUser_GoMock_UnexpectedArgs(t *testing.T) {
    mockCtrl := gomock.NewController(t)
    defer mockCtrl.Finish()

    mockDoer := mocks.NewMockDoer(mockCtrl)
    testUser := &user.User{Doer: mockDoer}

    mockDoer.EXPECT().Do(2, "def")

    // Calls mockDoer with (1, "abc")
    testUser.Use()
}

func TestUser_Testify_UnexpectedArgs(t *testing.T) {
    mockDoer := &mocks.Doer{}
    testUser := &user.User{Doer: mockDoer}

    mockDoer.On("Do", 2, "def")

    // Calls mockDoer with (1, "abc")
    testUser.Use()
    mockDoer.AssertExpectations(t)
}

GoMock’s output is terse, and does not relate expected calls to actual calls. Here Testify clearly wins: not only does it print the types and values of the actual arguments, it also finds the closest matching expected call.

• GoMock:

  1=== RUN   TestUser_GoMock_UnexpectedArgs
  2--- FAIL: TestUser_GoMock_UnexpectedArgs (0.00s)
  3    controller.go:113: no matching expected call: *doer.MockDoer.Do([1 abc])
  4    controller.go:158: missing call(s) to *doer.MockDoer.Do(is equal to 2, is equal to def)
  5    controller.go:165: aborting test due to missing call(s)

• Testify:

  1=== RUN   TestUser_Testify_UnexpectedArgs
  2--- FAIL: TestUser_Testify_UnexpectedArgs (0.00s)
  3panic:
  4 
  5mock: Unexpected Method Call
  6-----------------------------
  7 
  8Do(int,string)
  9        0: 1
  10        1: "abc"
  11 
  12The closest call I have is:
  13 
  14Do(int,string)
  15        0: 2
  16        1: "def"
  17(repeated error message omitted)
  18(stack trace omitted)

Missing calls

The converse situation, that of expected calls that do not occur, happens frequently enough to warrant investigation, but is not quite as interesting — if a call does not occur there’s not much the framework can tell you. It doesn’t know where the call was supposed to come from.

Nevertheless, there is some information we want to see. Consider the following example:

comparison/user/user_missing_call_test.go

package user_test

import (
    "testing"

    "github.com/golang/mock/gomock"
    "github.com/sgreben/gomock-vs-testify/mocks"
)

func TestUser_GoMock_MissingCall(t *testing.T) {
    mockCtrl := gomock.NewController(t)
    defer mockCtrl.Finish()

    mockDoer := mocks.NewMockDoer(mockCtrl)

    mockDoer.EXPECT().Do(1, "abc").Return(nil)
    mockDoer.EXPECT().Do(2, "def").Return(nil)

}

func TestUser_Testify_MissingCall(t *testing.T) {
    mockDoer := &mocks.Doer{}

    mockDoer.On("Do", 1, "abc").Return(nil)
    mockDoer.On("Do", 2, "def").Return(nil)

    mockDoer.AssertExpectations(t)
}

While GoMock provides each missing call’s argument values (or, more precisely, argument matchers), Testify provides only the types:

• GoMock:

  1=== RUN   TestUser_GoMock_MissingCall
  2--- FAIL: TestUser_GoMock_MissingCall (0.00s)
  3        controller.go:158: missing call(s) to *doer.MockDoer.Do(is equal to 1, is equal to abc)
  4        controller.go:158: missing call(s) to *doer.MockDoer.Do(is equal to 2, is equal to def)
  5        controller.go:165: aborting test due to missing call(s)

• Testify:

  1=== RUN   TestUser_Testify_MissingCall
  2--- FAIL: TestUser_Testify_MissingCall (0.00s)
  3    mock.go:380: ❌	Do(int,string)
  4    mock.go:380: ❌	Do(int,string)
  5    mock.go:394: FAIL: 0 out of 2 expectation(s) were met.
  6            The code you are testing needs to make 2 more call(s).
  7            at: [user_missing_call_test.go:28]

Mock generation

Summary: Testify’s mockery tool is more convenient and less confusing to use than GoMock’s mockgen. It supports regex-based interface selection and, unlike mockgen, has just one mode of operation that supports all its features.

Both tools rely on boilerplate code for the mock implementations. Writing it by hand is tedious and error-prone. Fortunately, both tools also come with code generators.

Manual usage

GoMock’s mockgen has two modes of operation — the „source“ and „reflect“ modes. In source mode, mockgen is applied to single Go source files and generates mocks for all interfaces found in the given file. In reflect mode, mockgen is applied to Go packages and generates mocks for the specific interfaces given to it as a second argument.

Testify’s mockery operates on directories. If not provided a -dir flag, mockery operates on the current directory by default. The tool can generate mocks for single interfaces (via -name ), for sets of interfaces matching a regular expression (via -name ), or for all interfaces in the directory tree ("." or given via -dir).

Each tool’s argument sets for several scenarios are given in the following table:

Usage with go:generate

Both packages can be used with go:generate comments to integrate with the go generate tool.

• GoMock: Using the source mode of GoMock mocks for all interfaces in a given file can be comfortably generated using

  //go:generate mockgen -source=$GOFILE -destination=$PWD/mocks/${GOFILE} -package=mocks
  

  In the more powerful reflect mode, you need to explicitly specify both the package as well as the interfaces in that package to generate mock implementations for:

  //go:generate mockgen -destination=$PWD/mocks -package mocks github.com/sgreben/gomock-vs-testify/comparison/gogenerate Interface1,Interface2
  

  There is currently no way to specify that all interfaces in a package should be mocked when using reflect mode.

• Testify: When using mockery, you have several options on which annotation to place where:
  Single interface in a given directory

  //go:generate mockery -name MyInterface
  

  Multiple specific interfaces in a given directory:

  //go:generate mockery -name "MyInterface1|MyInterface2"
  

  Interfaces matching a regex in a given directory:

  //go:generate mockery -name "My.*"
  

  All interfaces in a given file’s parent directory (and its child directories):

  //go:generate mockery -all -output $PWD/mocks
  

Embedded interfaces

For imported embedded interfaces, such as the following ReadCloser example, the source mode of GoMock fails:

package readcloser

import "io"

type ReadCloser interface {
    io.Reader
    Close() error
}

$ mockgen -source readcloser/readcloser.go -package readcloser

2017/07/16 19:54:59 Loading input failed: readcloser/readcloser.go:8:2: unknown embedded interface io.Reader

This is a known issue and the recommended workaround is to use reflect mode:

$ mockgen -destination readcloser/mock_readcloser.go github.com/sgreben/gomock-vs-testify/readcloser ReadCloser

The downside is that you now need to explicitly specify the package and list the interfaces to generate mocks for. Testify has no problem with embedded imported interfaces, and you can still use the -all option to generate mocks for all interfaces:

$ mockery -dir readcloser/ -all

Generating mock for: ReadCloser

Mock usage

Summary: GoMock provides a more powerful expectation API. It feels more consistent than Testify’s, using a single Matcher type rather than a complicated Diff function that combines matching with diff-building.

We look at two specific aspects of mock usage:

• Construction and assertion: what does it take to get a mock object and assert its expectations?
• Expectations: what kinds of assertions are we able to express using the expectation API, and how intuitive is it?

Construction and assertion

Using Testify, you can directly construct your mock object via

mockDoer := &mocks.Doer{}
mockOther := &mocks.Other{}

// (set up expectations)

mockDoer.AssertExpectations(t)
mockOther.AssertExpectations(t)

and assert each mock object’s expectations using .AssertExpectations(t).

GoMock requires you to first instantiate a mock controller — though note that you only need one mock controller per test (not per mock). Finally, we need to call mockCtrl.Finish() to assert all mock’s expectations.

mockCtrl := gomock.NewController(t)
mockDoer := doer.NewMockDoer(mockCtrl)
mockOther := other.NewMockOther(mockCtrl)

// (set up expectations)

mockCtrl.Finish()

With GoMock, a single gomock.Controller can be shared between any number of mocks, and all their expectations can be asserted at once. With Testify you have to remember to assert each mock’s expectations.

Expectations

This is the part of the API we spend the most time with when dealing with mocks. Hence, it is particularly important that the expectation API is convenient and powerful. We consider three features of an expectation API — argument matchers, call frequency, and call order.

• Argument matchers: Can we easily specify classes of acceptable arguments? (type, value range, arbitrary).
  • GoMock provides four matchers out of the box — (Any,Eq,Nil, and Not) as well as providing a simple interface gomock.Matcher for custom matchers. See my last post for an implementation of a type matcher OfType.

    // Any and Not matchers
    mockDoer.EXPECT().Do(gomock.Any(), gomock.Not("abc"))
    
    // Custom matcher definition
    type HasPrefix struct{ prefix string }
    
    func (h HasPrefix) Matches(x interface{}) bool {
        if s, ok := x.(string); ok {
            return strings.HasPrefix(s, h.prefix)
        }
        return false
    }
    
    func (h HasPrefix) String() string {
        return fmt.Sprintf("has prefix %v", h.prefix)
    }
    
    // Custom matcher usage
    mockDoer.EXPECT().Do(1, HasPrefix{"abc"})
    

  • Testify provides two matchers mock.Anything and mock.AnythingOfType and a facility to use custom matchers via mock.MatchedBy.

    // Anything and AnythingOfType matchers
    mockDoer.
        On("Do", mock.Anything, mock.AnythingOfType("string"))
    
    // Custom matcher
    mockDoer.
        On("Do", 1, mock.MatchedBy(func(x string) bool {
            return strings.HasPrefix(x, "abc")
        })
    

• Call frequency: Is there a way to assert that a call occurs only once, or between n and m times? GoMock supports all these cases:

  // Once
  mockDoer.EXPECT().
      Do(gomock.Any(), gomock.Any()).
      Times(1)
  
  // Range
  mockDoer.EXPECT().
      Do(gomock.Any(), gomock.Any()).
      MinTimes(2).
      MaxTimes(10)
  
  // Arbitrary number of times
  mockDoer.EXPECT().
      Do(gomock.Any(), gomock.Any()).
      AnyTimes()
  

  Testify supports only fixed call counts, but provides convenience methods for calls that occur once or twice:

  // Once/Twice
  
  mockDoer.
      On("Do", mock.Anything, mock.Anything).
      Once()
  
  mockDoer.
      On("Do", mock.Anything, mock.Anything).
      Twice()
  
  // Given number of times
  mockDoer.
      On("Do", mock.Anything, mock.Anything).
      Times(10)
  

• Call order: Does the framework provide a way to assert that mock calls must occur in a certain order?Only GoMock provides this feature:
  • Strict order between calls

  gomock.InOrder(
      mockDoer.EXPECT().Do(1, "first"),
      mockDoer.EXPECT().Do(2, "second"),
      mockDoer.EXPECT().Do(3, "third"),
  )
  

  • Relaxed order — some calls must come before other calls, but not all calls‘ order is fixed:

  callWithFoo := mockDoer.EXPECT().Do(1, "foo")
  callWithBar := mockDoer.EXPECT().Do(2, "bar")
  
  // callWithFoo and callWithBar must occur before this call
  mockDoer.EXPECT()
      .Do(3, "baz")
      .After(callWithFoo)
      .After(callWithBar)
  

Type safety

This is a category GoMock narrowly wins:

• GoMock:

  myMockObj.EXPECT().MyCall(1, "abc", nil).Return(123, nil)
  

  For GoMock, the code generation tool mockgen generates a MockRecorder object (returned by EXPECT()) that expectations can be set on by calling methods with the same names and argument counts as the methods to be mocked.

• Testify:

  myMockObj.On("MyCall", 1, "abc", nil).Return(123, nil)
  

  Testify mocks provide a single On method that takes the mock method’s name as a string parameter and an arbitrary number of mock arguments.

With Testify, you are free to

• Misspell the method name
• Use an incorrect number of arguments

and both frameworks let you

• Use arguments of incorrect types
• Specify an incorrect number of return values

For both tools, type safety is not fully given — both accept expected arguments of type interface{}, despite having the exact signatures available in the interface definition. This is likely a trade-off to support argument matchers.

Conclusion

Despite its official status as part of the github.com/golang namespace, GoMock is not a strictly better choice than Testify/mock and mockery. Testify and mockery are user-friendlier and more actively maintained.