Measuring collaboration tool success – Still a fool with a tool? – Part 1 of 2

“Knowledge-worker productivity is the biggest of the 21st century management challenges.” – Realized by Peter Drucker as early as 1999, the vast majority of today’s companies need to make use of their collective intelligence to compete. Thus, organizations from all kinds of industries use collaboration tools like Atlassian Jira or Confluence to manage their daily work. In the face of the spread of COVID-19, collaboration tools as enablers for decentral and asynchronous teamwork have seen an additional push.

But how can we measure the success of a collaboration tool roll-out (apart from the generic measures of completing the project in time, in quality and on budget)? And more importantly, how can we measure the actual business impact of a collaboration tool after six month, a year, or several years?

Collaboration tools: A more holistic approach is required

Today’s preferred measures are quantitative, like the number of page views in an enterprise wiki, or tasks added to an agile project management tool. Whereas analyzing such numbers may be a good start, what do they really reveal about the actual business impact of a collaboration tool?

In this two-part blog post, I would like to present a holistic, four-step approach to measure the success of your collaboration tools. This first part is concerned with developing a suitable measurement framework (steps one to three), whereas the second part focuses on identifying suitable KPIs (step four).

The approach is based on my 2018 Master’s dissertation that was concerned with developing a KPI-based measurement framework for an international telecommunications company’s collaboration platforms. Those findings were amplified with key learnings from 25+ collaboration tool projects as Atlassian consultant with codecentric.

So let’s get started with the first step:

Step 1: Understanding the underlying motivations

A common (anti-)pattern is to measure the use of the collaboration tool itself, e.g. by tracking the amount of page views. Whereas increased tool usage might be a positive development, one should keep in mind that collaboration tools always serve underlying business goals.

So as a first step, bring to mind why you are using collaboration tools. Obviously, motivations and their importance vary for each company, but these typical motivations may serve as a starting point:

• Increasing efficiency and productivity: Reducing communication costs, and making employees more productive
• Retaining and transferring company knowledge: Making employees’ implicit knowledge available and breaking down knowledge silos
• Connecting employees and/or experts: Allowing to identify subject matter experts, and enabling interactions between employees regardless of their departments or whereabouts. The latter has experienced a significant increase during COVID-19.
• Promoting open, participatory culture: Enabling employees to participate, share information, and flatten hierarchies
• Increasing transparency and awareness: Allowing employees a better understanding of the bigger picture, and to proactively see where they can support with their expertise. Thus, avoiding potential duplicate work.
• Increase velocity and innovation: Utilizing innovation impulses from collaboration platforms, fostering communication of innovations, and promoting creativity
• Increase/improve collaboration: Expectation to increase and improve the way of working together within the company, but also with partners or customers
• Enable work across physical locations: Enable working together independent from whereabouts

Step 2: Understanding what makes your collaboration tool(s) successful

Once you are clear about why you are using collaboration tools, you should also consider the factors that make your collaboration tool successful, as those success factors are an important foundation for reaching your goals.

As with the motivations, the success factors depend on your individual environment, but aspects with a pretty universal character include:

• Management support: It is crucial that collaboration tools receive support from top management – by allocating sufficient resources, meeting people’s concerns, and leading by example, ie. by actively using the collaboration tool(s).
• Open culture: For collaboration tools to work, there should be a culture characterized by employees’ willingness to work together across hierarchies and organizational units, to work in a more transparent way, and to share knowledge and opinions.
• Governance: It is important to have an overall concept and guidelines for the collaboration tool; governance should provide and steer consulting and change concepts and measures and ensure accompanying communication to drive new ways of working. Moreover, governance should take care of integrating collaboration tools in the overall IT portfolio to realize the maximum benefit.
• Usability: A crucial success factor for user acceptance is usability: If a tool is not easy to use, employees will not use it. With regards to network effects, the goal should be to realize the highest participation level possible.
• User qualification: Comprehensive qualification measures are a fundamental success factor for collaboration tools. In this context, training should not only cover tool usage, but also the indirect impacts, e.g. changes in communication flows. In addition, the respective company particularities should be considered.
• Support structures: To help users with their questions and requirements and increase user acceptance, support structures with sufficient capacities and competence should be set up. Moreover, one should not underestimate the impact of accompanying measures like on-site barcamps or workshops.

Step 3: Bringing motivations and best practices into a framework

The next step is to put the identified motivations and best practices into context. The Balanced Scorecard (BSC) has proven as a valuable tool for this.

As a BSC can also be implemented on a functional level, we are adopting the framework for a company’s collaboration tool landscape.

Considering the exemplary goals and best practices mentioned above, potential development areas might be:

• Organizational framework conditions: A perspective covering all framework conditions that are important for collaboration tools to be successful but relate to the entire organization – e.g. the corporate culture
• Collaboration tool framework conditions: A perspective covering all aspects associated with operating the collaboration platform – e.g. collaboration tool governance, or end user qualification and support
• Collaboration tool: A perspective covering all aspects related to the collaboration tool itself. – e.g. the level of usability
• Business impact: A perspective concerned with a collaboration tool’s impact for a company’s actual business and strategy. It might cover both financial and non-financial aspects – e.g. the costs for the collaboration tool, or derived improvements in innovation

Author’s illustration, based on Kühnapfel (2014): Balanced Scorecards im Vertrieb, S. 4

Now we have developed an exemplary framework that considers different relevant development areas. In the the second and final part of this blog post, we will tackle the trickiest part: Finding suitable key performance indicators (KPIs) for the identified development areas.