Measuring KM

Just one day after posting about the dangers of ROI, someone suggested that we should look at KM as a set of services with deliverables, service levels, etc.

My initial reaction was similar to my reaction to requests for ROI. It is easy to see where this is leading: decomposing KM into its constituent "service" parts so the business groups can pick and choose the ones they want -- the ones that they see as directly contributing to their own bottom line and leave the fuzzier and more complex goals of KM behind.

KM is not just a menu of services. If it were, there would be very little purpose in separating it from the rest of business operations. On the other hand, I understand the desire to get a handle on exactly what is the business benefit of a program such as KM.

The analogy that came to mind at the time was to the difficulties involved in funding public education. Individual citizens do not get to choose whether they fund one service over another. There is a need to educate our young (reinforced by federal laws) that cannot be broken down into individual services.

At the same time, my own analogy betrays me because I don't believe the US educational system is well run or evenly remotely efficient in achieving its goals.

So how do you retain the proper perspective on KM's long-term goals and still make the program accountable? I don't claim to have the answer, but I know several things you shouldn't do:

• Don't "put KM in business terms" and measure it against short-term business objectives such as increased sales, shorter time to market, etc. In other words, ROI. Although KM practices influence many of these objectives, KM's contribution is only part of the story and cannot be accurately measured or, more importantly, can't be "adjusted" as management likes to do to increase the return.
• Don't measure KM value as "savings" against the potential cost if KM was not involved. This is extremely appealing (along the lines of "now it only takes five minutes to find a sample proposal to copy versus the day and a half it would take to write it from scratch"). However, the result is often a return so astronomical (200%, 300%, 400%) that no rational manager would believe it. And in fact they shouldn't. Because although the benefits are real, the alternative is false: without a viable KM environment, employees will find a workaround -- such as stealing from an inappropriate sample or leaving out sections (usually those needed to mitigate risk) to save time.
  

The fact is that there is a natural tension between business operations and complementary initiatives such as KM. (Others that come to mind are R&D, usability, sustainability, etc.) The larger goals of KM cannot be measured in standard operational terms, whether that be ROI, chargeability, time-to-market, etc. But that doesn't mean they can't be measured. It just means you have to be careful to measure them on their own terms.

For example, from a business perspective the role of communities of practice (CoPs) is to increase the flow of "deep" knowledge among practitioners in fields important to the business. Two ways to measure the impact of a community initiative would be to measure the level of participation in the CoPs or the level of interaction between members.

But you do have to be careful. CoPs are communities of practitioners -- a mutually beneficial network of peers. As soon as you talk about CoPs and convince management of their benefit, someone will try to strictly "align" the communities with the business objectives, either by changing the community's objective or by controlling (and only measuring) the individuals they believe belong in the community due to job role or organization.

You can encourage people to participate in communities, even measure and reward that participation. but you cannot require or control it. If you require participation, it is no longer mutually beneficial. It is then performed out of compunction and the culture and attitude will no longer be one of sharing to mutual benefit but complying (and consequently doing as little as possible to meet the requirement).

Achieving balance between maintaining the autonomy of KM initiatives and retaining enough connection with the business organization to garner support is a tricky business. One way to make the connection real is to measure based on KM objectives but report along organizational lines. For example, what percentage of the people in organization X are participating in communities vs. organization Y. This, plus the participation rates on specific communities allows you to turn the tables: it gives managers a clear metric (especially if you set a target goal) that they can work towards. But rather than "turning the dial" of KM funding or services, they can work to encourage the employees in their organizations to participate to raise the metrics for their organization or the communities they consider critical to their mission.