Editor’s Note - Again, I am very pleased to introduce to our readers, my friend and a very trusted technologist Tom Tosh. ToTom has agreed to write a series of articles about his unique analytical approach to helping clients get more value from their network management tools. The concepts and techniques have been honed from his many years of hands-on experience in helping find the problems that have plagued and solutions that were needed by many network managers and companies that he has helped. His company, Chi-Metrix (pronounced: “key metrics”) and his efforts are all based on these “Metrics” that he will be sharing with you in these articles.I hope you enjoy reading this series as I have and I wish everyone Great Success … Oldcommguy.
Part 1: Tools, Process and Metrics for Network Managers
Part 2: Measuring for Business Impact – The “Level 4 Measurement”
Measuring for Business Impact – The “Level 4 Measurement” (by Tom Tosh)
In this series of articles, my goal is to broaden the focus on tools beyond just features and prospective benefits alone, to consider the environment and processes in which tools are used to provide genuine, meaningful results. I would like to share and advance to readers, some concepts on how to better evaluate and get more out of the tools and services you may have invested in, or may be considering.
If you can’t measure, you can’t really manage. A site dedicated to tools should ultimately be keenly interested in the overall effectiveness and the success of the network manager. (And how do we propose to measure that?) Our very approach to measuring things should also be subject to scrutiny and assessment, since nothing can lead to greater waste of effort and resources than measuring the wrong things, or measuring the right things in the wrong way. We are all familiar of “GIGO”. Garbage in, Garbage out, so we want to make sure what answers we seek and data acquire are valuable sources of information.
If we are going to get the most out of the investments made to managing a network, I believe we have to begin by looking at our own approach to measuring things and consider how that approach aligns to four tightly-linked dimensions, or levels. Of equal importance is the ability to evaluate the assessments that others present to us.
The Four Dimensions or “How to Love a Tool”
The systematic approach that the IBM Corporation has used to design and develop training, in order to have a positive impact on business results, gave me my first exposure to the concept of four levels of measurement. The approach has proven very helpful to me and my clients for over twenty years, and I hereby acknowledge my debt.
Four dimensions at a glance.
Level 1 – Emotional / Aesthetic (How do you feel about the tool/process/etc.?)
Feelings and attitudes are present during every minute of our waking life. Just as each person puts trust in their feelings, if you are a designer of a product or a service, you learn to seek out and take seriously the feelings of others. Feelings are always based on something and, taken in this light, even misperceptions and misunderstandings contain valuable information.
We are all familiar with the typical formal instrument for collecting Level 1 measurements: the opinion/feedback survey. And we are likely to agree that, often times, we don’t always let our true feelings be known when we are asked how we liked something. Likewise, we are prone to remain skeptical when presented only with positive feedback.
All that being said, there is nothing more powerful than creating a truly delighted customer, and companies invest a great deal to reach that lofty goal. Satisfied or not satisfied, Level 1 results may be the most significant initial indicator of all that takes place at the other three layers.
As it pertains to network tools, a well-designed function will cause us to feel a certain way. Likewise, the requirements of our work may reveal areas where a certain tool is woefully inadequate and a pain to use. Using poorly designed tools often turns routine tasks into drudgery and pain. A vendor can achieve a clear competitive advantage over another by finding the pain points and relieving them. Looked at another way, the organization whose analyst is tied up for several hours producing a result is not as productive as one using a different tool where the same task takes a fraction of the time.
Level 2 – Practical application (Are you actually using the tool?)
An alarm bell should go off when a network management tool sits on a shelf, or in the shipping container, gathering dust. Most network management tools have functions that make them appropriate for regular use. And, to state the obvious, a tool that is used on a regular basis returns greater value to the organization than one that sits on a shelf.
Level 2 measurements are the “glue” that ties all of the other three levels together. It is this level that answers detailed questions of how results are achieved. The questions encompass the key metrics about the tool itself:
• What does it cost to purchase?
• How easy is it to deploy?
• How long does it take to learn to use its features?
• How easy is it to use?
• How valuable are its readings, and are they presented effectively?
• How much time and effort is required to maintain it?
The questions above are closely tied to a concept of workflow: the series of definable steps needed to produce a result which has value. Future articles will delve into a discussion of the workflow of some key network management activities, and I will present some things which help distinguish workflow in a purely reactive environment from a more proactive workflow.
I believe that many of us know that the condition I described in the first article: Vendors imply that all we need to do is purchase the tool and deploy it and, before long, network management nirvana will be ours. However, the reality is that a large gap exists between implementation of a tool and deriving high value results from it. Relatively few network management staffs, in my experience, attain a condition anywhere near the promise.
Proper training and coaching can certainly close this “value gap.” (However, most one-size-fits-all training programs do little to address the gaps that are unique to every organization or business.) People are very often not aware of what the tool can do for them; the person on staff who recommended the tool in the first place may have moved on to another organization. Many people aren’t even aware of what they don’t know. (Perhaps this is why network analysts always chuckle knowingly at the saying, “In the land of the blind….”) Consequently, many of those who attempt to use a particular tool may not have the knowledge to interpret the results of a tool reading, much less question its validity, and so miss the full value.
Level 3 – Departmental impact (What tangible results has using the tool brought about – results that we can measure at our local level?)
Our tools-testimonials are often essentially Level 3 assertions, and supply us with tempting promises to the multitude of important questions we all have, like the following: Has mean-time-to-repair decreased? Are we better able to predict how changes to the network will impact current performance? Are user complaints about performance decreasing? What is normal performance? Are we better able to note and correct degrading conditions before they impact our user community?
If you are following along, you will likely better follow my jargon when I say that a good Level 3 indicator of a training program for a given tool is a measurable increase in how often and how effectively the tool gets used (a Level 2 indicator for the tool itself). This, in turn, will also likely cause an increase in the staff’s satisfaction with the tool. (Right, our Level 1 indicator.) So while each level can be drawn out and analyzed separately, they are always linked together as a part of system.
But, make no mistake about it, most advertisements skip over the gritty details of Level 2 and are more often Level 1-to-Level 3 leaps of faith.
Level 4 – Organizational impact (What impact do the Level 3 results have on the larger organization? What is the impact at a global level?)
In other words, to paraphrase the title of a famous article which appeared in the Harvard Business Review a few years ago: Does IT really matter?
Level 4 measurements can be said to be the Holy Grail of any measurement system. Vendors of network tools are always looking for compelling ways to show the real value – the ROI – of the tools they sell. And many times, like the Grail, trying to come up with a dollars and cents number for the business contribution made by a particular tool is the stuff of a very lively imagination. And yet, while quantification can be elusive at best, more often than not a link between Level 3 and Level 4 results is quite evident.
Consider the online brokerage firm whose internal network developed a problem whereby traders are being kicked out in the middle of a trade. The problem persists for several days until the firm decides to call for help. Once engaged, the outside expert is able to locate the cause and the problem is fixed in a matter of a few hours. Business value could be represented as the decrease in dissatisfied customers had the problem been solved within a few hours after its first occurrence.
On the other hand, consider the money spent on tools that are not used. Can this be anything other than a net negative: money wasted by the organization?
Are the users of applications – and the managers of the various departments within the organization – happy with the performance of the applications they depend upon? (I hear you saying, “Happy, Tom? That sure sounds like a Level 1 measurement to me.” And so it is!)
So let’s frame the question in a way that makes it more substantive: Why are they so happy? Does the increased performance of applications enable users to be more productive, and, if so, by how much? Although you would be right in calling this a Level 3 measurement, can you see how easily it can be made to correlate with an impact on the overall goals of the business or organization? The key point here is that IT has no way of knowing by itself what the dollar value of the impact is; only the leaders of the departments which represent the user communities – particularly the customer-facing ones – can provide that information.
Summary – and a case example:
Let us now process the high points of the story from my first article in the context of the Four Levels:
- A large commodities-trading firm experienced performance problems which started impacting the timeliness of display-panel updates and, in turn, the timeliness of trades. (Level 3, with definite link to Level 4)
- The value being exchanged on the floor was many millions of dollars per minute (Level 4)
- The performance problems were not a sudden occurrence, but represented a slowly degrading situation that grew steadily worse over a period of many months. Level 1 readings from the trading floor indicated growing dissatisfaction and action was finally taken when the business was impacted. By this point, high negative Level 1 readings were being received from the executives of the firm.
- The root cause was diagnosed fairly quickly but the recommended actions that were needed to correct the situation were resisted because, it was feared, that the changes might make things become even worse. (Level 1)
- A tool and process were proposed that would simulate a trading floor network segment, so that the recommended change could be fully tested and validated. One of the tests involved a simple process that could be rerun before and after each subsequent change (Level 2) to the main networks so that improved performance could be measured and recorded.
- The components involved in the tool/process were essentially off-the-shelf and thus presented no additional cost to the client, which pleased the network operations department greatly (Level 1).
- One part of the testing involved moving a file of a certain size between various segments of the trading floor and production networks. The timing of the file transfer was initially around two minutes and soon became the metric that everyone looked to (Level 1) in order to measure progress. (Level 3). Running the test became a twice-daily, standard procedure (Level 2).
- Over the next few weeks, as the changes were gradually implemented and the timing of the key metric fell from two minutes towards a final reading which settled around 20 seconds (Level 3), the corresponding impact on trading floor operations (Levels 3&4) gave everyone, including the non-technical executives, a high degree of confidence in the procedure, and a high level of satisfaction with the outcome (Level 1).
Elements of the case can easily be opened up for further value analysis: For example, the trading firm had the tools they needed to have diagnosed the problem before it impacted trading floor operations; why had the problem not been detected and resolved prior to the meltdown?
Two important points:
1. The assurance of a positive Level 4 impact is most often directly linked to the improvement in Level 3 metrics.
2. The absence of solid Level 2 information and metrics means that getting to Level 3 and 4 results requires a leap of faith. The key elements: cost, effort, learning curve, methods, etc. all lie at Level 2 – the realm of the technologist.
Finally, I invite you to go to any article on this site, pick out the statements that you find most significant, and determine which level of measurement each of those statements represents. (Yes, that would be a Level 2 measurement for this article.) As we start to consider how to get better Level 3 and 4 results out of our network management tools and processes, we can reflect on how important it has become to first distinguish and identify the interrelationship of the Four Levels.
Part 1: Tools, Process and Metrics for Network Managers
Part 2: Measuring for Business Impact – The “Level 4 Measurement”
Author Profile - Tom Tosh is a seasoned trainer and proven senior level consultant and analyst. He spent the last 10+ years at Network General where he was there Senior Consultant and Project Manager. He has recently founded Chi-Metrix where Tom prides himself on bringing his considerable skills to help network customers with their network issues. From planning to troubleshooting, Tom is one of the best and is a long-time Oldcommguy’s associate and a proven solution focused technologist!
Tom’s skills, training and experience allows him to -
- Diagnose network and application performance issues.
- Perform network assessments, health checks and application baselines
- Provide coaching and training to help IT groups make better use of network tools.
- Derive actionable information from captured trace files
- Evaluate effectiveness of problem management practices (per ITIL)
- Develop visibility assessments to better target investments in tools
- Design and manage projects related to application rollouts and network changes.
Tom has served a wide variety of clients (corporations and non-profit organizations, as well as local, state and federal government agencies) all over North America, Australia and Japan.
Tom can be reached at - tom.tosh (at) chi-metrix (dot) com.
Recent Comments