Measuring Device Recovery Time

 Figuring out how long it takes a device to recover from a reboot or power outage sounds trivial, except for those of you who are troubleshooting remotely or managing a monitoring system. This also helps to figure out how long it takes for a redundant link to fail over, routing, etc…

When I speak to clients about this simple exercise, their common response is of course “How do I do that”?

The obvious methodology is to start a stopwatch and power on the device. But what do you do if its remote and how do you prevent people from hitting the start too soon or stop too late?

Over the years I figured out a pretty simple way to do this using my favorite free utility, hrping (https://www.cfos.de/en/ping/ping.htm)

Let me start with the basic disclaimer that there are a ton of ways to do this, even with hrping, so feel free to share how you do it. Before I get to the exercise, let me explain why I chose to use hrping. Hrping pings every 500 milliseconds but allows me to ping with an interval option (-s), where I can define how often the utility will ping (in milliseconds). Microsoft ping pings at a 1 second interval and you can’t change it.

Before we start, please note any switch configuration details that might prevent a device from immediately getting online, like Cisco’s portfast spanning tree command that will shorten or bypass the STP timer and gets the port up quicker. Also note if the device is cabled or wireless.

Methodology;

I simply use hrping and ping a device, reboot it and note the timeout message. In this case I used the following command hrping 10.44.10.1 -t -s 100 -T

· -t continuous ping

· -s 100 ping every 100ms

· -T display date and time in results

You could optionally add >log.txt if you want the output to go to a file, or if there is a lot of output. Just note that you will not see any output and will have to break from the command when the device is back up (use a separate command prompt ping to determine)

Here is an example of a successful ping and one that failed

2022-10-25 15:44:17.261: From 10.44.10.1: bytes=60 seq=005f TTL=64 ID=da27 time=1.152ms

2022-10-25 15:44:17.370: Timeout waiting for seq=004c

Here’s the trick to save some time, I add the following to the hrping command; | find “Timeout waiting”

This will only display failed pings making it easier to document how long the device was down. Here is an example of a wireless camera rebooting.

hrping 10.44.10.39 -t -s 100 -T | find "Timeout"

2022-10-25 17:09:03.671: Timeout waiting for seq=03e4

2022-10-25 17:09:03.671: Timeout waiting for seq=03e5

2022-10-25 17:09:03.671: Timeout waiting for seq=03e6

…..

2022-10-25 17:10:01.179: Timeout waiting for seq=0625

2022-10-25 17:10:01.179: Timeout waiting for seq=0626

2022-10-25 17:10:01.179: Timeout waiting for seq=0627

In this example the device was not reachable from 17:09:03 to 17:10:01 , or 58 seconds and since I used 100 ms as my ping interval, this is accurate up to 100 ms, which is good enough for me.

Heres comes the table of results from devices in my office

DEVICE

RECOVERY TI

Server churchill	Recovery Time (secs)
Phone	44
Printer .86	67
Switch Dexter	175
Wireless Access Point (Stewie)	54
Router .1	51
Server (Churchill)	187

When Done Right, Network Visibility Can Pay For Itself

 

Over the years, I’ve met people that seem to think that investing in network visibility is either unnecessary, too expensive, or too complicated. None of these preconceptions could be further from the truth.

When done right, a visibility architecture (consisting of taps, packet brokers, and network tools) is absolutely essential to your ability to detect security threats and optimize both network and application performance. You can’t prevent or fix what you can’t see. This lack of visibility is why successful security attacks go undetected for days, weeks, and months. Literally, no one sees them.

Show the Value of What You Do by Patricia and Jack Phillips - A Review

 

Bragging at work, even the humble kind, is generally not looked upon favorably. In performance appraisals where I have been required to rate myself, I feel I am treading on dangerous ground – too high and I’ll appear arrogant, too low and my superiors will suspect I know something that they don’t. In Show the Value of What You Do, authors Patricia and Jack Phillips offer a detailed process for showcasing the tangible value of your work product. Their short (~ 140 pp.) book makes use of numerous stories from their consulting practice to illustrate each step. Whether you are proposing a new program to senior management, or merely calling attention to your own value, there are many useful tips to be found here.

The best way to avoid being dismissed as a braggart is to back up claims and promises with compelling, credible evidence. The book is aimed at individuals trying to rescue a stalled career as well as team leaders who are gathering resources for a wider reaching goal. Though we live in a “show me” world”, the process for building a compelling case is often unclear. The authors state right up front that they will show individuals and teams how to measure success, how to change their thinking from merely completing activities to making significant, impactful investments, and how to project results to obtain future support.

We get a first glimpse of how difficult projects can be handled with the story of a Hospital Chaplain, a profession which most of us believe should be immune from bottom line accountability. The ultimate value of this activity will lie with the answer to “Was it worth it?”, and clear measures of patient outcomes are the aim here. The importance of good leadership is highlighted, as buy-in from those involved is key to a successful result. Critics can be deterred by considering several perspectives – hospital accountants may be focused on length of patient stays, while friends and family might care about less tangible things like spiritual support. This first chapter provides an overview of the process for showing value.

In the ensuing 5 chapters, the authors walk through the process in detail, covering Why?, How?, What?, How Much? and then wrapping up with “What’s It Worth?”. Connecting a project to a business need, and thus establishing impact, is an essential place to start. Each of us needs to be aware that we operate with competing mindsets - either an inward facing one (our own personal goals) or one that is outward facing (collective results and the challenges faced by others). An excellent way of answering “Why?” and thereby assessing impact is to ask, “What happens if we do nothing?”. Even with good intentions and planning, some stories still don’t end as expected.

In April of 2018, Starbucks Coffee responded to a racial bias incident at one of its locations by closing over 8,000 stores for an entire day, giving up millions in sales, and providing special training to its employees. Pundits noted that numerous studies show this type of training doesn’t actually work. Although the story drew lots of positive press coverage, the actual ROI has never been clear. The book suggests numerous ways of measuring tangible and even intangible results which can help in improving similar projects.

Some of the concepts introduced in the book, like the measurement of those intangibles, can seem a bit nebulous, but there is always a story to relate to, followed by some diagnostic tools to stimulate thinking and planning. Examples like building a traffic round-a-bout, improving police-community relations, or the Starbucks story above are clear and introduce situations we can all relate to. Objectives, critical for guiding any project, are explained using concrete definitions and multiple examples. One of the data-gathering considerations that caught my attention was “culture” - associated with trust, openness, transparency, and inclusivity – that can have a profound effect on information collection.

A very relevant contemporary example where culture plays a role is in the on-boarding process for millennials. Yet another timely example deals with the business case for working from home. It’s one thing to acknowledge that the world is changing and businesses need to adapt, but it’s not always obvious how to proceed. The book helps frame the process as an investment, and not just another cost.

The authors conclude with a discussion guide which can jump-start team collaboration and buy-in, both of which are important elements in a successful project. It is often said that “change is inevitable, but progress is optional”. I would add that change can be frightening but Show the Value of What You Do will benefit both individuals and groups in navigating the process.

Author Profile - Paul W. Smith - leader, educator, technologist, writer - has a lifelong interest in the countless ways that technology changes the course of our journey through life. In addition to being a regular contributor to NetworkDataPedia, he maintains the website Technology for the Journey and occasionally writes for Blogcritics. Paul has over 40 years of experience in research and advanced development for companies ranging from small startups to industry leaders. His other passion is teaching - he is a former Adjunct Professor of Mechanical Engineering at the Colorado School of Mines. Paul holds a doctorate in Applied Mechanics from the California Institute of Technology, as well as Bachelor’s and Master’s Degrees in Mechanical Engineering from the University of California, Santa Barbara.

Ok, I'm Guilty..

A few years ago my garage door died after a lightning storm. Lesson learned, i now have a whole home surge protection system installed. Since I was on the road, my wife called "the professionals" to install a new unit.

When I returned home, I was pleasantly surprised that there was one less thing for me to do. The first time I used it I knew something wasn't quite right but I had no time or energy to look into it. And like all IT implementations "It works, so move on.".

Over the past few years, I would comment to my wife that the garage opener looked misaligned but was quickly reminded that i had a ton of other stuff to worry about.

One day i went to use the garage door opener and heard a "crack and snap". Watching the chain dangling was a sure sign that something wasn't right ;)

I got the ladder to check things out hoping the chain just jumped the sprocket and I could fix it quickly. Nope... to my surprise, the shaft that the sprocket was connected to had snapped. upon closer inspection, I could see a pile of metal filings (see photo below) indicating that the shaft had been wearing away for a while.

This reminded me of the countless network equipment, applications, computer configurations and installs where I'm always preaching to make sure you test and check things out before you leave.

In this case, I am willing to accept being guilty, lazy, and putting off something I knew wasn't working correctly.

I purchased a new garage door opener and noticed all the shortcuts the 'professionals' took with the previous installation explaining the misaligned unit. Now I know why my wife and I prefer to do our own home renovations or check any work done by anyone other than us.

Cost of being lazy and procrastinating; $350 and a half a day to purchase/install the new unit.