Sharkfest 2013 - Correlating Traces from Multiple Tiers (by Paul Offord)
Netmon, Not Your Parents Analyzer (by Tony Fortunato)

Using Good Old Syslog When Troubleshooting (by Tony Fortunato)

When working on an intermittent problem that results in a device either rebooting on its own or a technician restarting it manually, usually results in loosing the information contained in its log.  Most system’s logs are stored in memory by default.  In many cases you can configure devices to use some of its storage but gets to be a bit messy and complicated if you want to track a few devices.

That’s where good old syslog comes in. Syslog is a standardized way to send a device’s log entries to a central device running a syslog collector for further analysis.

Some good syslog collectors will provide some cool features like email alerts on specific messages, the ability to run scripts and the ability to recognize non-standard syslog formats.

In this specific example we had a Cisco 2800 router intermittently reboot on its own or become non-responsive, requiring a technician to manually reboot it.  When the device is unresponsive we couldn’t even get a login prompt with the console cable or SSH. 

So I thought, maybe we can configure the router to send its system messages to a computer so we can see if there are any clues as to what happened before the router died or rebooted.

As you will see in the video, the syslog messages revealed that the router had many memory allocation issues.

Now armed with this new information, it looks like the router might have bad memory, no enough memory or an IOS memory leak.

Unfortunately I am currently working on this issue and will provide an update as soon as I figure it out.

The software I used can be downloaded from here; Syslog Watcher




Continue reading other LoveMyTool posts by Tony Fortunato »