A better way to visualize HTTP packets for analysis!
One of the more practical, and yet fundamental uses of packet capture analysis in todays networks is examining HTTP flows to isolate problems with the protocol or underlying network interactions. However, 1) getting accurate and 2) easily accessible/understandable data for a successful review can vary wildly depending on the tools used, particularly when it comes to timing information. This can be a difficult task but there is some new technology that can help. The QACafe Team has come up with a tool that will help you in this investigation with T-Shark by giving you a higher plane of useful visualization. T-Shark is an open source terminal based on Wireshark - http://www.wireshark.org/docs/wsug_html_chunked/AppToolstshark.html , that allows for easily programed packet capture interactions.
TShark - A Wireshark tool!
The HTTP request analysis tool from CloudShark, http://cloudshark.org/ , contains a solution for both problems. This tool takes advantage of a powerful and simple web interface which makes collaboration on HTTP data faster and easier while interacting with T-Shark.
T-Shark’s “-z” flag will give you statistical reports of any packet type you want. In the case below we have set the filter for http_req. Building a chart of requests per host this way is relatively simple – the real trick is twofold; adding contextual information and computing the request response time in an accurate way. Remember REAL TIME can only come from using a TAP, SPAN and other access technologies will not provide even near real time.
Here is an example of an http_req stats report:
CloudShark takes tshark’s http request reports and transforms them into context sensitive graphs that can be shared with colleagues directly via a linked URL. This new http tool provides graphs of host summaries, detailed requests-per-hosts tables, and graphs of http response codes to see where the more egregious errors may lie. We use the term “context sensitive” to describe the ability to not just view the graph, but also interact with it to dig deeper.
For example, take a look at this graph of traffic generated while visiting lovemytool.com. If you click on the “response codes” view, you’ll see detailed pie charts.
Clicking on any of the individual response code types will take you immediately to a filtered capture of the packets in question for a closed review.
The next trick is getting accurate timing information out of the report. Unfortunately, tshark’s statistical reports don’t do this on their own - some behind the scenes techniques are necessary to compute them. CloudShark does this by:
- Searching through all streams to find each HTTP request
- After having a list of all requests, search through all the responses
- Match each response to their respective request
- Calculate the time needed to complete a full response after a full request
- Second order data (minimum, maximum, average) can then be calculated on hosts that receive multiple requests.
We say “full response” and “full request” due to the possibility of fragments and retransmissions in the stream; some additional work is necessary to get around this. This entire process makes the response times as accurate as they can be within the limits of tshark - which, if you’re using anyway, is about as close as you can get - Wireshark timing accuracy.
We recently offered up an HTTP specific packet capture challenge at this year’s Sharkfest.
What other problems do you solve on a regular basis with accurate HTTP analysis?
Jason Walls is Director of Technical Marketing for QA Cafe/CloudShark. Jason has been involved with computer networking since 2001, focusing on broadband access networks, customer premises equipment, and home networking. He is currently the chair of the BroadbandHome Working Group at the Broadband Forum. A packet geek like most of his peers, Jason spent ten years in trenches of network testing and analysis at the University of New Hampshire's InterOperability Laboratory before coming on board as the Marketing Director at QA Cafe.
Recent Comments