quick and dirty way to analyze IDS traffic using ngrep

This article is from SANS Diary

http://isc.sans.edu/diary/User+Agent+007/11455

Published: 2011-08-26,
Last Updated: 2011-08-26 00:11:28 UTC
by Daniel Wesemann (Version: 1)

Recently, while conducting an audit at a financial services company, I decided to verify their claim that their "desktop build is standardized" and "no other devices are on the network". The network team provided access to a SPAN port on their Internet uplink, where I attached my pen-test workstation to take a look.

$sudo ngrep -qt -W single -s1514 -d eth0 -P~ 'User-Agent:' 'port 80'

"ngrep" works like grep, but on network traffic. Thus, the above command digs through everything on port 80 (http) that the span port provides, and searches for the string "User-Agent:", which commonly contains the "signature" of the web client making the access. A little bit of cleanup was needed to make the output usable:

| sed 's/.*User-Agent/User-Agent/' | sed 's/~.*//' | sed '/^$/d'

This takes care of empty lines, and throws out everything that isn't part of the User-Agent: string.  Collect the output into a file for a while, and then tally:

$cat output.txt | sort | uniq -c | sort -rn

And lookie, we ended up with about 80 distinct user agents. In only five minutes of traffic. Well, so far for "standardized desktop build" and "nothing else on the network". Among the user agent strings seen were

User-Agent: Mozilla/4.0 (compatible; Lotus-Notes/6.0; Windows-NT)
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15 (.NET CLR 3.5.30729)

Hmm, peculiar, some users are surfing with IE7 on Windows XP, while others are using an oooold version of Lotus Notes, and again others are using a vulnerable version of Firefox ??

User-Agent: Apple-iPhone3C1/812.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
User-Agent: BlackBerry9000/5.0.0.822
User-Agent: BlackBerry9700/5.0.0.656

A couple of mobile devices ... with what looks like a Windows7/IE9 system thrown in for good measure. The mobile devices turned out to be most interesting, because unless there is a WiFi gateway hooked into the corporate LAN, these devices usually surf via the mobile phone network, and shouldn't show up in the company's outbound Internet traffic. Guess what we found a couple minutes later ...: a little unauthorized wireless network extension, using WEP and the company name as SSID. Duh...!

And,  last but not least, we found some odd ducks that certainly warranted a closer look ..:
User-Agent: core
User-Agent: n1ghtCrawler
User-Agent: curl/7.8.1 (sparc-sun-solaris2.6) libcurl 7.9.6 (OpenSSL 0.9.6c)
User-Agent: Mozilla/4.0 (banzai)

Moral of the story: While your IDS probably alerts on "unusual" User Agent strings, it might nonetheless be a good idea to check out the full set of client applications that you have communicating with the Internet. The "User-Agent" string isn't failsafe, but it's a good start. You never know, you might just uncover a Secret (User) Agent who is busy squirreling away your data.

If you have other clever ways of auditing the user agent strings on your perimeter, please share in the comments below!

Comments

I'm glad you brought up odd user-agents, because I've been stuck searching for a user-agent I saw on my network:

USER-AGENT: TSA Browser Agent

My google-fu is lacking because the only thing I found that dealt with this user agent (and not the government agency, TSA) was a snort signature.

Do you believe this is in fact the user agent for the Target Saver spyware?

http://doc.emergingthreats.net/bin/view/Main/2001871

posted by Mike, Fri Aug 26 2011, 12:36

You can put multiple actions into a single sed command:

| sed 's/.*User-Agent/User-Agent/
s/~.*//
/^$/d'

Good diary entry about a simple way to do some good intelligence.

posted by Nit, Fri Aug 26 2011, 13:46

You can use tshark and avoid all the sed stuff:

$sudo tshark -f "port 80" -R "http contains \"User-Agent:\"" -Tfields -e http.user_agent > output.txt

$cat output.txt | sort | uniq -c | sort -rn

Sample output:

132 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0
13 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.16) Gecko/20101130 Ubuntu/9.10 (karmic) Firefox/3.5.7
1 Microsoft-CryptoAPI/6.1

posted by Franck G., Fri Aug 26 2011, 14:06

Blue Coat Proxy (Or Bluecoat Reporter) administrators can get this pretty easily as well.

In Blue-Coat Reporter - There's a built-in report for User-Agents.

Or, if you want to watch real-time action flying by, you can go into your Blue Coat and start a tail in Access Logging. You'll be doing a lot of scrolling this way, but it shows the required information.

posted by Jeff H., Fri Aug 26 2011, 14:22

This has given me an idea..for finding rogue access points.

Search for iPhone or iPad user agent strings with source address which you know should only be wired and you probably have yourself a rogue AP there.

Just ran this report on my Bluecoat and found a few possibles.

posted by Matt R, Fri Aug 26 2011, 17:19