Project Example

Fishing Trip

440 W. Franklin

03/04/2009

Administration

ssh -p 2220 gomez.net.unc.edu

Get Ready ...

[hope@gomez ~]$ cd /opt/1q/lists
[hope@gomez lists]$ getTable -l 440-w-franklin.list -n $ROSTRR -prof RmonStatsTable -cFilter DroppedEvents,CRCAlignErrors,UndersizePackets,OversizePackets,Fragments,Jabbers,Collisions -dbo | grep ^1 | tr -s " " '\t' | awk '{ if ($3+$4+$5+$6+$7+$8+$9 > 0) print $0 }' | tee 440-a.out

Get Set ...

[hope@gomez lists]$ getTable -l 440-w-franklin.list -n $ROSTRR -prof RmonStatsTable -cFilter DroppedEvents,CRCAlignErrors,UndersizePackets,OversizePackets,Fragments,Jabbers,Collisions -dbo | grep ^1 | tr -s " " '\t' | awk '{ if ($3+$4+$5+$6+$7+$8+$9 > 0) print $0 }' | tee 440-b.out

Warning, diff barfs if the first line is different. Just add a dummy line if needed.

Go!

[hope@gomez lists]$ diff 440-a.out 440-b.out | tee 440-diff.out
45a46,63
> 172.29.158.57	2	0	2	0	0	0	0	0	
> 172.29.158.57	4	0	4	0	0	0	0	0	
> 172.29.158.57	7	0	4	89	0	89	0	0	
> 172.29.158.57	8	0	0	3	0	3	0	23971	
> 172.29.158.57	10	0	3	1	0	1	0	0	
> 172.29.158.57	14	0	2	0	0	0	0	0	
> 172.29.158.57	24	0	1	0	0	0	0	0	
> 172.29.158.57	27	0	1	0	0	0	0	0	
> 172.29.158.57	32	0	0	4	0	4	0	0	
> 172.29.158.57	35	0	2	0	0	0	0	0	
> 172.29.158.57	37	0	13	2	0	2	0	0	
> 172.29.158.57	39	0	1	0	0	0	0	0	
> 172.29.158.57	41	0	1	1	0	1	0	0	
> 172.29.158.57	43	0	162	2654	0	2654	0	0	
> 172.29.158.57	44	0	4	0	0	0	0	0	
> 172.29.158.57	45	0	4	0	0	0	0	0	
> 172.29.158.57	46	0	3	0	0	0	0	0	
> 172.29.158.57	47	0	1	1	0	1	0	0	
60,72d77
< 172.29.158.44	15	0	8	3	0	3	0	0	
< 172.29.158.44	16	0	9	2	0	2	0	0	
< 172.29.158.44	18	0	15	1	0	1	0	0	
< 172.29.158.44	19	0	3	0	0	0	0	0	
< 172.29.158.44	34	0	3	1	0	1	0	0	
< 172.29.158.44	36	0	124627	0	0	0	0	0	
< 172.29.158.44	37	0	13	4	0	4	0	0	
< 172.29.158.44	38	0	124693	0	0	0	0	0	
< 172.29.158.44	39	0	124698	0	0	0	0	0	
< 172.29.158.44	40	0	124613	0	0	0	0	0	
< 172.29.158.44	41	0	125995	0	0	0	0	0	
< 172.29.158.44	44	0	124616	0	0	0	0	0	
< 172.29.158.44	47	0	0	13910	0	13910	0	48691	
87a93,95
> 172.29.158.43	5	0	22	0	0	0	0	0	
> 172.29.158.43	16	0	3	181	0	181	0	0	
> 172.29.158.43	45	0	1	1	0	1	0	0

Filter that down

[hope@gomez lists]$ grep 172.29.158.44 440-b.out 
[hope@gomez lists]$ grep 172.29.158.57 440-a.out 
[hope@gomez lists]$ grep 172.29.158.43 440-a.out 
[hope@gomez lists]$

So 172.29.158.44 must have reset its counters ... odd ...
The other two switches went from no errors to many errors!
Reminder: the columns are DroppedEvents, CRCAlignErrors, UndersizePackets, OversizePackets, Fragments, Jabbers, and Collisions.

What's the worst?

Let's look at the very worst (large numbers, and look).

[hope@gomez lists]$ awk '{ if ($4+$5+$6+$7+$8+$9+$10 > 9999) print $0 }' 440-diff.out | grep ^">"
> 172.29.158.57	8	0	0	3	0	3	0	23971

23971 collisions on port 8 of 172.29.158.57 in 20 minutes is surprising!

What's also bad?

 [hope@gomez lists]$ awk '{ if ($4+$5+$6+$7+$8+$9+$10 > 9999) print $0 }' 440-diff.out | grep -v ^">"
< 172.29.158.44	36	0	124627	0	0	0	0	0	
< 172.29.158.44	38	0	124693	0	0	0	0	0	
< 172.29.158.44	39	0	124698	0	0	0	0	0	
< 172.29.158.44	40	0	124613	0	0	0	0	0	
< 172.29.158.44	41	0	125995	0	0	0	0	0	
< 172.29.158.44	44	0	124616	0	0	0	0	0	
< 172.29.158.44	47	0	0	13910	0	13910	0	48691

So ...

we might want to look at the undersized packets (fragments) and collisions on port 47 of 172.29.158.44 too.

[hope@gomez lists]$ aliasGrep -s 172.29.158.57 -n $ROSTRR -ports ge.1.8

------------
ctAliasTable
------------------------------------------------------------------------------------------------------------------------------------
DeviceIP        CreationTime        Ref   Port    MacAddress        VID  VLANName      Protocol Address            MarkedForDeletion
------------------------------------------------------------------------------------------------------------------------------------
172.29.158.57   05/16/2008_06:59:02 0     ge.1.8  00:60:B0:91:C4:BC 101  ITS           IP       152.2.203.41                     no
172.29.158.57   07/08/2008_14:15:57 0     ge.1.8  00:30:6E:FF:50:4E 101  ITS           IP       152.2.203.47                     no
[hope@gomez lists]$ nslookup 152.2.203.41
Server:		152.2.21.1
Address:	152.2.21.1#53

** server can't find 41.203.2.152.in-addr.arpa: NXDOMAIN

[hope@gomez lists]$ nslookup 152.2.203.47
Server:		152.2.21.1
Address:	152.2.21.1#53

47.203.2.152.in-addr.arpa	name = sp-hp3k-suite400.its.unc.edu.

It's probably an HP printer. They misbehave.

What else?

[hope@gomez lists]$ aliasGrep -s 172.29.158.44 -n $ROSTRR -ports ge.1.47

------------
ctAliasTable
------------------------------------------------------------------------------------------------------------------------------------
DeviceIP        CreationTime        Ref   Port    MacAddress        VID  VLANName      Protocol Address            MarkedForDeletion
------------------------------------------------------------------------------------------------------------------------------------
172.29.158.44   05/16/2008_06:14:45 0     ge.1.47 00:15:63:BC:84:16 3011 rtc-cisco-its IP       152.23.137.213                   no
[hope@gomez lists]$ nslookup 152.23.137.213
Server:		152.2.21.1
Address:	152.2.21.1#53

213.137.23.152.in-addr.arpa	name = dhcp00954.rtc.unc.edu.

Probably a VoIP phone. Hoo boy. What about the other ports with problems on that switch?

And?

[hope@gomez lists]$ aliasGrep -s 172.29.158.44 -n $ROSTRR -ports ge.1.36
------------
ctAliasTable
------------------------------------------------------------------------------------------------------------------------------------
DeviceIP        CreationTime        Ref   Port    MacAddress        VID  VLANName      Protocol Address            MarkedForDeletion
------------------------------------------------------------------------------------------------------------------------------------
172.29.158.44   05/16/2008_06:15:55 0     ge.1.36 00:15:2B:22:2C:69 178  WAP-ManagementIP       172.28.65.114                    no
172.29.158.44   09/12/2008_06:38:30 0     ge.1.36 00:1A:1E:C5:F9:38 178  WAP-ManagementIP       172.28.64.215                    no
172.29.158.44   09/12/2008_06:39:42 0     ge.1.36 00:1A:1E:C5:F9:38 178  WAP-ManagementIP       172.28.64.214                    no
[hope@gomez lists]$ aliasGrep -s 172.29.158.44 -n $ROSTRR -ports ge.1.38-41
------------
ctAliasTable
------------------------------------------------------------------------------------------------------------------------------------
DeviceIP        CreationTime        Ref   Port    MacAddress        VID  VLANName      Protocol Address            MarkedForDeletion
------------------------------------------------------------------------------------------------------------------------------------
172.29.158.44   05/16/2008_06:15:58 0     ge.1.38 00:15:2B:22:32:E3 178  WAP-ManagementIP       172.28.65.115                    no
172.29.158.44   05/16/2008_06:15:59 0     ge.1.39 00:15:2B:22:32:D9 178  WAP-ManagementIP       172.28.65.116                    no
172.29.158.44   05/16/2008_06:16:00 0     ge.1.40 00:15:2B:22:2C:2D 178  WAP-ManagementIP       172.28.65.117                    no
172.29.158.44   05/16/2008_06:16:00 0     ge.1.41 00:15:2B:22:34:A9 178  WAP-ManagementIP       172.28.65.118                    no
172.29.158.44   09/10/2008_06:58:32 0     ge.1.41 00:1A:1E:C5:F8:2A 178  WAP-ManagementIP       172.28.64.218                    no
172.29.158.44   09/12/2008_06:43:40 0     ge.1.39 00:1A:1E:C5:FA:00 178  WAP-ManagementIP       172.28.64.212                    no
172.29.158.44   09/12/2008_06:45:55 0     ge.1.38 00:1A:1E:C5:F8:30 178  WAP-ManagementIP       172.28.64.211                    no
172.29.158.44   09/12/2008_08:20:37 0     ge.1.40 00:1A:1E:C5:F9:F6 178  WAP-ManagementIP       172.28.64.243                    no
[hope@gomez lists]$ aliasGrep -s 172.29.158.44 -n $ROSTRR -ports ge.1.44
------------
ctAliasTable
------------------------------------------------------------------------------------------------------------------------------------
DeviceIP        CreationTime        Ref   Port    MacAddress        VID  VLANName      Protocol Address            MarkedForDeletion
------------------------------------------------------------------------------------------------------------------------------------
172.29.158.44   05/16/2008_06:16:01 0     ge.1.44 00:15:2B:22:32:E6 178  WAP-ManagementIP       172.28.65.113                    no
172.29.158.44   09/12/2008_06:41:28 0     ge.1.44 00:1A:1E:C5:F9:88 178  WAP-ManagementIP       172.28.64.213                    no

And (2)

Oh, that's lovely. We have a ton of CRC alignment errors on ports with wireless access points. It's not like wireless doesn't have enough trouble on its own, but now the WAPs have more problems once they're on the wired side of the network?

Administration

Warning: run Wireshark on a computer in the main VLAN for your project, not tcpdump on gomez!
Remember what VLANs do ... that's why.

Packet Capture

[hope@gomez lists]$ time sudo tcpdump -c 200 -nn -w 440.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
200 packets captured
200 packets received by filter
0 packets dropped by kernel

real	0m6.518s
user	0m0.006s
sys	0m0.006s
[hope@gomez lists]$ tcpdump -r 440.cap > 440.decode
reading from file 440.cap, link-type EN10MB (Ethernet)
[hope@gomez lists]$ grep -c arp 440.decode 
162
[hope@gomez lists]$ grep -c aruba20.net.unc.edu 440.decode 
29

Packet Capture Analysis

[hope@gomez lists]$ grep -v arp 440.decode | grep -v aruba20.net.unc.edu
16:03:50.346532 802.1d unknown version
16:03:51.410847 00:0f:8f:34:7c:81 > 01:00:0c:cc:cc:cd snap ui/C len=39
16:03:51.637424 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:64:9c:41:16, length: 278
16:03:52.352116 802.1d unknown version
16:03:53.415777 00:0f:8f:34:7c:81 > 01:00:0c:cc:cc:cd snap ui/C len=39
16:03:54.346984 802.1d unknown version
16:03:55.420780 00:0f:8f:34:7c:81 > 01:00:0c:cc:cc:cd snap ui/C len=39
16:03:56.285919 IP its-net00038.dhcp.unc.edu.1240 > 255.255.255.255.7100: UDP, length 40
16:03:56.347056 802.1d unknown version

Not much in the way of loose ends (that's good), but snap??? and weird broadcasts. 802.1d is switches talking to each other. "UDP port 7100" from Google suggests this is x-font-service (for X11).

ExampleProject

Project Example

Fishing Trip

440 W. Franklin

03/04/2009

Administration

Get Ready ...

Get Set ...

Go!

Filter that down

What's the worst?

What's also bad?

So ...

What else?

And?

And (2)

Administration

Packet Capture

Packet Capture Analysis