|
Classes
FinalExam
Troubleshooting
edit SideBar
|
ProjectExample
ProjectExample as slideshow
Fishing at 440 W. Franklin
SNMP, mostly RMON
[hope@gomez ~]$ cd /opt/1q/lists
[hope@gomez lists]$ getTable -s 172.29.158.55 -n ${ROSTR}12 -prof RmonStatsTable -dbo
- interesting fields are Key or Index (can't omit that anyway), DroppedEvents, CRCAlignErrors, UndersizePackets, OversizePackets, Fragments, Jabbers, Collisions
[hope@gomez lists]$ getTable -s 172.29.158.55 -n ${ROSTR}12 -prof RmonStatsTable -cFilter DroppedEvents,CRCAlignErrors,UndersizePackets,OversizePackets,Fragments,Jabbers,Collisions -dbo
- this has got to be easier/shorter! (note the
tr -s " " '\t' step next)
[hope@gomez lists]$ getTable -l 440-w-franklin.list -n ${ROSTR}12 -prof RmonStatsTable -cFilter DroppedEvents,CRCAlignErrors,UndersizePackets,OversizePackets,Fragments,Jabbers,Collisions -dbo | grep ^1 | tr -s " " '\t' | awk '{ if ($3+$4+$5+$6+$7+$8+$9 > 0) print $0 }' | tee 440.out
- run the same command later, but use a different filename!
[hope@gomez lists]$ diff 440-am.out 440-pm.out
[hope@gomez lists]$ diff 440-am.out 440-pm.out
1c1
< 172.29.158.11 11 0 0 31921 0 31921 0 100793
---
> 172.29.158.11 11 0 0 31921 0 31921 0 100794
14,19c14,19
< 172.29.158.20 10 0 0 5 0 5 0 2829
< 172.29.158.20 11 0 0 1903 0 1903 0 7258
< 172.29.158.20 12 0 0 211 0 211 0 1153
< 172.29.158.20 13 0 0 10 0 10 0 49
< 172.29.158.20 14 0 0 27 0 27 0 121
< 172.29.158.21 2 0 88 11 0 11 0 0
---
> 172.29.158.20 10 0 0 5 0 5 0 2960
> 172.29.158.20 11 0 0 1903 0 1903 0 7263
> 172.29.158.20 12 0 0 212 0 212 0 1154
> 172.29.158.20 13 0 0 10 0 10 0 54
> 172.29.158.20 14 0 0 27 0 27 0 127
> 172.29.158.21 2 0 89 11 0 11 0 0
27c27
< 172.29.158.24 23 0 0 0 0 0 0 3746
---
> 172.29.158.24 23 0 0 0 0 0 0 3948
28a29,39
> 172.29.158.25 16 0 1 0 0 0 0 0
> 172.29.158.25 18 0 1 0 0 0 0 0
> 172.29.158.25 19 0 1 0 0 0 0 0
> 172.29.158.25 20 0 2 0 0 0 0 0
> 172.29.158.25 21 0 1 0 0 0 0 0
> 172.29.158.25 22 0 3 0 0 0 0 0
> 172.29.158.25 23 0 2 0 0 0 0 0
> 172.29.158.25 5 0 0 8 0 8 0 0
> 172.29.158.25 6 0 0 7 0 7 0 0
> 172.29.158.25 8 0 2 7 0 7 0 0
> 172.29.158.25 9 0 1 11 0 11 0 0
55,57c66,68
< 172.29.158.44 10 0 136 0 0 0 0 0
< 172.29.158.44 15 0 2 1 0 1 0 0
< 172.29.158.44 47 0 0 486 0 486 0 1485
---
> 172.29.158.43 16 0 2 0 0 0 0 0
> 172.29.158.43 31 0 1 0 0 0 0 0
> 172.29.158.43 5 0 1 0 0 0 0 0
60,61c71,72
< 172.29.158.45 31 0 0 1 0 1 0 5754
< 172.29.158.49 40 0 0 4 0 4 0 10929
---
> 172.29.158.45 31 0 0 1 0 1 0 5996
> 172.29.158.49 40 0 0 4 0 4 0 11205
63c74
< 172.29.158.51 13 0 0 604 0 604 0 6938
---
> 172.29.158.51 13 0 0 624 0 624 0 7055
65c76
< 172.29.158.55 12 0 21 13 0 13 0 0
---
> 172.29.158.55 12 0 34 22 0 22 0 0
67c78
< 172.29.158.55 7 0 0 1098 0 1098 0 0
---
> 172.29.158.55 7 0 0 1101 0 1101 0 0
75c86
< 172.29.158.59 43 0 0 2 0 2 0 15595
---
> 172.29.158.59 43 0 0 2 0 2 0 15913
Other SNMP Games
[hope@gomez lists]$ getTable -s 172.29.158.55 -n ${ROSTR}12 -prof ctCDPNeighborTable -dbo -cFilter MAC,IP,Port,Type | sed '1,7d' | sed 's/Device\ IP/Device_IP/g' | sed 's/[0-9].00\:/\ 00:/g' | awk -F " " '{ print $1"\t"$3"\t"$4"\t"$5"\t"$6 }'
---------------------------------------------------------------------------------------------------
Device_IP MAC IP Port Type
---------------------------------------------------------------------------------------------------
172.29.158.55 00:11:88:16:03:A3 172.29.158.42 12017 dot1dBridge
172.29.158.55 00:11:88:13:EA:C9 172.29.128.218 48 dot1dBridge
172.29.158.55 00:01:F4:44:82:85 172.29.158.74 1 router
172.29.158.55 00:11:88:3F:A3:B0 172.29.128.248 1 dot1dBridge
172.29.158.55 00:E0:63:BB:50:FC 172.29.128.99 1 dot1dBridge
172.29.158.55 00:E0:63:86:AE:2A 172.29.212.15 39 dot1dBridge
172.29.158.55 00:11:88:13:20:41 172.29.128.204 1 dot1dBridge
[hope@gomez lists]$ aliasGrep -s 172.29.158.55 -n ${ROSTR}12 -active
DeviceIP CreationTime Ref Port MacAddress VID VLANName Protocol Address MarkedForDeletion
172.29.158.55 03/04/2007_06:36:43 0 ge.1.7 00:00:E2:64:55:F9 615 ITS-Infrastructure-MonitorsIP 172.27.232.89 no
172.29.158.55 03/04/2007_06:36:08 0 ge.1.12 00:11:25:0C:FC:66 101 101 IP 152.2.202.237 no
172.29.158.55 03/04/2007_12:11:26 0 ge.1.12 00:09:6B:32:81:5E 101 101 IP 152.2.21.35 no
172.29.158.55 03/04/2007_14:20:32 0 ge.1.37 08:00:20:ED:C8:B0 101 101 IP 152.2.145.34 no
Packet Capture
[hope@gomez lists]$ sudo tcpdump -c 200 -nn -w 440.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
200 packets captured
201 packets received by filter
0 packets dropped by kernel
[hope@gomez lists]$ tcpdump -r 440.cap > 440.decode
reading from file 440.cap, link-type EN10MB (Ethernet)
[hope@gomez lists]$ grep -c arp 440.decode
85
[hope@gomez lists]$ grep -c netgenvis 440.decode
88
[hope@gomez lists]$ grep -c icmp6 440.decode
21
[hope@gomez lists]$ grep -v "arp\|icmp6\|netgenvis" 440.decode > 440.trim
[hope@gomez lists]$ less 440.trim
15:34:33.408459 00:0c:31:6a:65:81 > 01:00:0c:cc:cc:cd snap ui/C len=39
15:34:33.688973 IP el-loco-v101.net.unc.edu > OSPF-ALL.MCAST.NET: OSPFv2, Hello (1), length: 56
15:34:33.999325 IP aissysdil.ais.unc.edu.netbios-ns > 152.2.255.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:34:34.137979 (NOV-802.2) 98031cfa.00:30:6e:f9:54:f8.0452 > 98031cfa.ff:ff:ff:ff:ff:ff.0452:ipx-sap-resp[|ipx 64]
15:34:34.489347 802.1d unknown version
15:34:34.749282 IP aissysdil.ais.unc.edu.netbios-ns > 152.2.255.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
Project Report: Fishing at 440 W. Franklin
- These ports had incrementing errors over the course of a few hours:
- 172.29.158.11, port 11: 1 collision
- 172.29.158.20, port 10: 31 collisions
- 172.29.158.20, ports 11 and 13: 5 collisions
- 172.29.158.20, port 12: 1 collision
- 172.29.158.20, port 14: 6 collisions
- 172.29.158.24, port 23: 2 collisions
- 172.29.158.25, ports 16, 18, 19, and 21: 1 CRCAlignError
- 172.29.158.25, ports 8, 20, and 23: 2 CRCAlignErrors
- 172.29.158.25, port 22: 3 CRCAlignErrors
- 172.29.158.25, port 5: 8 UndersizePackets / Fragments
- 172.29.158.25, port 6: 7 UndersizePackets / Fragments
- 172.29.158.25, port 8: 2 CRCAlignErrors, 7 UndersizePackets / Fragments
- 172.29.158.25, port 9: 1 CRCAlignError, 11 UndersizePackets / Fragments
- looks like counters were reset to zero on 172.29.158.44 today, and no new errors
- counters were not reset by a reboot either:
- 172.29.158.43, ports 5 and 31: 1 CRCAlignError
- 172.29.158.43, port 16: 2 CRCAlignErrors
- 172.29.158.45, port 31: 242 collisions
- 172.29.158.49, port 40: 276 collisions
- 172.29.158.51, port 13: 117 collisions
- 172.29.158.55, port 7: 3 CRCAlignErrors
- 172.29.158.55, port 12: 13 CRCAlignErrors, and 9 UndersizePackets / Fragments
- 172.29.158.59, port 43: 318 collisions
- The errors are almost entirely collisions. The good news is that, in a switched environment, the collisions are not going past the switch port logging the errors, but the bad news is that theres a ton!
- This building needs to have something fixed! Switches shouldn't see that many collisions! Since there are not hubs in academic buildings, look for another cause. In this case, we in Networking were troubleshooting a flow control issue with VoIP phones this afternoon, and caused most of these errors.
- The fragments and CRC alignment errors are also troubling, and Im not sure they can all be explained by mucking around with flow control.
- These errors could be traced to an offending IP address. For instance, on switch 172.29.158.55, port 7 (CRC errors) has host 152.2.21.35 (resolves to bogota.oit.unc.edu), and port 12 (fragments and CRC errors) has host 172.27.232.89 in the ITS-Infrastructure-Monitors VLAN. Some of the hosts in specialty VLANs are really screwy, but I don't know what's wrong with bogota.
- Seeing a lot of ARP traffic is expected, but the amount of traffic to netgenvis.net.unc.edu is surprising. (Networking just received a traffic visualizer from Network General. So I guess its asking for lots of traffic data!) Since the conversation appears to be one-sided, netgenvis has aged out of the switches SAT (source address table) and now the traffic is going many places. That traffic flooding should be addressed (by Network General, I would hope; just ping back every 5 minutes).
Gomez
- You can send SNMP across campus if it comes from gomez.
- Once I know what building, I will put list files in /opt/1q/lists for you.
- I need to upgrade the ATG Tools soon.
- No telnet, just ssh.
- Don't forget the 2220.
|
