Recent Changes - Search:

Classes

FinalExam

Troubleshooting

edit SideBar

PingPacketSize262

Thursday, January 31, 2008 8:39 AM

Jim,

Looks like the switch (172.29.158.18) has tossed its cookies again. Can you give it a kick.

Thanks, Patrick


Patrick -- well -- we were able to ping 152.2.22.211 when we moved it to a different port on the same switch, but NOT when we moved it to a different switch .....

any problems now?

-- jg


OK -- VLAN misconfig on the second switch.

Here's what we did:
- we moved ports 13-15 of the 172.29.158.18 switch to ports 17-19 of the 172.29.158.19 switch in same rack
- all three devices on that switch are now pinging
- if we lose them again, I'm inclined to start looking at physical layer hardware issues with the cables or the servers
- in any case, let's still plan on replacing the 158.18 switch in the morning

VLAN port changes -- on switch 172.29.158.19
- ge.1.17 - VLAN 101 (ITS)
- ge.1.18 - VLAN 11 (ITS-Servers)
- ge.1.19 - VLAN 11 (ITS-Servers)

Patrick -- you should be getting to your stuff now. If you have a repeat problem, let us know.

-- jg


Update on this --- looks like ports 13-24 on the 158.18 switch are bad -- jg


I may have spoken too soon. I can ping uncdc2.unc.edu and lcs.depts.unc.edu, but I don't seem to be able to connect to them in any other way. I'll try to get into them and troubleshoot. More to come...

Thanks, Patrick


I logged in to lcs.depts.unc.edu and I am unable to load any webpages (both on and off campus). However, I can ping stuff both on and off campus. I rebooted and that didn't make any difference. I don't understand why I would be able to get ICMP traffic but nothing else.

Let me know if I can provide more information.

Thanks, Patrick


hmmmm..... neither do we .... that sounds like the symptom Jim Kirkman was reporting.

We moved the device Patrick refers to below to the 172.29.158.19 switch this morning -- what switch is Kirkman having his problems with? -- jg


OK - Cindy has confirmed that 158.19 IS the switch that Kirkman was having similar problems with -- so different problem than .18 switch. Cindy has removed any and all policies from the .19 switch, but we may or may not need to reboot after that.

Patrick -- can you try again now?

-- jg


Unfortunately, they are still down.

Thanks, Patrick


OK -- let's see if the problem is directional or protocol.

Are you able to ping OUT FROM those machines? We know we can ping to them. Try pinging 152.2.21.1 and see what you get.

-- jg


Yes, I can ping out. DNS resolution is also working. But if I try to load a web page, no dice.

Thanks, Patrick


hmmmm..... wonder if there's a packet size issue ....

BTW -- we're planning now to replace BOTH switches (the one you were on and the one you're on now) in the morning

what happens if you try to ping out with 1000 byte packets?

-- jg


Looks like that's it. If I ping with 262B, it works, 263 or greater and no reply.

Thanks, Patrick


Yep -- that's the magic number that I've found as well:

gogan@peach.net.unc.edu$ ping 152.2.22.211 262
PING uncdc2.unc.edu (152.2.22.211): 262 data bytes
270 bytes from 152.2.22.211: seq=0 ttl=127 time=0.690 ms.
270 bytes from 152.2.22.211: seq=1 ttl=127 time=0.555 ms.
270 bytes from 152.2.22.211: seq=2 ttl=127 time=0.591 ms.
^C
---- uncdc2.unc.edu (152.2.22.211) PING Statistics ----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0.555/0.612/0.690 (std = 0.57)
gogan@peach.net.unc.edu$ ping 152.2.22.211 263
PING uncdc2.unc.edu (152.2.22.211): 263 data bytes
no reply from uncdc2.unc.edu
no reply from uncdc2.unc.edu
no reply from uncdc2.unc.edu
^C
---- uncdc2.unc.edu (152.2.22.211) PING Statistics ----
4 packets transmitted, 0 packets received, 100% packet loss

HOWEVER, it does not appear to be true for ALL devices on that switch ... 152.2.22.13 is also on that switch and does NOT have the same problem

-- jg


Wow, it would not have occurred to me to test that. What would cause it not to work with larger packets?

Thanks, Patrick


This verifies that it's NOT your system, but it IS the switch .... here's another system on the same switch - 158.19:

gogan@peach.net.unc.edu$ ping 152.2.0.92 262
PING kms0.depts.unc.edu (152.2.0.92): 262 data bytes
270 bytes from 152.2.0.92: seq=0 ttl=125 time=0.783 ms.
270 bytes from 152.2.0.92: seq=1 ttl=125 time=0.717 ms.
270 bytes from 152.2.0.92: seq=2 ttl=125 time=0.773 ms.
^C
---- kms0.depts.unc.edu (152.2.0.92) PING Statistics ----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0.717/0.757/0.783 (std = 0.29)
gogan@peach.net.unc.edu$ ping 152.2.0.92 263
PING kms0.depts.unc.edu (152.2.0.92): 263 data bytes
no reply from kms0.depts.unc.edu
no reply from kms0.depts.unc.edu
no reply from kms0.depts.unc.edu
^C
---- kms0.depts.unc.edu (152.2.0.92) PING Statistics ----
4 packets transmitted, 0 packets received, 100% packet loss

Note same 262 vs. 263 issue.

Now, the ORIGINAL problem appears to be bad ports 13-24 on the 158.18 switch, so devices were moved to different ports on the 158.19 switch.

So far, I'm finding that the packet size issue is affecting devices on -- you guessed it - ports 13-24 on the 158.19 switch.

I'll let you all know if I see anything otherwise.

-- jg


This is CONFIRMED .... where every port 13-24 on the 172.29.158.18 switch doesn't work unless you reboot the switch and they work for a while and then stop working, on the 172.29.158.19 switch, ports 13-24 can't pass packets larger than 262 bytes. On BOTH switches, ports 1-12 are fine.

Chris -- do we have a failing hardware component on C2s that's affecting ports 13-24? Are we likely to start seeing this across an entire run of C2s?

We will be replacing these two 24-port C2s with a single 48-port C2 in the morning.

-- jg


George, Chris wrote:

Jim,
On both of these units....FAN 2 has failed....this is likely causing some type of overheating condition on the units, thus causing some problems.

Regards,
Christopher George
Enterasys Networks


OK - that points then to whether or not we have a fan defect on a run of C2 switches ....

-- jg


I've gone through ALL of the ITS-Franklin machine room C2s and we have THREE switches with a bad FAN2

172.29.158.18 (which we know about)
172.29.158.19 (which we know about)
172.29.158.50 (which we DIDN'T know about)

We'll probably have to plan on replacing 158.50 sooner rather than later as well. Can someone do an inventory of what devices are on that switch?

-- jg

Edit - History - Print - Recent Changes - Search
Page last modified on February 06, 2008, at 10:32 PM EST