Internal Network Monitoring tool?

[TiG]

Got what appears to be a dodgy connection for one of my dev servers.

Its run through 3 switches before it goes to a leased line to a third party where the info i need comes from. However approx every 12 minutes the connection drops out?.

Any ideas on how to track this down?. I've tried perfmon, changing cables etc etc, but its two servers locally and i have a feeling its something on a switch level and i'm not priveleged enough to have access to two of the switches in the route :/

TiG

[Jonny M]

This is a long shot from a newb to this sort of setup, but when you tracert does it report the IPs of the switches like it does when I tracert a site using my router (adding the router as a hop and giving me the IP). If you you could tracert when the connection drops and see where it stops? Very crude solution, no doubt Chris will come up with something better that tells you where your keys are as well. And makes toast.

[TiG]

heh fraid icmp is turned off, as its a requirement for the security, I've got a feeling what it is - i just want evidence before i go stomping in my size 9's

I know the IP addresses of the routers, tho droppign connection every 12 minutes then recovering is the sign of someone playing about with stuff they shouldn't...

TiG

[scottyman]

Odd - had a similar problem (but every 24 mins) turned out to be faulty pci bus on my server mobo

I'd use ethereal and spend some time writing a filter (run it on a pomiscuous card) monitoring point to point between the last router in the chain, and your dev server.

BTW - why not use ICMP internally - just block IGMP instead - much safer, reduces risk of network pollution!

[TiG]

Because i don't have rights on the live server switches, there is a link between development and live for a few things that we can't duplicate either due to cost or lack of equipment, Last thing we want is ICMP on a live system.... especially when we do content for lots of 3rd parties all with links into us...

TiG

[scottyman]

buggerific.

seriously - sounds like a hardware problem.

You should at least have SNMP status monitoring of those switches - in fact all the related kit.

[TiG]

SNMP isn't part of ICMP tho. And the hardware is all fine, they are things like Cisco 3548 switches, i.e not cheap 3K+ switches.

You can however put timeouts on the connections and i've come to the belief that that Cat 6500 has had its configs changed to do this, however i can't go jumping in till i've got proof especially if i turn out to be wrong, yet i've got no access to it. If i did i could prove it from the configs.

How to prove that this switch is dropping the connection?.

TiG

[scottyman]

it's going to have to be a traffic analysis...
i.e.
doing a traffic capture point to point on the network to analyse that fault - you'll need two machines, and just run it side by side.
If you can't use ICMP traffic to monitor whether it's still responsive

Unfortunately, you may be restricted to SMNP monitoring, which won't highlight a fault - it will simply tell you that the switch is functioning within acceptable parameters. if there is a faulty setting smnp won't necessarily tell you that there is a fault, as it's a passive protocol.

doesn't help I know - but it may give you another idea or two!

[TiG]

Thanks for all the help and being a wall to bounce things off.

Just managed to nail the problem. found a new way of using netstat command (using some of the -s or -a tags). You can force win2k server to give you a breakdown of whats happen on all comms channels with errors, force closures.

Taking regular captures from this i could pick up the forced closure events and make someone deal with the problem on the config on another switch

Thanks again
TiG

[scottyman]

dude - that's great... glad it's worked out.

One of those commands I use in *nix all the time, but never on 2000 oddly enough!

[Nimrod]

SNMP is not passive, you can use SNMP to update values as long as you have the community string for the "write" community

Glad you got it sorted chap.

I was just going to post to say that there is currently a known problem with a certain Cisco implementation, I cannot remember which. Where if the router is sent x number of lookup requests to servers that do not exist the router eventually locks up and requires rebooting.

But it looks like you've got enough information to get teh problem off of your desk which is the main thing