Solved! Server 2008 R2 Slow Network Writes

[jim]

At present, I've got a basic Server 2008 R2 SP1 installation on my HP N36L, no VMs or anything. Only role added is file services. (EDIT: tried removing the role, no change)

OS is installed on the standard 250GB drive, data drive is spanned across two 2TB drives.

Transfer speeds look like so:

Server OS to Server Data = 105 MB/s
Server OS to Windows 7 = 70 MB/s
Server Data to Windows 7 = 70 MB/s
Windows 7 to Server OS = 1 MB/s
Windows 7 to Server Data = 1 MB/s

What on earth could be causing that? The hard disks are clearly fine, I can write to them at 105 MB/s. And the network is clearly fine, since I can transfer at 70 MB/s provided I'm only reading data. Haven't turned up anything useful yet but I'm still searching

[jim]

Seems like xcopy /j might be the answer... not sure. Kinda difficult to test xcopy easily though.

[jim]

Nope, that doesn't seem to work either... and small files are no quicker, so that's definitely out.

[jim]

Tried iperf, reports 800 Mb/s transfer speeds

[jim]

Hmm, still not having any luck, and google is turning up absolutely nothing of any use. I think I'm going to have to ditch this and go back to unix, really cannot be bothered with this nonsense right now

[Jay]

any antivirus on these PCs?

[spoon_]

Did you turn write cache in BIOS Jim? Just curious, it might not have anything to do with the problems you're experiencing..

[Paul Adams]

I would try disabling SMB2 to see if it makes any difference, and also look at any offloading features of the NIC on the machines involved (TCP, UDP, IPv4, checksum... different manufacturers have different names & options, you find them in the properties of the NIC.

[jim]

any antivirus on these PCs?

Not server to server, no - I didn't list that, but it didn't work either.

Did you turn write cache in BIOS Jim? Just curious, it might not have anything to do with the problems you're experiencing..

Yes I did, saw the warning in the thread before I even plugged it in thankfully

[jim]

I would try disabling SMB2 to see if it makes any difference, and also look at any offloading features of the NIC on the machines involved (TCP, UDP, IPv4, checksum... different manufacturers have different names & options, you find them in the properties of the NIC.

Yeah, I saw a few warnings about SMB2. Isn't that only invoked between Server 2008 and Windows 7? I didn't mention earlier, but I had exactly the same problems with Server 2008 --> Server 2003.

I did begin to look at the NIC options, didn't make much progress though.

I will probably pick this up again in a couple of months and have another go.

[Paul Adams]

Yeah, I saw a few warnings about SMB2. Isn't that only invoked between Server 2008 and Windows 7?

NT 6.0+ talking to NT 6.0+, so anything from Vista onwards.

I didn't mention earlier, but I had exactly the same problems with Server 2008 --> Server 2003.

That would definitely only negotiate SMB then.

I did begin to look at the NIC options, didn't make much progress though.

When you do, look at Receive Side Scaling (RSS) and TCP Chimney too (controlled using netsh.exe) to see if they have any impact - not all NICs cope well with the more advanced networking features but rather unfortunately try.

[jim]

Amusingly, I am seeing exactly the same thing with Openfiler.

I'm now wondering whether it's to do with the advanced format.

[pardal51]

Wireshark against the NIC.

Do NIC + switch support jumbo frames? Any difference?

[jim]

Okay, that would suggest that there's something up. I've never used Wireshark before, but I'm guessing that thousands of red lines isn't a good thing.

Loads of them seem to come up with header checksums of 0x0000. The vast majority of the other red lines seem to be the result of "duplicate ACK".

I've got one here that says "TCP Dup ACK 9642#153" - does that mean that it's the 153rd time it's failed?

I've got huge chains of:

PC to Server: TCP segment of a reassembled PDU [Header checksum: 0x0000]
Server to PC: TCP Dup Ack 9642#xxx [This is a TCP duplicate ack]

And repeated up to over 100 times until presumably it works and moves onto the next one.

[Jay]

I think your packets are arriving out of order, or the server is not sending a packet to let the sender know the data has been recived, the sender then re sends the data. I would try another NIC or different drivers.

Do you have any VLANs?

[jim]

Interestingly, I have loads of very similar data when transferring from my desktop to my working server at 90MB/s.

Loads of packets flagged up with header checksum of 0x0000.

It's not producing the same kind of chains of TCP Dup Ack though which is interesting. It seems to produce one each time, rather than hundreds.

What this all means I have no idea. Is the general suggestion that my network is borked?