Slower concurrent disk access with NCQ ?

[Rosaline]

*i.e.
Opening of notepad when copying large file without NCQ may take 3 seconds, file copy takes 2 minutes. With NCQ, the file copy will take less than 2 minutes and notepad will take less than 3 seconds to load.

In terms of discrete time needed to load from disk, yes.
In terms of perceived time, these times may be interleaved in a manner which results in the more distant operation appearing to take longer, as it's commands are delayed, but in total the timings are shorter. That was what I was referring to.

Edit: I just re-read my initial post, and it turns out I had already stated that. So why exactly people want to believe otherwise is beyond me Yes, I am willing to chart this all out if need be, but it is exam season, and I'd rather that right now, people read into how this works themselves. If you just plot out the access points over time for a random set of requests, unsorted and sorted, this is an immediately self-evident result. Individual response time may increase, to allow system total response time to increase. Text-book stuff, quite frankly, and not for the module I have an exam on tomorrow.

Edit 2: Yes, the above is slightly over the top. Regulars know that I normally love explaining this sort of thing, and I'm just really stressed and really sad that I cannot put the effort into explaining how NCQ works right now

It doesn't matter if the loading is faster, if there is a delay before it begins. Where the benefit from NCQ ordering reduces the time needed to load the program by a factor greater than the delay, then NCQ is beneficial.

A lot of this will be dependant upon the SATA controller on the motherboard (as these can also do various 'optimisations' themselves), the operating system (which traditionally also does a small amount command queuing) and upon the vendor specific implementation of NCQ and the command buffer size.

[TiG]

NCQ was designed for servers on scsi systems years ago in the first place, be it web servers, database servers, we're talking a lot more disk access than those tests above, concurrent heavy reads, writes, scsi still has far more io capability than sata.

What sometimes I can't understand is why people worry about whether something loads in 2.8 seconds or 3 seconds.

TiG

[badass]

Edit: I just re-read my initial post, and it turns out I had already stated that. So why exactly people want to believe otherwise is beyond me Yes, I am willing to chart this all out if need be, but it is exam season, and I'd rather that right now, people read into how this works themselves. If you just plot out the access points over time for a random set of requests, unsorted and sorted, this is an immediately self-evident result. Individual response time may increase, to allow system total response time to increase. Text-book stuff, quite frankly, and not for the module I have an exam on tomorrow.

Individual response times will only increase when you are not talking about multiple access.
When you are talking about multiple access, with NCQ, percieved individual response times for items will not increase. They will decerease. Even if one item has only a few MB to load.
This reordering doesn't take a few seconds. It takes milliseconds.
So, yes, Wordpad may get its first bytes a few ms earlier without NCQ, it may not, however when we are taling about the app taking over a second to load, it will still finish loading faster with NCQ because it is getting data off the disk faster than it otherwise would without NCQ.

If you notice anything slower when you are using NCQ, something is wrong. end of. It does not just make the several jobs get done quicker all together, every individual part of the multiple file copy, open etc will also be quicker. Any theory that says otherwise is based on a flawed understanding.

[Rosaline]

Badass, I do not have the time to debate this right now. I would love to, and as my posting history shows, I normally am quite happy to drag out the actual theory. However, I have exams going on at the moment, and so cannot spend the time correcting you.

Please, go away and read into how it works, and the fact that some commands will hence be processed later than when they originally arrived at the hard drive. This is not even hard to understand.

I will concur however that a delay in the magnitude of seconds is unlikely to be caused by NCQ, given the speed of disk rotation and low seek times. But you are quite frankly wrong that NCQ will always magically speed up individual response times - it depends on the load, and NCQ focuses on speeding up the system as a whole which under heavy load, speeds up most individual requests, but may favour only a few common ones under small loads.

Please, read into this, and stop declaring others to have flawed understanding. If you like, I'll return to this in a few weeks to show how individual processes may find that their requests take longer. It is a simple text book issue that I am frankly shocked to have to debate at all.

Edit: As already stated, though, a lot of this is highly dependant upon the performance of the system, hence why sweeping generalisations and accusations are ultimately futile.

[aidanjt]

Roseline, as far as your understanding goes of the hardware technology, you're absolutely correct, however, you can't ignore kernel disk I/O queuing factors, and how it works with NCQ. When I was speaking of NCQ, and how it optimises I/O in the way I described, I took OS optimisations into account. NCQ is also less aggressive in minimalist seeks compared to TCQ, which is why it won out, despite TCQ being technically superior, it's suboptimal for desktop kernels.

[Rosaline]

Aidanjt, are you referring to the following, as I can't find another post by you talking about NCQ:

NCQ is only useful for stacking SATA commands in an order which is optimal for i/o across several sections of the disk surface in as little rotations as possible, so, great for 10 different processes doing i/o in different files... pointless for copying 2gb file from a to b. Your benchmark wouldn't make use of it.

I think we're speaking on the same wavelength in general, and I did mention that the entire issue is complicated by things such as OS optimisations (which yes, it too tries to order them nicely in general).

[aidanjt]

I think we may be

*edit* Also, good luck with the exams.

[xerces8]

PLEASE STOP the "I know NCQ better than thou" flame!!!!!!!!!!!!

Thank you.

It is completely irrelevant to this thread, because all we know, NCQ might not be used in the problem case at all. All we know is that the HW is in AHCI mode and an AHCI driver is controlling it.

Also half second up or down does not matter. The problem is 3 seconds versus 100 seconds. A factor of THIRTY.

[Rosaline]

As already stated, that sort of time difference will not be down to NCQ at all, but more likely a driver or hardware fault.

If things run fine with AHCI mode turned off, then there's an obvious fix

Have you tried checking in the windows error log to see if there's any reported issues with the hard drive? That long a delay I would suspect to mean that it tries several times then falls back to a slower interfacing method.

[badass]

Badass, I do not have the time to debate this right now. I would love to, and as my posting history shows, I normally am quite happy to drag out the actual theory.

And that is the point.
Theoretically NCQ can make individual processes slower, however because of the hundreds of transactions per second, in practice it never does.
I understand how NCQ works just fine thank you and I understand what you are saying and the theory is indeed correct, however arguing this based on theory is like someone coming up lots of completely sound theory as to why the sky is green.
I understand that you have exams to worry about so dont have the time now, but the only way to prove me wrong on this is to get some actual graphs where people have tested real world scenarios. You'll see that in practice the only thing that is measurably slower is single application disk access.

EDIT: Just thought of a faaaar better analogy.

Its like arguing with someone that Windows* copies files across a network when the fast ethernet cards are both running full duplex is in fact slower thean when you set both NIC's to half duplex.
You find that in practice, with the NIC's set to half duplex, the copying of single files is 4 times faster. The theory suggests that full duplex should be a miniscule bit faster.

*2000 and prior. Not sure of you get this effect with XP or Vista

[|FAT|Punisher]

This topic is somewhat older now, and I don't know if somebody still watches it, but I have the same problem as xerces8.

I recently bought a Western Digital WD6400AAKS, which is a great drive, big, really really fast, cheap. I'm using a Foxconn Mars which has an Intel ICH9R. I have to run the ICH9R in RAID mode, because I have a RAID 0 consisting of two WD2500YDs. The RAID 0 works fine, like it used to do on an earlier mainboard with nForce4. The WD6400AAKS however shows the same behaviour xerces8 has seen: if there is only one application which reads (or writes, doesn't matter) huge amounts of data (sequentially), other applications accessing that drive hang completely (even the GUI) for a long time, meaning tens of seconds and more. Even the Explorer can hang when trying to access the drive.

I have never seen a behaviour like that before. Of course everything gets slower (a lot slower), if several apps access the same drive, but every app still works and reacts. At least that's what I've experienced until now, during the last decade and more.

My first thought was NCQ, too, and unfortunately I can't find a way to disable it. The jumper on the drive reduces the transfer mode to SATA generation 1, but NCQ seems to stay active (the Intel Matrix Storage Manager still says that it is "supported", it does not say if it is actually used). There's no option in the BIOS (because the controller has to stay in RAID mode), or in the RAID setup either. No option in Windows, neither in the Intel Matrix Storage Manager.

So I'm not sure if it is NCQ, but I am completely sure that this behaviour is not "normal". The Windows event log doesn't show anything suspicious, by the way.


Any ideas?

[xerces8]

Hi!

Can you give more details about your setup ?
What other disks do you have connected ?
Does the problem happen only with a WD disk ?


I tried a different HD on my system (WD MyBook Studio 500GB) and it has the same problem. I used RAID mode this time (used AHCI before).

Regards,
David

[|FAT|Punisher]

I've filled in my system now, and if you visit "my homepage", you'll see a more detailed description at sysProfile.de.

I have used a lot of WD discs (WD800JB, WD1600JB, WD2500YD, WD5000YS, WD6400AAKS), but this is the first hard disc ever that exposes this behaviour.

As far as I know, RAID mode implicates AHCI.

[|FAT|Punisher]

I think I have found the reason for our problems.

In certain situations, these extreme delays became unacceptable, so I put more time into this problem.
I gave the JMicron JMB361 on my board a chance, et voilà: no delays, no problems, everything is as it should be!

Unfortunately, my Foxconn Mars has only an eSATA port, which is not an option for my internal 6400AAKS. And even worse: I was thinking about buying more 6400AAKS, for a decent RAID 5, because they're cheap, big and incredibly fast. That does not seem to be a good idea anymore...

Unfortunately, I wasn't able to check if NCQ is activated on the JMicron controller.
[Edit] HD Tune states that it is active. Or supported? I don't know, but NCQ is checked.

[xerces8]

So you moved the disk from the ICH port to the JMB ?

I also have a JMB363. It gives even more trouble (with another HD, a WD MyBook Studio 500GB). I guess each controller has problems with different discs ...

[|FAT|Punisher]

Yeah, I moved it from the ICH9R port to the JMB361. I used an eSATA-to-SATA cable because the JMB361 on my board has only one eSATA port.

- And indeed, I have other problems now.
From time to time, the JMB hangs completely, all applications accessing the drive hang completely, for several seconds. The Windows Event Log shows: "The device \Device\Scsi\JRAID1 did not respond within the timeout period." (translated from German, might not be an exact translation).

The worst thing is: I have no clue if the reason for this is the JMicron controller (thus it would be a second, different problem), or if it is still the hard drive.

What are your experiences with the JMicron?


[Edit] I just realized that I had already encountered the "JMicron problem" with my WD5000YS, too!
But what do I do now? I have no money to spend on a "real" RAID controller for several 100 bucks...