Finding the bottleneck...

Ok, so I am running some software and it's taking an age to complete what it is doing but the CPU and memory are not fully used (CPU: ~50%, RAM: ~1.05GB of 2GB). It's been going for about 20 minutes.

Is there any way to identify where the bottleneck in my computer is? The software is reading a large data file from an external drive connected by eSATA.

How can i speed this up? I can't see how more RAM would help (even if it's not being fully used) or a faster processor - I must use the data on different machines so striping the data accross 2 drives isn't really an option.

I assume you're using a dual core CPU. With a program that only run single threaded, it will NOT use more than 50% CPU (or 25% on a quad core).

If you can get the program to process different things at the same time (e.g. running two copies). Then it should get to 100%. Otherwise your only choice is to get a faster CPU.

Yes, it's a dual core. But neither core reaches 100% usage, so something appears to be slowing the process down (which isn't the processor).

You might be saturating the sata bus. I'd try using something like (process explorer) to check your system I/O. You should be able to see if it's a flat line (i.e. maxed out).
Another option is that that is as efficient as that particular piece of software gets, if an algorithm is coded badly there's not much you can do to improve your performance.

No, arthurleung is right. It is a single threaded application that is CPU bound. The reason that it doesn't show a single core at 100% and the other idle is because windows is scheduling the 1 busy thread on each core randomly. Either get a faster CPU or a better piece of software that can use multiple cores.

I don't quite follow, fray said that neither core is showing 100% (i.e not talking about the "cpu usage" percentage that taskmanager shows, but rather the individual core utilisation graphs) .
If Windows is switching the single thread between cores why would you not see one of the cores hit 100%?

hmm, viper81's comment seems sensible-any ideas?

Quote:

---

Originally Posted by Viper81

If Windows is switching the single thread between cores why would you not see one of the cores hit 100%?

---

Because that 'single-threaded' process might actually have lots of threads that have to run concurrently, leading to what looks like on average some CPU time being run on one core and some on the other.

However, as soon as he mentioned large data then I/O becomes my primary suspect for a bottleneck. I/O is slow. I/O to a disk is even slower. I/O to an external disk is even slower still. Forget saturating the SATA bandwidth - you won't even get close to that.

The only way to speed it up is to either read that data more efficiently (like sucking it all up into a buffer sequentually rather than having lots of little reads), or transfer less data (for eg. by using a compressed version of the data and uncompressing in memory).

Or of course buy a faster disk.

Let me try an explain better.
Checkout staff are threads
Checkouts are cores
You have one member of staff and give them 2 checkouts. If you now force them to switch between the 2 checkouts (which is what the windows thread scheduler is doing) and measure how busy all the checkouts are, you will find that as a collective whole they are about 50% busy.
The windows scheduler causes the single busy thread to swap core becuase it has more threads that require cpu time than you have cores. Each time the single really busy threads gets a turn it arrives on whichever core windows has deemed free.
This is why a single busy thread will give you a CPU usage of 50% on a dual core. It might be slightly over 50% if other tasks are using the other core, it might be slightly under if there are lots of context switches happening.
If you want to check how memory and disk is being used then you can either look at perfmon which can tell you all sorts of things about what windows and your hardware is doing. Or you can take a look at spotlight for windows, it is made by quest and is now free. It shows the same information that perfmon captures but in a much easier to digest form.

may I ask what the two hard drives are?

What's in the external drive? What brand etc? And what's in your PC?

If you do that sorta stuff that is quite intesive then you might wanna buy a fast dual core, average Quad.

Things that speed up your Pc most is the Harddrive, what type of HDD are you reading from?

is it something that you have to do regularly?
It may be that investing in new disks (or even raid) helps you out.

The external drive is a seagate 1TB Xtreme, it's connected to the computer by eSATA into a pci-express eSATA card. I'm not totally sure what drive is the enclosure, only that it's a 7200rpm drive. I'm not at the computer now so not sure what the internal drive is, it was an off the shelf hp machine bought about 9 months ago (could find out later).

At the moment it's part of my work to deal with pretty large files of 3d data but i need to use more than one machine as different software is installed on different machines so i use the external drive as my working storage - so raid wouldnt really work for me.

if you had some money to spend on it, either a nas box with built in raid (can't recommend one, sorry, don't know enough about them) or building a small server might help.
can get mobo, cpu and ram for under £100, then either use on board raid (which might be good enough) or linux raid, or get a pci-e raid5 card. that, in theory, could max out a gigabit lan connection. more you spend, faster it gets.

From what you said in your original post I still don't think the problem is anything other than CPU bound. Spotlight that I mentioned previously is available from here http://www.quest.com/spotlight-on-windows/
You have to give name / e-mail address to download it, I have, and haven't recieved any spam as a result. It ought to be able to show you exactly what is happening, especially with regard to the disks. It will show you disk queue, number of operations per second, transfer time and % load for all logical and physical disks. If you see % load maxed out, or a sustained high disk queue then your problem is disks. If not then your problem is probably elsewhere. (High disk usage could be a symptom of high swap file use but spotlight will tell you if excessive paging is happening)

Quote:

---

Originally Posted by pipTheGeek

From what you said in your original post I still don't think the problem is anything other than CPU bound.

---

Then I question your knowledge of how computers work. Any speed problem involving moving files is 90% likely to be bound by disk IO, which is by far the slowest thing in your computer. For one thing, fray hasn't complained about his computer running slowly otherwise, just that this file is taking a long time to open. If the computer is still responding quickly while the process is ongoing, then it's not a CPU issue.

How big is the file you're trying to open, fray? And what program are you using to try to open it? Some programs will have a limit to the amount of virtual memory they may access, and it could be that the file is too big for the program's working set (if the program has been badly set up).