500GB - 1TB superfast scratch storage options

I'm contemplating using my machine more for cinema work which works with individual frame as files. So I end up with working with up to 200,000 files in a directory. In order to improve my workflow, I end up moving them around in bunches of 10,000. Whilst I can use my normal data drive for that (an old 640GB caviar black), it just grinds to a halt with the random access slowing everything down.

So I'm thinking about getting something else, between 500GB and 1TB

Using SSDs is the fastest option, but a little expensive if I end up JBODing 2 or 3 256GB or larger drives (stripe still loses out on trim support?). As it's temporary storage, whilst the reliability isn't a massive issue, I don't want to kill them in a year as they "wear" out. Maybe this isn't so much an issue with trim support, but if I end up pretty much filling them, all the areas will end up being used, so maybe trim doesn't help in the long run?

Maybe a 1TB 10k velociraptor is a good alternative. Or maybe a few smaller faster "ordinary" 7200rpm drives striped?

I have a couple of 6Gbps SATA 3 ports available.

What do you think?
Thanks

The latest Intel drivers do allow for TRIM support across a RAID array, I believe. I would go for two 512Gb Crucial M4's in RAID 0 myself, which should give you plenty of speed at a reasonable price(ish). Of course RAID 0 does halve your MTBF, but I'm not sure about how long the drive might last in terms of write cycles.

Not sure there is really a case for velociraptors any more. If you want performance, stump up for SSD. If you want capacity, get the rotating stuff. If you can't afford the amount of storage you need on SSD, a bunch of hard drives together can give you a hell of an aggregate performance.

I wouldn't worry too much about wear, it's a hugely overstated problem.

As for RAID, well the Crucial M500 with 960GB in one drive should be in stores pretty soon.

How about a hybrid drive?

http://www.ebuyer.com/321969-seagate...ssd-st750lx003

For the workload described, I would think a hybrid drive wouldn't be great.

I have two suggestions if space is necessity you could buy a 3tb from seagate barracuda XT http://www.ebuyer.com/319640-seagate...edium=products i purchased this last week for my self as I was in the same situation as your self. Or if you really need to save some cash buy 1tb samsung spinpoint f3 http://www.amazon.co.uk/Samsung-HD10...3031824&sr=8-1 is a good all rounder with good read and write performance see here http://www.bit-tech.net/hardware/sto...3-1tb-review/8

Quote:

---

Originally Posted by snootyjim

As for RAID, well the Crucial M500 with 960GB in one drive should be in stores pretty soon.

---

OOh I'll look into that.

Quote:

---

Originally Posted by Jonj1611

How about a hybrid drive?

http://www.ebuyer.com/321969-seagate...ssd-st750lx003

---

Don't think they would really help: the random access is on random data that would be far bigger than the cache / SSD area so probably wouldn't notice that much difference to a normal spinning drive.

Couple of old ex-enterpise 15k rpm SAS drives striped is your best bet with a pcie controller card.

I don't fully understand the work pattern, so I may be off base in my suggestion.

I'd suggest for a start that you try FancyCache. This costs you nothing to try, although you'd have to have some spare RAM for the cache. A lot of times software performs accesses of the same things multiple times, so it could help. If it does, also consider trying an SSD cache.

The work is with TIFF. DPX or JPEG2000 compressed files. Up to around 500,000 files in a couple of directories. Files are anything between 200kB and 6MB in size. I'm often moving them around in chunks of 10,000 to then process them.
I can't see how having a small cache will help as I'm effectively randomly accessing several 100GB at a time, or doing sequential read / write on the same disk which cripples its performance.
For now, I've tried to split my reads and writes across different physical drives to maximise contiguous file access. That's not an issue on SSDs, but my existing 128GB M4s just aren't big enough.

Now I haven't tried this, but I am wondering if another PC running FreeNas would work here?

This would allow you to use ZFS across a large number of spindles. Lots of spindles means lots of IO per second, and gigabit ethernet is probably fast enough and certainly faster than a single drive thrashing around on a faster interface. You can add an SSD to a ZFS setup to smooth writes

Quote:

---

Originally Posted by DanceswithUnix

Now I haven't tried this, but I am wondering if another PC running FreeNas would work here?

This would allow you to use ZFS across a large number of spindles. Lots of spindles means lots of IO per second, and gigabit ethernet is probably fast enough and certainly faster than a single drive thrashing around on a faster interface. You can add an SSD to a ZFS setup to smooth writes

---

Hmm I hadn't thought about going this far, but did think of an intermediate solution. My current NAS box is an HP Microserver with 4x 2TB drives driven off a proper p410i RAID controller with 512MB cache. So that is actually relatively fast and the half GB cache helps. It's WHS v2 running on it and I was contemplating making some of the storage iSCSI to make it appear relatively fast on my workstation, bypassing the inefficiencies of CIFS-based sharing...

Quote:

---

Originally Posted by tfboy

I can't see how having a small cache will help as I'm effectively randomly accessing several 100GB at a time, or doing sequential read / write on the same disk which cripples its performance.

---

As I said, it highly depends on your how optimal your software is. Even for file copying there are quite a few programs claiming to be better than the Windows copy (which doesn't mean they really are). I couldn't find good real world benchmarks of FancyCache, but I found one user reporting a marginal (~10%) gain: 23 seconds with FancyCache vs. 26 seconds without for copying 3GB from an SSD onto itself. It's possible that with an HDD you'll gain more from the cache.

It's free to try, so I don't see why you'd want to find arguments against it when it's pretty easy to test it for yourself. If it doesn't gain you anything, you've lost a little time trying it. If it does gain you something, you've gained a useful tool (that's free for now).

If you do try it (and I hope you will), please post your results.

Not having a digg, it might work. I'll give it a try next time I'm doing some work and will post results (positive or otherwise) :)