Results 1 to 15 of 15

Thread: 500GB - 1TB superfast scratch storage options

  1. #1
    Senior Member
    Join Date
    Jul 2003
    Location
    Reading, Berkshire
    Posts
    1,253
    Thanks
    64
    Thanked
    53 times in 34 posts
    • tfboy's system
      • Motherboard:
      • MSI X470 Gaming Plus
      • CPU:
      • AMD Ryzen 7 2700
      • Memory:
      • 2x8GB Corsair Vengeance LPX)
      • Storage:
      • Force MP600 1TB PCIe SSD
      • Graphics card(s):
      • 560 Ti
      • PSU:
      • Corsair RM 650W
      • Case:
      • CM Silencio 550
      • Operating System:
      • W10 Pro
      • Monitor(s):
      • HP LP2475w + Dell 2001FP
      • Internet:
      • VM 350Mb

    500GB - 1TB superfast scratch storage options

    I'm contemplating using my machine more for cinema work which works with individual frame as files. So I end up with working with up to 200,000 files in a directory. In order to improve my workflow, I end up moving them around in bunches of 10,000. Whilst I can use my normal data drive for that (an old 640GB caviar black), it just grinds to a halt with the random access slowing everything down.

    So I'm thinking about getting something else, between 500GB and 1TB

    Using SSDs is the fastest option, but a little expensive if I end up JBODing 2 or 3 256GB or larger drives (stripe still loses out on trim support?). As it's temporary storage, whilst the reliability isn't a massive issue, I don't want to kill them in a year as they "wear" out. Maybe this isn't so much an issue with trim support, but if I end up pretty much filling them, all the areas will end up being used, so maybe trim doesn't help in the long run?

    Maybe a 1TB 10k velociraptor is a good alternative. Or maybe a few smaller faster "ordinary" 7200rpm drives striped?

    I have a couple of 6Gbps SATA 3 ports available.

    What do you think?
    Thanks

  2. #2
    Evil Monkey! MrJim's Avatar
    Join Date
    Aug 2012
    Location
    London
    Posts
    2,313
    Thanks
    300
    Thanked
    472 times in 362 posts
    • MrJim's system
      • Motherboard:
      • MSI Tomahawk X570
      • CPU:
      • AMD Ryzen 5900X
      • Memory:
      • 32gb Kingston 3600 DDR4
      • Storage:
      • Aorus 1Tb NVME SSD, Samsung 1Tb 970 Evo SSD, Crucial 2tb MX500 SSD, Seagate Ironwolf 4Tb SSD
      • Graphics card(s):
      • EVGA 3080Ti
      • PSU:
      • Seasonic Prime Ultra Platinum 1300W
      • Case:
      • Fractal Meshify 2
      • Operating System:
      • Windows 11 Pro
      • Monitor(s):
      • Viewsonic 27" XG2703-GS
      • Internet:
      • BT 900 mb/s FTTP

    Re: 500GB - 1TB superfast scratch storage options

    The latest Intel drivers do allow for TRIM support across a RAID array, I believe. I would go for two 512Gb Crucial M4's in RAID 0 myself, which should give you plenty of speed at a reasonable price(ish). Of course RAID 0 does halve your MTBF, but I'm not sure about how long the drive might last in terms of write cycles.

  3. #3
    root Member DanceswithUnix's Avatar
    Join Date
    Jan 2006
    Location
    In the middle of a core dump
    Posts
    12,978
    Thanks
    778
    Thanked
    1,586 times in 1,341 posts
    • DanceswithUnix's system
      • Motherboard:
      • Asus X470-PRO
      • CPU:
      • 5900X
      • Memory:
      • 32GB 3200MHz ECC
      • Storage:
      • 2TB Linux, 2TB Games (Win 10)
      • Graphics card(s):
      • Asus Strix RX Vega 56
      • PSU:
      • 650W Corsair TX
      • Case:
      • Antec 300
      • Operating System:
      • Fedora 39 + Win 10 Pro 64 (yuk)
      • Monitor(s):
      • Benq XL2730Z 1440p + Iiyama 27" 1440p
      • Internet:
      • Zen 900Mb/900Mb (CityFibre FttP)

    Re: 500GB - 1TB superfast scratch storage options

    Not sure there is really a case for velociraptors any more. If you want performance, stump up for SSD. If you want capacity, get the rotating stuff. If you can't afford the amount of storage you need on SSD, a bunch of hard drives together can give you a hell of an aggregate performance.

  4. #4
    jim
    jim is offline
    HEXUS.clueless jim's Avatar
    Join Date
    Sep 2008
    Location
    Location: Location:
    Posts
    11,457
    Thanks
    613
    Thanked
    1,645 times in 1,307 posts
    • jim's system
      • Motherboard:
      • Asus Maximus IV Gene-Z
      • CPU:
      • i5 2500K @ 4.5GHz
      • Memory:
      • 8GB Corsair Vengeance LP
      • Storage:
      • 1TB Sandisk SSD
      • Graphics card(s):
      • ASUS GTX 970
      • PSU:
      • Corsair AX650
      • Case:
      • Silverstone Fortress FT03
      • Operating System:
      • 8.1 Pro
      • Monitor(s):
      • Dell S2716DG
      • Internet:
      • 10 Mbps ADSL

    Re: 500GB - 1TB superfast scratch storage options

    I wouldn't worry too much about wear, it's a hugely overstated problem.

    As for RAID, well the Crucial M500 with 960GB in one drive should be in stores pretty soon.

  5. #5
    Super Moderator Jonj1611's Avatar
    Join Date
    Jun 2008
    Posts
    5,718
    Thanks
    1,753
    Thanked
    994 times in 761 posts

    Re: 500GB - 1TB superfast scratch storage options

    Jon

  6. #6
    Anthropomorphic Personification shaithis's Avatar
    Join Date
    Apr 2004
    Location
    The Last Aerie
    Posts
    10,857
    Thanks
    645
    Thanked
    872 times in 736 posts
    • shaithis's system
      • Motherboard:
      • Asus P8Z77 WS
      • CPU:
      • i7 3770k @ 4.5GHz
      • Memory:
      • 32GB HyperX 1866
      • Storage:
      • Lots!
      • Graphics card(s):
      • Sapphire Fury X
      • PSU:
      • Corsair HX850
      • Case:
      • Corsair 600T (White)
      • Operating System:
      • Windows 10 x64
      • Monitor(s):
      • 2 x Dell 3007
      • Internet:
      • Zen 80Mb Fibre
    For the workload described, I would think a hybrid drive wouldn't be great.
    Main PC: Asus Rampage IV Extreme / 3960X@4.5GHz / Antec H1200 Pro / 32GB DDR3-1866 Quad Channel / Sapphire Fury X / Areca 1680 / 850W EVGA SuperNOVA Gold 2 / Corsair 600T / 2x Dell 3007 / 4 x 250GB SSD + 2 x 80GB SSD / 4 x 1TB HDD (RAID 10) / Windows 10 Pro, Yosemite & Ubuntu
    HTPC: AsRock Z77 Pro 4 / 3770K@4.2GHz / 24GB / GTX 1080 / SST-LC20 / Antec TP-550 / Hisense 65k5510 4K TV / HTC Vive / 2 x 240GB SSD + 12TB HDD Space / Race Seat / Logitech G29 / Win 10 Pro
    HTPC2: Asus AM1I-A / 5150 / 4GB / Corsair Force 3 240GB / Silverstone SST-ML05B + ST30SF / Samsung UE60H6200 TV / Windows 10 Pro
    Spare/Loaner: Gigabyte EX58-UD5 / i950 / 12GB / HD7870 / Corsair 300R / Silverpower 700W modular
    NAS 1: HP N40L / 12GB ECC RAM / 2 x 3TB Arrays || NAS 2: Dell PowerEdge T110 II / 24GB ECC RAM / 2 x 3TB Hybrid arrays || Network:Buffalo WZR-1166DHP w/DD-WRT + HP ProCurve 1800-24G
    Laptop: Dell Precision 5510 Printer: HP CP1515n || Phone: Huawei P30 || Other: Samsung Galaxy Tab 4 Pro 10.1 CM14 / Playstation 4 + G29 + 2TB Hybrid drive

  7. #7
    Member
    Join Date
    Nov 2011
    Posts
    161
    Thanks
    4
    Thanked
    4 times in 2 posts
    • NitrousX's system
      • Motherboard:
      • MSi X99 Xpower Gaming Titanium
      • CPU:
      • i7 5930k 4.3Ghz w/EKWB Supremacy Evo
      • Memory:
      • Corsair Vengeance LED 2666mhz 32GB DDR4 4x 8GB sticks
      • Storage:
      • Samsung 960 Evo 1TB, HGST 5TB Deskstar, Seagate Barracuda XT 3TB
      • Graphics card(s):
      • EVGA GTX 1080 Ti FE w/FE EKWB Waterblock
      • PSU:
      • EVGA 750W GS
      • Case:
      • Corsair 600C
      • Operating System:
      • Windows 10 Pro 64Bit
      • Monitor(s):
      • Dell S2716DG
      • Internet:
      • Utility Warehouse 10mpbs

    Re: 500GB - 1TB superfast scratch storage options

    I have two suggestions if space is necessity you could buy a 3tb from seagate barracuda XT http://www.ebuyer.com/319640-seagate...edium=products i purchased this last week for my self as I was in the same situation as your self. Or if you really need to save some cash buy 1tb samsung spinpoint f3 http://www.amazon.co.uk/Samsung-HD10...3031824&sr=8-1 is a good all rounder with good read and write performance see here http://www.bit-tech.net/hardware/sto...3-1tb-review/8

  8. #8
    Senior Member
    Join Date
    Jul 2003
    Location
    Reading, Berkshire
    Posts
    1,253
    Thanks
    64
    Thanked
    53 times in 34 posts
    • tfboy's system
      • Motherboard:
      • MSI X470 Gaming Plus
      • CPU:
      • AMD Ryzen 7 2700
      • Memory:
      • 2x8GB Corsair Vengeance LPX)
      • Storage:
      • Force MP600 1TB PCIe SSD
      • Graphics card(s):
      • 560 Ti
      • PSU:
      • Corsair RM 650W
      • Case:
      • CM Silencio 550
      • Operating System:
      • W10 Pro
      • Monitor(s):
      • HP LP2475w + Dell 2001FP
      • Internet:
      • VM 350Mb

    Re: 500GB - 1TB superfast scratch storage options

    Quote Originally Posted by snootyjim View Post
    As for RAID, well the Crucial M500 with 960GB in one drive should be in stores pretty soon.
    OOh I'll look into that.
    Quote Originally Posted by Jonj1611 View Post
    Don't think they would really help: the random access is on random data that would be far bigger than the cache / SSD area so probably wouldn't notice that much difference to a normal spinning drive.

  9. #9
    Account closed at user request
    Join Date
    Aug 2003
    Location
    Elephant watch camp
    Posts
    2,150
    Thanks
    56
    Thanked
    115 times in 103 posts
    • wasabi's system
      • Motherboard:
      • MSI B85M-G43
      • CPU:
      • i3-4130
      • Memory:
      • 8 gig DDR3 Crucial Rendition 1333 - cheap!
      • Storage:
      • 128 gig Agility 3, 240GB Corsair Force 3
      • Graphics card(s):
      • Zotac GTX 750Ti
      • PSU:
      • Silver Power SP-S460FL
      • Case:
      • Lian Li T60 testbanch
      • Operating System:
      • Win7 64bit
      • Monitor(s):
      • First F301GD Live
      • Internet:
      • Virgin cable 100 meg

    Re: 500GB - 1TB superfast scratch storage options

    Couple of old ex-enterpise 15k rpm SAS drives striped is your best bet with a pcie controller card.

  10. #10
    Senior Member
    Join Date
    Apr 2010
    Posts
    492
    Thanks
    2
    Thanked
    32 times in 23 posts

    Re: 500GB - 1TB superfast scratch storage options

    I don't fully understand the work pattern, so I may be off base in my suggestion.

    I'd suggest for a start that you try FancyCache. This costs you nothing to try, although you'd have to have some spare RAM for the cache. A lot of times software performs accesses of the same things multiple times, so it could help. If it does, also consider trying an SSD cache.

  11. #11
    Senior Member
    Join Date
    Jul 2003
    Location
    Reading, Berkshire
    Posts
    1,253
    Thanks
    64
    Thanked
    53 times in 34 posts
    • tfboy's system
      • Motherboard:
      • MSI X470 Gaming Plus
      • CPU:
      • AMD Ryzen 7 2700
      • Memory:
      • 2x8GB Corsair Vengeance LPX)
      • Storage:
      • Force MP600 1TB PCIe SSD
      • Graphics card(s):
      • 560 Ti
      • PSU:
      • Corsair RM 650W
      • Case:
      • CM Silencio 550
      • Operating System:
      • W10 Pro
      • Monitor(s):
      • HP LP2475w + Dell 2001FP
      • Internet:
      • VM 350Mb

    Re: 500GB - 1TB superfast scratch storage options

    The work is with TIFF. DPX or JPEG2000 compressed files. Up to around 500,000 files in a couple of directories. Files are anything between 200kB and 6MB in size. I'm often moving them around in chunks of 10,000 to then process them.
    I can't see how having a small cache will help as I'm effectively randomly accessing several 100GB at a time, or doing sequential read / write on the same disk which cripples its performance.
    For now, I've tried to split my reads and writes across different physical drives to maximise contiguous file access. That's not an issue on SSDs, but my existing 128GB M4s just aren't big enough.

  12. #12
    root Member DanceswithUnix's Avatar
    Join Date
    Jan 2006
    Location
    In the middle of a core dump
    Posts
    12,978
    Thanks
    778
    Thanked
    1,586 times in 1,341 posts
    • DanceswithUnix's system
      • Motherboard:
      • Asus X470-PRO
      • CPU:
      • 5900X
      • Memory:
      • 32GB 3200MHz ECC
      • Storage:
      • 2TB Linux, 2TB Games (Win 10)
      • Graphics card(s):
      • Asus Strix RX Vega 56
      • PSU:
      • 650W Corsair TX
      • Case:
      • Antec 300
      • Operating System:
      • Fedora 39 + Win 10 Pro 64 (yuk)
      • Monitor(s):
      • Benq XL2730Z 1440p + Iiyama 27" 1440p
      • Internet:
      • Zen 900Mb/900Mb (CityFibre FttP)

    Re: 500GB - 1TB superfast scratch storage options

    Now I haven't tried this, but I am wondering if another PC running FreeNas would work here?

    This would allow you to use ZFS across a large number of spindles. Lots of spindles means lots of IO per second, and gigabit ethernet is probably fast enough and certainly faster than a single drive thrashing around on a faster interface. You can add an SSD to a ZFS setup to smooth writes

  13. #13
    Senior Member
    Join Date
    Jul 2003
    Location
    Reading, Berkshire
    Posts
    1,253
    Thanks
    64
    Thanked
    53 times in 34 posts
    • tfboy's system
      • Motherboard:
      • MSI X470 Gaming Plus
      • CPU:
      • AMD Ryzen 7 2700
      • Memory:
      • 2x8GB Corsair Vengeance LPX)
      • Storage:
      • Force MP600 1TB PCIe SSD
      • Graphics card(s):
      • 560 Ti
      • PSU:
      • Corsair RM 650W
      • Case:
      • CM Silencio 550
      • Operating System:
      • W10 Pro
      • Monitor(s):
      • HP LP2475w + Dell 2001FP
      • Internet:
      • VM 350Mb

    Re: 500GB - 1TB superfast scratch storage options

    Quote Originally Posted by DanceswithUnix View Post
    Now I haven't tried this, but I am wondering if another PC running FreeNas would work here?

    This would allow you to use ZFS across a large number of spindles. Lots of spindles means lots of IO per second, and gigabit ethernet is probably fast enough and certainly faster than a single drive thrashing around on a faster interface. You can add an SSD to a ZFS setup to smooth writes
    Hmm I hadn't thought about going this far, but did think of an intermediate solution. My current NAS box is an HP Microserver with 4x 2TB drives driven off a proper p410i RAID controller with 512MB cache. So that is actually relatively fast and the half GB cache helps. It's WHS v2 running on it and I was contemplating making some of the storage iSCSI to make it appear relatively fast on my workstation, bypassing the inefficiencies of CIFS-based sharing...

  14. #14
    Senior Member
    Join Date
    Apr 2010
    Posts
    492
    Thanks
    2
    Thanked
    32 times in 23 posts

    Re: 500GB - 1TB superfast scratch storage options

    Quote Originally Posted by tfboy View Post
    I can't see how having a small cache will help as I'm effectively randomly accessing several 100GB at a time, or doing sequential read / write on the same disk which cripples its performance.
    As I said, it highly depends on your how optimal your software is. Even for file copying there are quite a few programs claiming to be better than the Windows copy (which doesn't mean they really are). I couldn't find good real world benchmarks of FancyCache, but I found one user reporting a marginal (~10%) gain: 23 seconds with FancyCache vs. 26 seconds without for copying 3GB from an SSD onto itself. It's possible that with an HDD you'll gain more from the cache.

    It's free to try, so I don't see why you'd want to find arguments against it when it's pretty easy to test it for yourself. If it doesn't gain you anything, you've lost a little time trying it. If it does gain you something, you've gained a useful tool (that's free for now).

    If you do try it (and I hope you will), please post your results.

  15. #15
    Senior Member
    Join Date
    Jul 2003
    Location
    Reading, Berkshire
    Posts
    1,253
    Thanks
    64
    Thanked
    53 times in 34 posts
    • tfboy's system
      • Motherboard:
      • MSI X470 Gaming Plus
      • CPU:
      • AMD Ryzen 7 2700
      • Memory:
      • 2x8GB Corsair Vengeance LPX)
      • Storage:
      • Force MP600 1TB PCIe SSD
      • Graphics card(s):
      • 560 Ti
      • PSU:
      • Corsair RM 650W
      • Case:
      • CM Silencio 550
      • Operating System:
      • W10 Pro
      • Monitor(s):
      • HP LP2475w + Dell 2001FP
      • Internet:
      • VM 350Mb

    Re: 500GB - 1TB superfast scratch storage options

    Not having a digg, it might work. I'll give it a try next time I'm doing some work and will post results (positive or otherwise)

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •