Results 1 to 15 of 15

Thread: Huge RAID required - how do I do it

  1. #1
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Huge RAID required - how do I do it

    I am going to be needing to implement a system for archiving hours of uncompressed video and am wondering how to do it or if it will need a proper company to come to our offices and install it.

    I would like allow for storage 30-50TB of video and it must be secure in that if one disk fails I do not lose everything stored on it.

    I guess if it was on a small scale in a PC with 4 drives I would have 2 pairs of striped drives mirrored? How is this done on a large scale and what would be a ballpark cost?

    Thanks...

  2. #2
    Not a good person scaryjim's Avatar
    Join Date
    Jan 2009
    Location
    Manchester
    Posts
    15,070
    Thanks
    1,200
    Thanked
    2,250 times in 1,851 posts
    • scaryjim's system
      • Motherboard:
      • Dell Inspiron
      • CPU:
      • Core i5 8250U
      • Memory:
      • 1x 8GB DDR4 2400
      • Storage:
      • 128GB M.2 SSD + 1TB HDD
      • Graphics card(s):
      • Radeon R5 230
      • PSU:
      • Battery/Dell brick
      • Case:
      • Dell Inspiron 5570
      • Operating System:
      • Windows 10
      • Monitor(s):
      • 15" 1080p laptop panel

    Re: Huge RAID required - how do I do it

    I'd generally recommend two mirrored drives striped It gives slightly better fault tolerance.

    Given that 2TB disks are available now you can quite easily get up to 8TB of mirrored / striped storage. Beyond that and you're out of my area of knowledge I'm afraid (I doubt I'll work in a situation that requires 8TB of storage for a long time). It's probably a far better idea to go round some proper storage companies and get quotes / recommendations. The scale of storage you're looking for is going to need some very enterprisey hardware indeed...

  3. #3
    DR
    DR is offline
    on ye old ship HEXUS DR's Avatar
    Join Date
    Jul 2003
    Location
    HEXUS HQ, Elstree
    Posts
    13,375
    Thanks
    1,035
    Thanked
    786 times in 355 posts

    Re: Huge RAID required - how do I do it

    I'd get a proper company in to do it.

    https://secure.dnuk.com/store/storage-disk.php

    Unless of course the content is worth nothing then feel free to bodge it and string a few HDD's off a PC.....

  4. #4
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    Haha, well the person who's content it is feels it's worth something so probably best I don't

  5. #5
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    Thanks for the link... Can you recommend them or was it just a company you came across?

  6. #6
    Administrator Moby-Dick's Avatar
    Join Date
    Jul 2003
    Location
    There's no place like ::1 (IPv6 version)
    Posts
    10,664
    Thanks
    53
    Thanked
    385 times in 314 posts

    Re: Huge RAID required - how do I do it

    Some form of storage server would probably be required - the 3par units can certinaly fit that level of storage in 4u of rack space.
    my Virtualisation Blog http://jfvi.co.uk Virtualisation Podcast http://vsoup.net

  7. #7
    YUKIKAZE arthurleung's Avatar
    Join Date
    Feb 2005
    Location
    Aberdeen
    Posts
    3,280
    Thanks
    8
    Thanked
    88 times in 83 posts
    • arthurleung's system
      • Motherboard:
      • Asus P5E (Rampage Formula 0902)
      • CPU:
      • Intel Core2Quad Q9550 3.6Ghz 1.2V
      • Memory:
      • A-Data DDR2-800 2x2GB CL4
      • Storage:
      • 4x1TB WD1000FYPS @ RAID5 3Ware 9500S-8 / 3x 1TB Samsung Ecogreen F2
      • Graphics card(s):
      • GeCube HD4870 512MB
      • PSU:
      • Corsair VX450
      • Case:
      • Antec P180
      • Operating System:
      • Windows Server 2008 Standard
      • Monitor(s):
      • Dell Ultrasharp 2709W + 2001FP
      • Internet:
      • Be*Unlimited 20Mbps

    Re: Huge RAID required - how do I do it

    Compress it to some lossless codec and you'll immediate cut that requirement by 50~80%. Once you cut it down to about 12TB, it will fit in 8x2TB RAID6, or 12x1.5TB RAID6

    Good read on lossless codec
    http://compression.ru/video/codec_co...codecs_en.html
    Although it heavily depends on the video source
    Workstation 1: Intel i7 950 @ 3.8Ghz / X58 / 12GB DDR3-1600 / HD4870 512MB / Antec P180
    Workstation 2: Intel C2Q Q9550 @ 3.6Ghz / X38 / 4GB DDR2-800 / 8400GS 512MB / Open Air
    Workstation 3: Intel Xeon X3350 @ 3.2Ghz / P35 / 4GB DDR2-800 / HD4770 512MB / Shuttle SP35P2
    HTPC: AMD Athlon X4 620 @ 2.6Ghz / 780G / 4GB DDR2-1000 / Antec Mini P180 White
    Mobile Workstation: Intel C2D T8300 @ 2.4Ghz / GM965 / 3GB DDR2-667 / DELL Inspiron 1525 / 6+6+9 Cell Battery

    Display (Monitor): DELL Ultrasharp 2709W + DELL Ultrasharp 2001FP
    Display (Projector): Epson TW-3500 1080p
    Speakers: Creative Megaworks THX550 5.1
    Headphones: Etymotic hf2 / Ultimate Ears Triple.fi 10 Pro

    Storage: 8x2TB Hitachi @ DELL PERC 6/i RAID6 / 13TB Non-RAID Across 12 HDDs
    Consoles: PS3 Slim 120GB / Xbox 360 Arcade 20GB / PS2

  8. #8
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    This will be uncompressed digital betacam broadcast quality captured from tape. I can't start compressing it, even with so called lossless codecs there is still quality degradation.

  9. #9
    Does he need a reason? Funkstar's Avatar
    Join Date
    Aug 2005
    Location
    Aberdeen
    Posts
    19,874
    Thanks
    630
    Thanked
    965 times in 816 posts
    • Funkstar's system
      • Motherboard:
      • Gigabyte EG45M-DS2H
      • CPU:
      • Intel Core2Quad Q9550 (2.83GHz)
      • Memory:
      • 8GB OCZ PC2-6400C5 800MHz Quad Channel
      • Storage:
      • 650GB Western Digital Caviar Blue
      • Graphics card(s):
      • 512MB ATI Radeon HD4550
      • PSU:
      • Antec 350W 80+ Efficient PSU
      • Case:
      • Antec NSK1480 Slim Mini Desktop Case
      • Operating System:
      • Vista Ultimate 64bit
      • Monitor(s):
      • Dell 2407 + 2408 monitors
      • Internet:
      • Zen 8mb

    Re: Huge RAID required - how do I do it

    Quote Originally Posted by Olimain View Post
    even with so called lossless codecs there is still quality degradation.
    ok, i'll bite. If a codec is lossless (ie. there is no loss in information, just the storage size) how on earth will there be a quality degredation?

    Does a piece of software run slower after it has been zipped then un-zipped? no. Lossless compression is exactly the same.

    And I'll agree with DR, if this is a commercial project, get a vendor that specialises in this kind of thing to supply what you need. SMC, Dell, HP and Sun all produce NAD, DAS and SAN systems that will do what you need. Granted they will be a lot more expensive that rolling your own, but at least they come with a service contract.

  10. #10
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.

  11. #11
    Senior Member Pob255's Avatar
    Join Date
    Apr 2007
    Location
    The land of Brum
    Posts
    9,945
    Thanks
    572
    Thanked
    1,196 times in 1,099 posts
    • Pob255's system
      • Motherboard:
      • Asus M5A99X EVO
      • CPU:
      • PhenomII x4 955 & CM Hyper 212+
      • Memory:
      • 2x 4gb Corsair Vengence 1600mhz cas9
      • Storage:
      • 250gb SATA Westerndigital + 500gb samsung +1tb Samsung
      • Graphics card(s):
      • Asus GTX560ti CuII
      • PSU:
      • Be Quiet E9 680w
      • Case:
      • HAF 912+
      • Operating System:
      • W7 Pro
      • Monitor(s):
      • Dell 21" ips something

    Re: Huge RAID required - how do I do it

    True lossless is but most "lossless" compression codecs are not truly lossless.

    Have you considered keeping it on the tape? if correctly stored it's a very good backup medium.

  12. #12
    Does he need a reason? Funkstar's Avatar
    Join Date
    Aug 2005
    Location
    Aberdeen
    Posts
    19,874
    Thanks
    630
    Thanked
    965 times in 816 posts
    • Funkstar's system
      • Motherboard:
      • Gigabyte EG45M-DS2H
      • CPU:
      • Intel Core2Quad Q9550 (2.83GHz)
      • Memory:
      • 8GB OCZ PC2-6400C5 800MHz Quad Channel
      • Storage:
      • 650GB Western Digital Caviar Blue
      • Graphics card(s):
      • 512MB ATI Radeon HD4550
      • PSU:
      • Antec 350W 80+ Efficient PSU
      • Case:
      • Antec NSK1480 Slim Mini Desktop Case
      • Operating System:
      • Vista Ultimate 64bit
      • Monitor(s):
      • Dell 2407 + 2408 monitors
      • Internet:
      • Zen 8mb

    Re: Huge RAID required - how do I do it

    Quote Originally Posted by Olimain View Post
    There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.
    Quote Originally Posted by Pob255 View Post
    True lossless is but most "lossless" compression codecs are not truly lossless.
    Then they aren't lossless.

    Is the video industry that messed up that they are touting codecs as bing lossless are actually aren't? At least in the audio side of things, lossless means lossless. FLAC, ALAC, WMAlossless, WavPack and MonkeyAudio are all true lossles codecs.

    I fail to see why there can't be a true lossless video codec. But this conversation is straying OT now

  13. #13
    YUKIKAZE arthurleung's Avatar
    Join Date
    Feb 2005
    Location
    Aberdeen
    Posts
    3,280
    Thanks
    8
    Thanked
    88 times in 83 posts
    • arthurleung's system
      • Motherboard:
      • Asus P5E (Rampage Formula 0902)
      • CPU:
      • Intel Core2Quad Q9550 3.6Ghz 1.2V
      • Memory:
      • A-Data DDR2-800 2x2GB CL4
      • Storage:
      • 4x1TB WD1000FYPS @ RAID5 3Ware 9500S-8 / 3x 1TB Samsung Ecogreen F2
      • Graphics card(s):
      • GeCube HD4870 512MB
      • PSU:
      • Corsair VX450
      • Case:
      • Antec P180
      • Operating System:
      • Windows Server 2008 Standard
      • Monitor(s):
      • Dell Ultrasharp 2709W + 2001FP
      • Internet:
      • Be*Unlimited 20Mbps

    Re: Huge RAID required - how do I do it

    Quote Originally Posted by Olimain View Post
    There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.
    That is perhaps you use compressor that doesn't match your colorspace. I use Lagarith and I've never seen it give me anything other than bit-identical.

    There is not much difference between compressing a AVI file with Winzip/rar/7zip than lossless codec (providing you use the correct colourspace), other than the algorithm is optimized for video content, and it can be played directly by media players without need to first extracting the files.

    Speaking of tapes, even stuff like LTO-2 gives 200GB native (Should take about 400GB of your uncompressed AVI with tape compression).
    Though you can't store much with just 200GB?

    This will be uncompressed digital betacam broadcast quality captured from tape
    Perhaps you can just keep those tapes?
    Last edited by arthurleung; 25-03-2009 at 06:13 PM.
    Workstation 1: Intel i7 950 @ 3.8Ghz / X58 / 12GB DDR3-1600 / HD4870 512MB / Antec P180
    Workstation 2: Intel C2Q Q9550 @ 3.6Ghz / X38 / 4GB DDR2-800 / 8400GS 512MB / Open Air
    Workstation 3: Intel Xeon X3350 @ 3.2Ghz / P35 / 4GB DDR2-800 / HD4770 512MB / Shuttle SP35P2
    HTPC: AMD Athlon X4 620 @ 2.6Ghz / 780G / 4GB DDR2-1000 / Antec Mini P180 White
    Mobile Workstation: Intel C2D T8300 @ 2.4Ghz / GM965 / 3GB DDR2-667 / DELL Inspiron 1525 / 6+6+9 Cell Battery

    Display (Monitor): DELL Ultrasharp 2709W + DELL Ultrasharp 2001FP
    Display (Projector): Epson TW-3500 1080p
    Speakers: Creative Megaworks THX550 5.1
    Headphones: Etymotic hf2 / Ultimate Ears Triple.fi 10 Pro

    Storage: 8x2TB Hitachi @ DELL PERC 6/i RAID6 / 13TB Non-RAID Across 12 HDDs
    Consoles: PS3 Slim 120GB / Xbox 360 Arcade 20GB / PS2

  14. #14
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    Quote Originally Posted by Pob255 View Post
    True lossless is but most "lossless" compression codecs are not truly lossless.

    Have you considered keeping it on the tape? if correctly stored it's a very good backup medium.
    Unfortunately space to store 20000 tapes comes at quite a price, hence digitising the lot.

  15. #15
    Member
    Join Date
    Dec 2008
    Posts
    122
    Thanks
    8
    Thanked
    4 times in 4 posts

    Re: Huge RAID required - how do I do it

    Quote Originally Posted by arthurleung View Post
    That is perhaps you use compressor that doesn't match your colorspace. I use Lagarith and I've never seen it give me anything other than bit-identical.
    I haven't used Lagarith but will look in to it and see how it compares - thanks for that.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Old School - KT7A Raid USB Problem - Help Required
    By Knoxville in forum abit.care@HEXUS
    Replies: 8
    Last Post: 07-07-2006, 11:12 AM
  2. Old School - KT7A Raid USB Problem - Help Required
    By Knoxville in forum Help! Quick Relief From Tech Headaches
    Replies: 0
    Last Post: 05-07-2006, 05:34 AM
  3. Expandable RAID / JBOD
    By mikerr in forum PC Hardware and Components
    Replies: 1
    Last Post: 08-04-2006, 07:02 PM
  4. RAID 0 or 5?
    By Bleek in forum PC Hardware and Components
    Replies: 37
    Last Post: 26-03-2006, 11:16 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •