Huge RAID required - how do I do it

[Olimain]

I am going to be needing to implement a system for archiving hours of uncompressed video and am wondering how to do it or if it will need a proper company to come to our offices and install it.

I would like allow for storage 30-50TB of video and it must be secure in that if one disk fails I do not lose everything stored on it.

I guess if it was on a small scale in a PC with 4 drives I would have 2 pairs of striped drives mirrored? How is this done on a large scale and what would be a ballpark cost?

Thanks...

[scaryjim]

I'd generally recommend two mirrored drives striped It gives slightly better fault tolerance.

Given that 2TB disks are available now you can quite easily get up to 8TB of mirrored / striped storage. Beyond that and you're out of my area of knowledge I'm afraid (I doubt I'll work in a situation that requires 8TB of storage for a long time). It's probably a far better idea to go round some proper storage companies and get quotes / recommendations. The scale of storage you're looking for is going to need some very enterprisey hardware indeed...

[DR]

I'd get a proper company in to do it.

https://secure.dnuk.com/store/storage-disk.php

Unless of course the content is worth nothing then feel free to bodge it and string a few HDD's off a PC.....

[Olimain]

Haha, well the person who's content it is feels it's worth something so probably best I don't

[Olimain]

Thanks for the link... Can you recommend them or was it just a company you came across?

[Moby-Dick]

Some form of storage server would probably be required - the 3par units can certinaly fit that level of storage in 4u of rack space.

[arthurleung]

Compress it to some lossless codec and you'll immediate cut that requirement by 50~80%. Once you cut it down to about 12TB, it will fit in 8x2TB RAID6, or 12x1.5TB RAID6

Good read on lossless codec
http://compression.ru/video/codec_co...codecs_en.html
Although it heavily depends on the video source

[Olimain]

This will be uncompressed digital betacam broadcast quality captured from tape. I can't start compressing it, even with so called lossless codecs there is still quality degradation.

[Funkstar]

even with so called lossless codecs there is still quality degradation.

ok, i'll bite. If a codec is lossless (ie. there is no loss in information, just the storage size) how on earth will there be a quality degredation?

Does a piece of software run slower after it has been zipped then un-zipped? no. Lossless compression is exactly the same.

And I'll agree with DR, if this is a commercial project, get a vendor that specialises in this kind of thing to supply what you need. SMC, Dell, HP and Sun all produce NAD, DAS and SAN systems that will do what you need. Granted they will be a lot more expensive that rolling your own, but at least they come with a service contract.

[Olimain]

There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.

[Pob255]

True lossless is but most "lossless" compression codecs are not truly lossless.

Have you considered keeping it on the tape? if correctly stored it's a very good backup medium.

[Funkstar]

There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.

True lossless is but most "lossless" compression codecs are not truly lossless.

Then they aren't lossless.

Is the video industry that messed up that they are touting codecs as bing lossless are actually aren't? At least in the audio side of things, lossless means lossless. FLAC, ALAC, WMAlossless, WavPack and MonkeyAudio are all true lossles codecs.

I fail to see why there can't be a true lossless video codec. But this conversation is straying OT now

[arthurleung]

There's no such thing as a truly lossless AVI compressor - though no actual detail gets lost, noise and grain levels will increase and are noticeable, particularly on darker areas of the image especially if viewed on a grade 1 monitor.

That is perhaps you use compressor that doesn't match your colorspace. I use Lagarith and I've never seen it give me anything other than bit-identical.

There is not much difference between compressing a AVI file with Winzip/rar/7zip than lossless codec (providing you use the correct colourspace), other than the algorithm is optimized for video content, and it can be played directly by media players without need to first extracting the files.

Speaking of tapes, even stuff like LTO-2 gives 200GB native (Should take about 400GB of your uncompressed AVI with tape compression).
Though you can't store much with just 200GB?

This will be uncompressed digital betacam broadcast quality captured from tape

Perhaps you can just keep those tapes?

[Olimain]

True lossless is but most "lossless" compression codecs are not truly lossless.

Have you considered keeping it on the tape? if correctly stored it's a very good backup medium.

Unfortunately space to store 20000 tapes comes at quite a price, hence digitising the lot.

[Olimain]

That is perhaps you use compressor that doesn't match your colorspace. I use Lagarith and I've never seen it give me anything other than bit-identical.

I haven't used Lagarith but will look in to it and see how it compares - thanks for that.