Raid storage and backup advice needed

[Palhil]

Hi

I hope you'll be able to give me some technical and buying advice. Please excuse how wordy this is. I want to make sure I explain the situation correctly.

I work for a small animation company and we have been working on a project that requires some live action filming. We shoot HD footage on a Panasonic P2 camera which has two 64gb cards. We use the first card until it is full and then swap to the second so that we can keep shooting continuously. Whilst using the second I need to copy all the data off the first card onto a hard drive so that the card can be reformatted for use once the second card is full. For safety I have to make sure that I always have two copies of the data at anyone time.

On previous recording days I have copied the data via a USB card reader and my laptop to two separate portable hard drives. The trouble is that copying the gigabytes of data twice can often take longer than the amount of time that it takes to fill up the second card in the camera. So it often turns into a juggling act as I shift the data around. Sometimes requiring me to just dump the files on my laptop's smallish hard drive because at least I can get it there quicker. At the end of the day I take the two portable hard drives back to the office and make another archive copy of the files to larger network storage in the building. Again this is just for safety. I can't use the data from archive storage when I'm editing because it simply can't access the data quickly enough. This mean that I have to work from the portable drives connected directly to my computer.

So my question is could I use a small raid box with removable hard drives to speed things up?

What I'm imagining is that I have four 1TB drives in the box. They are mirrored into two pairs. So that when I copy the data from the camera card the raid box automatically makes a second copy. Once I get it back to the office I would still make a copy to the archive. However, I also want to be able to remove the two copied hard drives so that they are off site back ups. I then continue to use the half empty raid box as I edit the footage. I won't actually be modifying the data on the drives as I edit, I just need to be able to access it.

I'm not very clear on raid set up so I'm not even sure if what I've just said is possible. If I take two of the drives out would it still function? And would it then continue working without any loss of data when I put the two drives back in for the next shoot? I want to keep all the footage we've shot from previous days. Which is why I would be using terabyte drives.

If this makes any sense and you could recommend a reasonably portable raid box or similar, I would be most happy. Otherwise it's back to juggling data.

I've been given a £700 budget but I obviously want to make sure it can work before I spend anything.

Thanks.

[nonstandard]

Looks like those Panasonic P2 cards are 4 x RAID-striped SD cards in a box with a PCMCIA interface. If that's the case, the theoretical max. transfer rate of the P2 is ~32MB/s, which is likely the choke-point in your workflow. Maybe you could speed things up by copying the files from the P2 to the USB disk as normal, but have a file-sync program (like rsync or Unison) running that can copy from the usb disk to another disk at the same time? Of course, you could get another P2 card - I might be tempted to see what happens if you put 4 x 32GB class 6 SDHC cards in a P2 casing? There is a very flexible build-it-yourself solution that will do all sorts of RAID and rsync or Unison copying as well - FreeNAS . . . if you Google for bit-tech.net and freenas there's a good article on their recent experiences building a FreeNAS storage device.
HTH

[ehhhhhhh]

Looks like those Panasonic P2 cards are 4 x RAID-striped SD cards in a box with a PCMCIA interface.

??? raid striped sd cards???


on the topic though...

if you buy a nas and you have 100mbit connection then it wont be faster than 10MB/s

I would get a 1000mbit switch...if you connect both the PC and the NAS to it, then they should communicate at 1000mbit...otherwise you have USB3 as an option.

Hopefully someone here can recommend a reliable NAS.

btw, why don't you ask your company to buy more cards for the cam? if it affects productivity then your bosses shouldn't hesitate to get more cards asap.

'm not very clear on raid set up so I'm not even sure if what I've just said is possible. If I take two of the drives out would it still function? And would it then continue working without any loss of data when I put the two drives back in for the next shoot? I want to keep all the footage we've shot from previous days. Which is why I would be using terabyte drives.

If you take out the drives it should function, but it's not a good idea

Again, your company should support you with adequate IT infrastructure (fast network, fast portable drives {SSDs?}) to get your job done easily. I'm not even sure why *you* need to search for solutions?

[nonstandard]

"??? raid striped sd cards???"

- yup! I'm not making it up, go and fetch the docs from www.electra-tv.com/pdf/P2_booklet.pdf
and look on page #2. The reason I mentioned it was that P2 cards cost *way more* than the bill of parts, so hunting for end-of-life ones and upgrading the capacity might be a winner for the experimentally minded. Wonder if anyone has teardown pics of a P2?

If you had something with RAID-1 (mirrored) storage on board, then you could copy once (leaving you free to use the P2 again, as you have 2x copies, one on each of the mirrored drives), then sync this data with a (for e.g) another portable USB drive using rsync/Unison/whatever, then you could take the drive home at the end of the day, providing an offsite copy overnight. The 'something' at the top of the paragraph could be a NAS, or just a desktop with a matched pair of RAIDed drives.

[scaryjim]

Copying onto mirrored media then unmirroring them does not sound like a great idea. Theoretically it's possible, yes, but I really wouldn't want to be relying on it for mission critical tasks. As ehhhhhh says, the company needs to be providing you with suitable equipment to do your job, which in this case sounds like *a lot* more P2 storage cards. Either that or they need to upgrade your recording equipment to something with a much higher native capacity.

The fact that you're seriously considering doing something as goldsteinien as carrying a raided NAS around so you can make 2 copies of the same files quickly is a clear indication that the entire strategy needs rethinking: the current situation clearly isn't working for you. So go back to the drawing board and work out what you actually need to make the job manageable, then work out how to implement that. Don't throw good money at a bad process: it won't help

[peterb]

Personally I think the bottleneck is in the data transfer. You either need additional SD cards so that you aren't swapping one with one - just adding a third would make life easier. However having just looked it the eyewatering price f the 64Gb cards...

The spec shows that the transfer speed of the card is 1.2Gb/sec,which is much faster than you will get over a lan. One solution might be to get external drives with an Esata interface, and buy an Esata/PCMCIA interface card. Ttat would certaily speed up the transfer process.

You might want to look at this http://usbnow.co.uk/articles/tag/esata-pcmcia/

This is a company I have deat with (I bought a PC express/SD card reader from them) Fast service. Esata/PCMCIA cards are also sold by Amazon - athough I would be tempted to buy from a specialist.

Asaka make a range of caddies that will take 2,5 or 3.5" SATA drives and presents either an Esata or USB interface. You just add the drive. Scan are a good source,

http://www.scan.co.uk/Shop/Computer-...25%29-IDE-SATA

http://www.scan.co.uk/products/akasa...enclosure-only

I have used the Asaka housings - quite robust so should be OK for field work.

[jim]

Despite the cost, I would just get a third card. Would be a lot easier for you in the end.

[shaithis]

Something like this might be worth looking in to. I know someone that swears by it for archiving his photos on-the-move, not sure if it would truly full-fill your needs though.

[peterb]

Despite the cost, I would just get a third card. Would be a lot easier for you in the end.

At nearly $1000 a pop (if I looked at the right device) not a cheap option!

Something like this might be worth looking in to. I know someone that swears by it for archiving his photos on-the-move, not sure if it would truly full-fill your needs though.

I thought about something like but the interface on the camera card is (again if I understood it correctly) PCMCIA, so it is a fairly specialised device. (And that is no longer available (from Amazon at least) although they do have others with smallercapacity)

[Biscuit]

Which panasonic camera is it? If its one of the higher models like HPX500 then you should have an HDSDI port. Using this combined with a cheap blackmagic decklink in a PC which has a decent RAID1 array would probably make your entire workflow a lot more straight forward.

[Palhil]

Thanks for all the great advice and info.

So it looks like I'll have to have a rethink. The cost of P2 cards mean that it's not an easy option to just buy more. As I said I only have a budget of £700. Budgets are always really tight and we are a small company. So unfortunately there is always going to be an element of just muddling along with what we've got. I can always continue doing what I'm already doing it's just a bit of a headache at times.

I'll have a look at some of the options you've given me.

Biscuit: I believe the camera is a HPX171. It does have a HD-SDI so I'll have a look at the Blackmagic Deck link.

Thanks guys. Really appreciate your comments.

[Biscuit]

No Problem.

If you do have any broacdcast/pro video type questions im quite happy to pass on more of my details as i work for a big systems integrator.

[jim]

At nearly $1000 a pop (if I looked at the right device) not a cheap option!

You're right, I don't disagree. But dropping £700 on a half-baked solution that will likely overcomplicate things to the point of it being wasted money, and putting that money towards a third card might be the best option in the end. It's certainly the easiest, and easiest solutions tend to be the best.

[Funkstar]

At nearly $1000 a pop (if I looked at the right device) not a cheap option!

Starting at about £660

http://www.google.co.uk/products?q=6...ed=0CC8QrQQwAA

[Biscuit]

Depends if your comfortable using the E-Series which have a shorter life span (IIRC)

[peterb]

Express card -> esata adaptor £20
2.5" drive caddy (x2) max £30
2* 2.5" drives say £100
Ocmcia-> expresscard adaptor (if required) £20

Plus £20 max for cables and so on - total £170 (and probably a little less)

Process would be:

copy camera drive to laptop
Copy laptop to external drive one,
Copy laptop to external drive 2
Reformat camera drive.
(if push came to shove, the camera drive could be reformatted after taking the first copy).

Still a bit labour intensive, but certainly low cost, and should remove the back up bottleneck caused by transferring over the USB2 link and means the camera isn't taken out of use while downloading. (although if that isn't important and/ the speed of the HD-SDI link is sufficient that might not matter)

Biscuit's solution is certainly more elegant and less labour intensive, but does require a PC (I haven't found any blackmagic SDI express/PCMCIA cards) which might not be as convenient, and almost certainly be pushing the £700 budget if you had to build a dedicated PC for the purpose.