Moving large amounts of data

[Reggie12]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

[chuckskull]

Only one I can think of would be 5 2TB discs or 20 500GB might be quicker actually, depending on where the bottleneck is. Networking and in particular the internet have come a long way, but it's still pretty hard to beat just putting it on a disc and moving it physically.

[Moby-Dick]

depends on the distance between A & B

[Andeh13]

lots of SSDs?

[Flibb]

Option 1. Take drive out and put it in the the 2nd rig
Option 2 (computers in same room). Use a long sata cable to connect the drive to the 2nd computer. I did this using IDE ables before USB2 was common.

[Tattysnuc]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

Was the data copied from A onto the 10 hot swappable disks and then onto B? so the disks were the transport mechanism or was the data physically removed from machine A and inserted into Machine B.

If the later is to be considered, this will take considerably longer than 5 minutes based on my SATA 2 backup times for 2Tb...

Also, if Method A is employed, then there is no availability of that data while in transit, nor is there any backup if a drive should fail, so I'm trying to understand the point to the question....

[kalniel]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

Don't move the data, just move the query.

[jimbouk]

Don't move the data, just move the query.

Hehe, 'ln -s', no-one will ever know the data isn't really there...

[Reggie12]

I think the point of the question was to make us think differently. All of us came back with an answer along the lines of "100 gigabit ethernet" "infiniband" "invent a means of connecting two machines together via PCI-E".
but to answer some other posts, whats the fastest method with between two machines in the same room? and whats the fastest way when they are 500 miles apart? (while keeping the data available on the first machine

[Unique]

I think the point of the question was to make us think differently. All of us came back with an answer along the lines of "100 gigabit ethernet" "infiniband" "invent a means of connecting two machines together via PCI-E".
but to answer some other posts, whats the fastest method with between two machines in the same room? and whats the fastest way when they are 500 miles apart? (while keeping the data available on the first machine

physically moving was the first through when i saw the thread title earlier. as you say, it depends on the distance and budget. sticking the drives on the back of a motorcycle courier or by car/train/plane might get the data there quicker than any other method (ie. internet)

i usually just pull drives in and out machines when copying large amounts of data, partly because my network isn't gigabit. i'll maybe use a flashdrive or external usb for larger amounts of data, or external SATA for big loads

back in the day we had to write code on bits of paper and retype into our zx81's as it was quicker than saving on a tape recorder and loading back up again

[oolon]

1) On line replication so the data is already there at the destination using a link (could just be the new server).... rsync for file level stuff or some san based system for block level devices.

We used rsync to move 5.7TB of data from a linux server to a new AIX server while still running on the linux one.

2) Using an SVC (IBM virtual san) to duplicate data on one san to another san while we are working with the data on the orginal san, then ship the san to the destination, another SVC will then allow use of data from the "portable" san while it copies it in background to the new san.

3) Use mirrored drives.
Take one of the disks from the mirror out, and give it a new disk to rebuild onto.
Take the other disk from the mirror to your destination and put that in the new machine and get it to mirror up with a new blank.

4) Back up the server onto tape, before the key date then write a quick increment to the backup and restore the increment.

All of these I have done in real data centre migrations. (In total we move 46TB of data and 50+ servers)

The key problems are.... links are slow, writting/reading disks are slow, down time is bad.

Disks
So if you can write the drive or use the link WHILE still running the server and then the same at the other end you have saved most of the time. The thing is it is NOT the time taken to transport the data that is an issue its the time to read and write the disks.

When using links you to save a MASSIVE transportation time, you need to drip feed the destination.

Other notes:
LTO4 (800GB) is faster than most hard drives at writing and reading.
Coping disks at file level is more than twice as slow as copying the whole disk (due to head seeking).

[badass]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

IPoAC


P.S. If it was 5 minutes, he was lying. Unless the 1TB disks were in fact very fast SSD's.
Even writing to all 10 Disks at once, you'll only get around 1GB/sec so 15 minutes minimum, assuming no other bottlenecks.

However I suspect the point he was making was "never underestimate the bandwidth of a truck full of tapes" in a different way.

[oolon]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

Sorry I call *rubbish* to that... you can't do that in the real world, I suspect he was not using source drives and use a program to generate 10GB of data. Even worse he may have used Zero data which does not get written to drives using "sparse files". A using random device will not generate number fast enough, using the same file over and over again will result in memory caching of the read. I bet he also didn't sync and unmount the drives so writes could have been cached.

(10*1000/(5*60))*8 = 266 Gigabits per second... right! I believe you (33Gbits per second per drive)... I think not. he is claiming 5 times faster than SATA 3 (SAS 2), per drive... sorry something feels very wrong... I think we have ... the following going on: Look in to my eyes.. Look in to my eyes... and we are back in the room.

Oh and finally a 1 TB drive cannot fit 1 TB on it due to FS over head and also drivers use decimal sizes and computers use binary, so they are 2.5% smaller than you might expect.

(personally I think he use 10 Terra bits not 10 TiB... but thats just me.)

I CALL shenanigans!

Let us look at an enterprise level drive.

cheetah-15k

First its only 600GB, so he is using a SATA disk to get that size an it is slower than that....

shenanigans

Delivers high performance with sustained data rates up to 171MB/s Peek spec is rate (600MB/s)

He is claiming 4 GB/s.

shenanigans

Sorry..... get to class and call him out...

shenanigans shenanigans shenanigans shenanigans

If drives were that fast why does it take a day for mirrored drives to sync 2TB, or to build a 4x2TB Raid 5, ok disks are operating in a degraded rate however he has a system that blows everything else out of the water...

[Phage]

An Estate full of tapes.

[duc]

sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
is there a faster way?

The way I read this it seems all he did was physically removed the 10 'hot swappable' 1TB drives from one machine and installed them into another, thus moving 10TB of data in 5 minutes. To go quicker would be to use less hard drives.

[oolon]

Good point DUC... its that is "ALL" he did, that seems a bit of a stupid question as you do need compatible servers, if its the boot drive of the server will the OS even boot on the new machine etc. The amount of data seems of little importance either, its nothing much to do with computers and more to do with mechanics. You "could" use DAS storage and just plug the cable into another computer etc... not matter if it is one bit or 1000TB.... its just mechanics.

Well if its just physically moving from one device to another like that.... to go quicker put it on a san and share the disk between two servers! In a cluster... time to move the storage .... ZERO! I WIN.......

You could also use DRDB to block level replicate the data, so its on "both" at the same time.

If it must only be on one server... reconfiguring the san, would take less than a minute, less if you automated it with a script that would be seconds.