Page 1 of 2 12 LastLast
Results 1 to 16 of 25

Thread: Moving large amounts of data

  1. #1
    Registered+
    Join Date
    Mar 2008
    Posts
    17
    Thanks
    0
    Thanked
    0 times in 0 posts
    • Reggie12's system
      • Motherboard:
      • EVGA SR-2 Classified
      • CPU:
      • 2 x Xeon X5680 @ 3.33 GHz
      • Memory:
      • 12GB Dominator GT
      • Storage:
      • 2 x 256GB SSD in Raid
      • Graphics card(s):
      • 2 x EVGA GTX 580. Tesla C2070
      • PSU:
      • Silverstone Strider 1500 watt
      • Case:
      • Lian Li PC-7
      • Operating System:
      • Win 7 Ult 64 bit Linux dual boot
      • Monitor(s):
      • 3 x Samsung 24" LED

    Moving large amounts of data

    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?

  2. #2
    Moderator chuckskull's Avatar
    Join Date
    Apr 2006
    Location
    The Frozen North
    Posts
    7,713
    Thanks
    951
    Thanked
    690 times in 463 posts
    • chuckskull's system
      • Motherboard:
      • Gigabyte Z77-D3H
      • CPU:
      • 3570k @ 4.7 - H100i
      • Memory:
      • 32GB XMS3 1600mhz
      • Storage:
      • 256GB Samsung 850 Pro + 3TB Seagate
      • Graphics card(s):
      • EVGA GTX 980Ti Classified
      • PSU:
      • Seasonic M12 700W
      • Case:
      • Corsair 500R
      • Operating System:
      • Windows 10 Pro
      • Monitor(s):
      • Asus VG278HE
      • Internet:
      • FTTC

    Re: Moving large amounts of data

    Only one I can think of would be 5 2TB discs or 20 500GB might be quicker actually, depending on where the bottleneck is. Networking and in particular the internet have come a long way, but it's still pretty hard to beat just putting it on a disc and moving it physically.

  3. #3
    Administrator Moby-Dick's Avatar
    Join Date
    Jul 2003
    Location
    There's no place like ::1 (IPv6 version)
    Posts
    10,662
    Thanks
    53
    Thanked
    383 times in 313 posts

    Re: Moving large amounts of data

    depends on the distance between A & B
    my Virtualisation Blog http://jfvi.co.uk Virtualisation Podcast http://vsoup.net

  4. #4
    Formerly known as Andehh Andeh13's Avatar
    Join Date
    Oct 2005
    Location
    Northampton
    Posts
    3,335
    Thanks
    855
    Thanked
    256 times in 151 posts
    • Andeh13's system
      • Motherboard:
      • Gigabyte GA-P35
      • CPU:
      • Intel Q6600
      • Memory:
      • 4gb Corsair XMS2 800mhz
      • Storage:
      • 1 x 250gb Western Digital AAKS, 2 x 500gb Western Digital AAKS, 1TB WD Caviar Green
      • Graphics card(s):
      • BFG Geforce 8800GTS 512mb
      • PSU:
      • Corsair HX520
      • Case:
      • Antec 900
      • Operating System:
      • Windows 7 64bit
      • Monitor(s):
      • Samsung 24" & Sony 17"
      • Internet:
      • Virgin 10mb... hate them!

    Re: Moving large amounts of data

    lots of SSDs?

  5. #5
    Studmuffin Flibb's Avatar
    Join Date
    Jul 2003
    Location
    Kent
    Posts
    4,904
    Thanks
    31
    Thanked
    324 times in 277 posts
    • Flibb's system
      • Motherboard:
      • Gigabyte GA-970A-UD3
      • CPU:
      • AMD FX-6300
      • Memory:
      • 16GB Crucial Ballistix DDR3 PC3-12800
      • Storage:
      • Samsung SSD 840 EVO 250G
      • Graphics card(s):
      • 3GB MSI Radeon HD 7950 Twin Frozr
      • PSU:
      • FSP
      • Operating System:
      • Win7 64bit
      • Monitor(s):
      • Deffl TFT thing

    Re: Moving large amounts of data

    Option 1. Take drive out and put it in the the 2nd rig
    Option 2 (computers in same room). Use a long sata cable to connect the drive to the 2nd computer. I did this using IDE ables before USB2 was common.

  6. #6
    Now with added Ruffus Dog Tattysnuc's Avatar
    Join Date
    Feb 2006
    Location
    Liverpool
    Posts
    1,347
    Thanks
    283
    Thanked
    206 times in 132 posts
    • Tattysnuc's system
      • Motherboard:
      • Asus Rampage IV Extreme
      • CPU:
      • Core i7 4690X @ stock
      • Memory:
      • 32Gb Assorted Corsair
      • Storage:
      • Sandisk Extreme 480Gb SSD
      • Graphics card(s):
      • eVGA Classified Hydros in SLI
      • PSU:
      • eVGA 1250W
      • Case:
      • Phobya Big Case
      • Operating System:
      • Win 10 Pro
      • Monitor(s):
      • Dell 3007WFP
      • Internet:
      • Virgin 30Mb

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?
    Was the data copied from A onto the 10 hot swappable disks and then onto B? so the disks were the transport mechanism or was the data physically removed from machine A and inserted into Machine B.

    If the later is to be considered, this will take considerably longer than 5 minutes based on my SATA 2 backup times for 2Tb...

    Also, if Method A is employed, then there is no availability of that data while in transit, nor is there any backup if a drive should fail, so I'm trying to understand the point to the question....
    Join the HEXUS Folding @ home team

  7. #7
    Senior Member kalniel's Avatar
    Join Date
    Aug 2005
    Posts
    29,195
    Thanks
    1,520
    Thanked
    2,933 times in 2,376 posts
    • kalniel's system
      • Motherboard:
      • Gigabyte X58A UD3R rev 2
      • CPU:
      • Intel Xeon X5680
      • Memory:
      • 12gb DDR3 2000
      • Graphics card(s):
      • nVidia GTX 1060 6GB
      • PSU:
      • Seasonic 600W
      • Case:
      • Cooler Master HAF 912
      • Operating System:
      • Win 10 Pro x64
      • Monitor(s):
      • Dell U2311H
      • Internet:
      • O2 8mbps

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?
    Don't move the data, just move the query.

  8. #8
    Editable... jimbouk's Avatar
    Join Date
    Aug 2005
    Location
    Bristol
    Posts
    2,548
    Thanks
    188
    Thanked
    188 times in 149 posts
    • jimbouk's system
      • Motherboard:
      • ASUS M5 A97 R2.0
      • CPU:
      • AMD FX-6300
      • Memory:
      • 8GB DDR3
      • Storage:
      • 128GB SSD + 500GB HDD
      • Graphics card(s):
      • AMD Radeon HD 7870
      • PSU:
      • Corsair HX 520W
      • Case:
      • Some large ATX case
      • Operating System:
      • Windows 7
      • Monitor(s):
      • Dell U2713HM
      • Internet:
      • Orange...

    Re: Moving large amounts of data

    Quote Originally Posted by kalniel View Post
    Don't move the data, just move the query.
    Hehe, 'ln -s', no-one will ever know the data isn't really there...

  9. #9
    Registered+
    Join Date
    Mar 2008
    Posts
    17
    Thanks
    0
    Thanked
    0 times in 0 posts
    • Reggie12's system
      • Motherboard:
      • EVGA SR-2 Classified
      • CPU:
      • 2 x Xeon X5680 @ 3.33 GHz
      • Memory:
      • 12GB Dominator GT
      • Storage:
      • 2 x 256GB SSD in Raid
      • Graphics card(s):
      • 2 x EVGA GTX 580. Tesla C2070
      • PSU:
      • Silverstone Strider 1500 watt
      • Case:
      • Lian Li PC-7
      • Operating System:
      • Win 7 Ult 64 bit Linux dual boot
      • Monitor(s):
      • 3 x Samsung 24" LED

    Re: Moving large amounts of data

    I think the point of the question was to make us think differently. All of us came back with an answer along the lines of "100 gigabit ethernet" "infiniband" "invent a means of connecting two machines together via PCI-E".
    but to answer some other posts, whats the fastest method with between two machines in the same room? and whats the fastest way when they are 500 miles apart? (while keeping the data available on the first machine

  10. #10
    Senior Member
    Join Date
    Jul 2008
    Posts
    1,060
    Thanks
    6
    Thanked
    84 times in 78 posts

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    I think the point of the question was to make us think differently. All of us came back with an answer along the lines of "100 gigabit ethernet" "infiniband" "invent a means of connecting two machines together via PCI-E".
    but to answer some other posts, whats the fastest method with between two machines in the same room? and whats the fastest way when they are 500 miles apart? (while keeping the data available on the first machine
    physically moving was the first through when i saw the thread title earlier. as you say, it depends on the distance and budget. sticking the drives on the back of a motorcycle courier or by car/train/plane might get the data there quicker than any other method (ie. internet)

    i usually just pull drives in and out machines when copying large amounts of data, partly because my network isn't gigabit. i'll maybe use a flashdrive or external usb for larger amounts of data, or external SATA for big loads

    back in the day we had to write code on bits of paper and retype into our zx81's as it was quicker than saving on a tape recorder and loading back up again

  11. #11
    Senior Member oolon's Avatar
    Join Date
    Mar 2007
    Location
    London
    Posts
    2,294
    Thanks
    150
    Thanked
    302 times in 248 posts
    • oolon's system
      • Motherboard:
      • Asus P6T6
      • CPU:
      • Xeon w3680
      • Memory:
      • 3*4GB Kingston ECC
      • Storage:
      • 160GB Intel G2 SSD
      • Graphics card(s):
      • XFX HD6970 2GB
      • PSU:
      • Corsair HX850
      • Case:
      • Antec P183
      • Operating System:
      • Windows 7 Ultimate and Centos 5
      • Monitor(s):
      • Dell 2408WFP
      • Internet:
      • Be* Unlimied 6 down/1.2 up

    Re: Moving large amounts of data

    1) On line replication so the data is already there at the destination using a link (could just be the new server).... rsync for file level stuff or some san based system for block level devices.

    We used rsync to move 5.7TB of data from a linux server to a new AIX server while still running on the linux one.

    2) Using an SVC (IBM virtual san) to duplicate data on one san to another san while we are working with the data on the orginal san, then ship the san to the destination, another SVC will then allow use of data from the "portable" san while it copies it in background to the new san.

    3) Use mirrored drives.
    Take one of the disks from the mirror out, and give it a new disk to rebuild onto.
    Take the other disk from the mirror to your destination and put that in the new machine and get it to mirror up with a new blank.

    4) Back up the server onto tape, before the key date then write a quick increment to the backup and restore the increment.

    All of these I have done in real data centre migrations. (In total we move 46TB of data and 50+ servers)

    The key problems are.... links are slow, writting/reading disks are slow, down time is bad.

    Disks
    So if you can write the drive or use the link WHILE still running the server and then the same at the other end you have saved most of the time. The thing is it is NOT the time taken to transport the data that is an issue its the time to read and write the disks.

    When using links you to save a MASSIVE transportation time, you need to drip feed the destination.

    Other notes:
    LTO4 (800GB) is faster than most hard drives at writing and reading.
    Coping disks at file level is more than twice as slow as copying the whole disk (due to head seeking).
    (\__/) All I wanted in the end was world domination and a whole lot of money to spend. - NMA
    (='.*=)
    (")_(*)

  12. #12
    Senior Member
    Join Date
    Mar 2005
    Posts
    4,593
    Thanks
    145
    Thanked
    311 times in 249 posts
    • badass's system
      • Motherboard:
      • ASUS P8Z77-m pro
      • CPU:
      • Core i5 3570K
      • Memory:
      • 32GB
      • Storage:
      • 1TB Samsung 850 EVO, 2TB WD Green
      • Graphics card(s):
      • Radeon RX 580
      • PSU:
      • Corsair HX520W
      • Case:
      • Silverstone SG02-F
      • Operating System:
      • Windows 10 X64
      • Monitor(s):
      • Del U2311, LG226WTQ
      • Internet:
      • 80/20 FTTC

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?
    IPoAC


    P.S. If it was 5 minutes, he was lying. Unless the 1TB disks were in fact very fast SSD's.
    Even writing to all 10 Disks at once, you'll only get around 1GB/sec so 15 minutes minimum, assuming no other bottlenecks.

    However I suspect the point he was making was "never underestimate the bandwidth of a truck full of tapes" in a different way.
    "In a perfect world... spammers would get caught, go to jail, and share a cell with many men who have enlarged their penises, taken Viagra and are looking for a new relationship."

  13. #13
    Senior Member oolon's Avatar
    Join Date
    Mar 2007
    Location
    London
    Posts
    2,294
    Thanks
    150
    Thanked
    302 times in 248 posts
    • oolon's system
      • Motherboard:
      • Asus P6T6
      • CPU:
      • Xeon w3680
      • Memory:
      • 3*4GB Kingston ECC
      • Storage:
      • 160GB Intel G2 SSD
      • Graphics card(s):
      • XFX HD6970 2GB
      • PSU:
      • Corsair HX850
      • Case:
      • Antec P183
      • Operating System:
      • Windows 7 Ultimate and Centos 5
      • Monitor(s):
      • Dell 2408WFP
      • Internet:
      • Be* Unlimied 6 down/1.2 up

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?
    Sorry I call *rubbish* to that... you can't do that in the real world, I suspect he was not using source drives and use a program to generate 10GB of data. Even worse he may have used Zero data which does not get written to drives using "sparse files". A using random device will not generate number fast enough, using the same file over and over again will result in memory caching of the read. I bet he also didn't sync and unmount the drives so writes could have been cached.

    (10*1000/(5*60))*8 = 266 Gigabits per second... right! I believe you (33Gbits per second per drive)... I think not. he is claiming 5 times faster than SATA 3 (SAS 2), per drive... sorry something feels very wrong... I think we have ... the following going on: Look in to my eyes.. Look in to my eyes... and we are back in the room.

    Oh and finally a 1 TB drive cannot fit 1 TB on it due to FS over head and also drivers use decimal sizes and computers use binary, so they are 2.5% smaller than you might expect.

    (personally I think he use 10 Terra bits not 10 TiB... but thats just me.)

    I CALL shenanigans!

    Let us look at an enterprise level drive.

    cheetah-15k

    First its only 600GB, so he is using a SATA disk to get that size an it is slower than that....

    shenanigans

    Delivers high performance with sustained data rates up to 171MB/s Peek spec is rate (600MB/s)

    He is claiming 4 GB/s.

    shenanigans

    Sorry..... get to class and call him out...

    shenanigans shenanigans shenanigans shenanigans

    If drives were that fast why does it take a day for mirrored drives to sync 2TB, or to build a 4x2TB Raid 5, ok disks are operating in a degraded rate however he has a system that blows everything else out of the water...
    Last edited by oolon; 18-02-2011 at 02:59 PM.
    (\__/) All I wanted in the end was world domination and a whole lot of money to spend. - NMA
    (='.*=)
    (")_(*)

  14. #14
    Pork & Beans Powerup Phage's Avatar
    Join Date
    May 2009
    Location
    Kent
    Posts
    6,100
    Thanks
    1,523
    Thanked
    578 times in 494 posts
    • Phage's system
      • Motherboard:
      • Asus Crosshair VI
      • CPU:
      • 1700x @ 3.9
      • Memory:
      • 16Gb Corsair LPX
      • Storage:
      • Samsung 960 512Gb + 2Tb Seagate SSHD
      • Graphics card(s):
      • EVGA 1080ti
      • PSU:
      • BeQuiet 850w
      • Case:
      • Fractal R4
      • Operating System:
      • W10 64
      • Monitor(s):
      • Agon Gsync

    Re: Moving large amounts of data

    An Estate full of tapes.
    Society's to blame,
    Or possibly Atari.

  15. #15
    duc
    duc is offline
    Senior Member
    Join Date
    Aug 2005
    Posts
    287
    Thanks
    26
    Thanked
    32 times in 28 posts
    • duc's system
      • Motherboard:
      • Lenovo ThinkPad T440S
      • CPU:
      • Intel Core i5 4210U
      • Memory:
      • 12GB DDR3
      • Storage:
      • 128GB MSATA
      • Graphics card(s):
      • Intel
      • Case:
      • Lenovo ThinkPad T440S
      • Operating System:
      • Windows 10 Pro 64bit
      • Monitor(s):
      • 14" FHD
      • Internet:
      • Vodafone Fibre

    Re: Moving large amounts of data

    Quote Originally Posted by Reggie12 View Post
    sort of a hypothetical question, a lecturer once challenged a group of us to come up with the fastest way to move a huge amount of data, 10TB.
    we were given 24 hours to come up with a better idea than him, needless to say, we all lost (trying to be clever and over complicated), he did it in less than 5 minutes with 10 hot swappable 1TB disks
    is there a faster way?
    The way I read this it seems all he did was physically removed the 10 'hot swappable' 1TB drives from one machine and installed them into another, thus moving 10TB of data in 5 minutes. To go quicker would be to use less hard drives.

  16. #16
    Senior Member oolon's Avatar
    Join Date
    Mar 2007
    Location
    London
    Posts
    2,294
    Thanks
    150
    Thanked
    302 times in 248 posts
    • oolon's system
      • Motherboard:
      • Asus P6T6
      • CPU:
      • Xeon w3680
      • Memory:
      • 3*4GB Kingston ECC
      • Storage:
      • 160GB Intel G2 SSD
      • Graphics card(s):
      • XFX HD6970 2GB
      • PSU:
      • Corsair HX850
      • Case:
      • Antec P183
      • Operating System:
      • Windows 7 Ultimate and Centos 5
      • Monitor(s):
      • Dell 2408WFP
      • Internet:
      • Be* Unlimied 6 down/1.2 up

    Re: Moving large amounts of data

    Good point DUC... its that is "ALL" he did, that seems a bit of a stupid question as you do need compatible servers, if its the boot drive of the server will the OS even boot on the new machine etc. The amount of data seems of little importance either, its nothing much to do with computers and more to do with mechanics. You "could" use DAS storage and just plug the cable into another computer etc... not matter if it is one bit or 1000TB.... its just mechanics.

    Well if its just physically moving from one device to another like that.... to go quicker put it on a san and share the disk between two servers! In a cluster... time to move the storage .... ZERO! I WIN.......

    You could also use DRDB to block level replicate the data, so its on "both" at the same time.

    If it must only be on one server... reconfiguring the san, would take less than a minute, less if you automated it with a script that would be seconds.
    Last edited by oolon; 18-02-2011 at 03:49 PM.
    (\__/) All I wanted in the end was world domination and a whole lot of money to spend. - NMA
    (='.*=)
    (")_(*)

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. FREE for 3 days only - EASEUS Data Recovery Wizard
    By Mithrandir in forum Retail Therapy and Bargains
    Replies: 8
    Last Post: 18-03-2010, 10:13 PM
  2. Mr Brown Saving us 'Again'?
    By TheAnimus in forum Question Time
    Replies: 21
    Last Post: 29-05-2008, 03:28 PM
  3. Business Server - Data Backup Methods
    By cptwhite_uk in forum Networking and Broadband
    Replies: 12
    Last Post: 27-02-2008, 12:25 AM
  4. Replies: 4
    Last Post: 11-06-2004, 07:39 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •