Page 1 of 3 123 LastLast
Results 1 to 16 of 38

Thread: 10Gbps home network, on the cheap. Sort of.

  1. #1
    Senior Member Shad's Avatar
    Join Date
    Jul 2003
    Location
    In front
    Posts
    2,773
    Thanks
    22
    Thanked
    40 times in 24 posts

    10Gbps home network, on the cheap. Sort of.

    Inspired by the series of articles by David Hunt I too decided that I fancied having a crack at a super fast home network without spending a fortune. 10G Ethernet cards are still £200-300 a pop on eBay which makes even a simple PC-to-PC connection quite expensive. On the other hand, older generation SDR and DDR (10Gbps and 20Gbps respectively) InfiniBand adapters are available for a fraction of the cost. The downside is the extra hassle to set it up and the restrictions with cabling (very short and a bit pricey or useful length and very pricey).

    So to whet your appetite here's a benchmark I just took:



    To achieve this I've used:

    • 2x Mellanox InfiniHost III MHEA28-XT dual port 10Gbps (£45 each)
    • 1x Zarlink ZLynx 30m CX4-CX4 20Gbps fibre optic cable (£170 )
    • 1x Dell PE2950 (2x E5160, 12GB) running Windows Server 2008 R2 and Microsoft iSCSI Target, and a 3.5GB RAM disk
    • 1x Phenom II 965 workstation running Windows 7 x64 and Microsoft iSCSI Initiator


    The cost of the cable I had to buy was a killer, but cheaper cables are available! The trouble I have is the distance between the server and the workstation. Although it's only about 13 metres and some 15m cables are available, the cost difference between 20m and 30m is so little it makes more sense to get the longer one and have some options down the road. There is effectively zero signal degradation over fibre (so the extra length doesn't matter) whereas the short copper cables suffer quite quickly and it's rare to find them even as 'long' as 10m.



    As David explains in his articles, InfiniBand works using RDMA which basically means the adapters read/write directly to the system memory in each connected machine, completely bypassing any protocols that slow things down. However, there is currently no way to achieve this in a Windows only environment (such as mine) and you need a Linux system to create the SRP target to enable the true high speed RDMA connectivity. Us Windows folk are stuck with IP over InfiniBand (IPoIB) which emulates TCP/IP and allows all the normal networking features. The downside is that it's all handled by the host CPU so a lot of the benefit of the 10Gb adapters is lost - for now.

    So at the moment I'm quite happy to have 'only' 400MB/s sequential read as well as some other pretty good numbers. In the future I will probably create a Linux system, but at least there is headroom there if I find that the current speed isn't sufficient. My next step is to buy some disks for a RAID array on the server as the RAM disk is only for testing and proof of concept.

    Also worth noting, Windows Server 2012 and SMB3 include support for RDMA on Mellanox's latest HCAs, but although these run at 40Gbps+ they are the same cost if not more than 10G Ethernet adapters. Microsoft and Mellanox look unlikely to add support for older (cheaper) IB adapters unfortunately, so Linux is the way to go to get the best speed possible at the moment.

    Hope that's interesting and/or useful
    Simon


  2. Received thanks from:

    Agent (15-09-2012),Apex (17-09-2012),blueball (16-09-2012),ehhhhhhh (17-09-2012),Funkstar (15-09-2012),Jonatron (16-09-2012),peterb (16-09-2012),pollaxe (16-09-2012)

  3. #2
    HEXUS.social member Agent's Avatar
    Join Date
    Jul 2003
    Location
    Internet
    Posts
    19,168
    Thanks
    735
    Thanked
    1,607 times in 1,045 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    Awesome post.

    I've been considering something similar for a while but I've not really done any research into it. I'm thinking several Gigabit links for me is going to be enough (and cheaper as I have all the equipment), but i'll defo consider this. I'd also need external fibre - no idea on the cost of that.

    Shame about the Windows issues, it's what I'd be using too.

    Thanks for this.
    Quote Originally Posted by Saracen View Post
    And by trying to force me to like small pants, they've alienated me.

  4. #3
    F.A.S.T. Butuz's Avatar
    Join Date
    Jul 2003
    Location
    Wales
    Posts
    4,708
    Thanks
    51
    Thanked
    72 times in 59 posts
    • Butuz's system
      • Motherboard:
      • MSI Z77 MPOWER
      • CPU:
      • I7 3770K @ 4.6
      • Memory:
      • 16GB Corsair XMS 1866
      • Storage:
      • Sandisk SSDs
      • Graphics card(s):
      • 3xR9 290
      • PSU:
      • be quiet! Dark Power Pro 10
      • Case:
      • Inwin H Frame
      • Operating System:
      • Windows 7

    Re: 10Gbps home network, on the cheap. Sort of.

    You'd better have some SERIOUS storage hardware to make 10g worth using on a handful of PCs.

    Most people's rigs can't even saturate 1Gbe regularly..

    Butuz

  5. #4
    F.A.S.T. Butuz's Avatar
    Join Date
    Jul 2003
    Location
    Wales
    Posts
    4,708
    Thanks
    51
    Thanked
    72 times in 59 posts
    • Butuz's system
      • Motherboard:
      • MSI Z77 MPOWER
      • CPU:
      • I7 3770K @ 4.6
      • Memory:
      • 16GB Corsair XMS 1866
      • Storage:
      • Sandisk SSDs
      • Graphics card(s):
      • 3xR9 290
      • PSU:
      • be quiet! Dark Power Pro 10
      • Case:
      • Inwin H Frame
      • Operating System:
      • Windows 7

    Re: 10Gbps home network, on the cheap. Sort of.

    For some reason your Diskmark screenies handnt shown up when I replied.

    What raid config are you using on the 2950?

    Nice one!

    Butuz

  6. #5
    Comfortably Numb directhex's Avatar
    Join Date
    Jul 2003
    Location
    /dev/urandom
    Posts
    17,074
    Thanks
    228
    Thanked
    1,027 times in 678 posts
    • directhex's system
      • Motherboard:
      • Asus ROG Strix B550-I Gaming
      • CPU:
      • Ryzen 5900x
      • Memory:
      • 64GB G.Skill Trident Z RGB
      • Storage:
      • 2TB Seagate Firecuda 520
      • Graphics card(s):
      • EVGA GeForce RTX 3080 XC3 Ultra
      • PSU:
      • EVGA SuperNOVA 850W G3
      • Case:
      • NZXT H210i
      • Operating System:
      • Ubuntu 20.04, Windows 10
      • Monitor(s):
      • LG 34GN850
      • Internet:
      • FIOS

    Re: 10Gbps home network, on the cheap. Sort of.

    IB in the home?

    You're a twisted, crazy man. Y'know that?

    Oh, and don't look at the price of IB switches

  7. #6
    Senior Member Shad's Avatar
    Join Date
    Jul 2003
    Location
    In front
    Posts
    2,773
    Thanks
    22
    Thanked
    40 times in 24 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    Quote Originally Posted by Agent View Post
    I'd also need external fibre - no idea on the cost of that.

    Shame about the Windows issues, it's what I'd be using too.
    It's not perfect but you could use the same sort of cable I've bought and run it inside some ducting/trunking to protect it. The only annoying thing is the connectors aren't removable from the fibre without special tools, as far as I know. But they're quite compact really.

    The whole IPoIB thing is annoying. At the moment it's using about 20% CPU on the server and 5-10% on my workstation, so I don't know if I've reached the limit with IPoIB period or if a pair of DDR/20Gbps adapters would double the speeds. I'm clearly not CPU limited at the moment. I'm going to ask Mellanox and see what they think (@mellanoxtech on Twitter, they're really helpful ). I could also be running into an MS iSCSI Target limitation so I'm going to try StarWind later and see if that's any different.

    Quote Originally Posted by Butuz View Post
    What raid config are you using on the 2950?
    Actually no RAID at the moment, I'm benchmarking with a RAM disk to push any limitations to the iSCSI or IPoIB layers, so I know what to expect before buying some disks. I'm probably going to go for 4x OCZ Agility 4 or Vertex 4 drives, which although will only run at SATA2 on the PERC 6/i I'm using, should still be capable of 800-900MB/s in RAID0 and give me headroom in the future if I upgrade to an H700 card for SATA3 goodness. It depends how nicely they play with the PERC. It's a bit of a moot point though with the IPoIB restriction in place, but I may end up changing things around a bit and using ESXi 5.1 to run a Linux flavour with the SSDs and use the IB adapter in pass through/Directed I/O. Doing that has the potential to max out the IB adapters

    Quote Originally Posted by directhex View Post
    IB in the home?

    You're a twisted, crazy man. Y'know that?

    Oh, and don't look at the price of IB switches
    Yeah... I know

    I want to use this storage for video editing but it's mainly just about trying it out really, it's all new to me. I've looked at IB switches on eBay... OMG, will be sticking to daisy chaining systems for now!!

    Cheers guys, I'll post some more when I've got the RAID running
    Simon


  8. #7
    Registered+
    Join Date
    Sep 2012
    Posts
    20
    Thanks
    0
    Thanked
    2 times in 2 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    can i ask why you have done this? Looks a lot of time, research and money spent to achieve something which may save you the odd second?

  9. #8
    Oh Crumbs.... Biscuit's Avatar
    Join Date
    Feb 2007
    Location
    N. Yorkshire
    Posts
    11,193
    Thanks
    1,394
    Thanked
    1,091 times in 833 posts
    • Biscuit's system
      • Motherboard:
      • MSI B450M Mortar
      • CPU:
      • AMD 2700X (Be Quiet! Dark Rock 3)
      • Memory:
      • 16GB Patriot Viper 2 @ 3466MHz
      • Storage:
      • 500GB WD Black
      • Graphics card(s):
      • Sapphire R9 290X Vapor-X
      • PSU:
      • Seasonic Focus Gold 750W
      • Case:
      • Lian Li PC-V359
      • Operating System:
      • Windows 10 x64
      • Internet:
      • BT Infinity 80/20

    Re: 10Gbps home network, on the cheap. Sort of.

    interesting little experiment... makes me want to try something new

  10. #9
    Dark side super agent
    Join Date
    Dec 2003
    Location
    Nirvana
    Posts
    1,895
    Thanks
    72
    Thanked
    99 times in 89 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    Totally unnecessary, expensive (relatively) and time consuming for very little benefit. Brilliant! Thanks for pointing everyone the right way
    An Atlantean Triumvirate, Ghosts of the Past, The Centre Cannot Hold
    The Pillars of Britain, Foundations of the Reich, Cracks in the Pillars.

    My books are available here for Amazon Kindle. Feedback always welcome!

  11. #10
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    Quote Originally Posted by pmcx9 View Post
    can i ask why you have done this? Looks a lot of time, research and money spent to achieve something which may save you the odd second?
    Ah, you're obviously missing the point, read what Bluecube said

    @Shad, where did you get the IB adapters from?

  12. #11
    Registered+
    Join Date
    Sep 2012
    Posts
    20
    Thanks
    0
    Thanked
    2 times in 2 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    I think I have missed the point. I have reread the opening post and cant see what benefit there is at home unless shad has money and time to burn. I am not criticising, just couldnt quite work out what he is trying to achieve that a normal home network can't albeit milliseconds slower.

  13. #12
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    You got it, its the difference between home user(s) and enthusiasts that this experiment comes down to...

  14. #13
    Dark side super agent
    Join Date
    Dec 2003
    Location
    Nirvana
    Posts
    1,895
    Thanks
    72
    Thanked
    99 times in 89 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    And it's fun no doubt! Better than sitting in front of the TV watching repeats of Jeremy Kyle. Or even worse - Sky plussed Jeremy Kyle...
    An Atlantean Triumvirate, Ghosts of the Past, The Centre Cannot Hold
    The Pillars of Britain, Foundations of the Reich, Cracks in the Pillars.

    My books are available here for Amazon Kindle. Feedback always welcome!

  15. #14
    F.A.S.T. Butuz's Avatar
    Join Date
    Jul 2003
    Location
    Wales
    Posts
    4,708
    Thanks
    51
    Thanked
    72 times in 59 posts
    • Butuz's system
      • Motherboard:
      • MSI Z77 MPOWER
      • CPU:
      • I7 3770K @ 4.6
      • Memory:
      • 16GB Corsair XMS 1866
      • Storage:
      • Sandisk SSDs
      • Graphics card(s):
      • 3xR9 290
      • PSU:
      • be quiet! Dark Power Pro 10
      • Case:
      • Inwin H Frame
      • Operating System:
      • Windows 7

    Re: 10Gbps home network, on the cheap. Sort of.

    Sky plussed Jeremy Kyle...???!!!!

    You need to take a good long hard look at your life

    Butuz

  16. #15
    Dark side super agent
    Join Date
    Dec 2003
    Location
    Nirvana
    Posts
    1,895
    Thanks
    72
    Thanked
    99 times in 89 posts

    Re: 10Gbps home network, on the cheap. Sort of.

    Not me. I got that from an episode of the Royle Family!
    An Atlantean Triumvirate, Ghosts of the Past, The Centre Cannot Hold
    The Pillars of Britain, Foundations of the Reich, Cracks in the Pillars.

    My books are available here for Amazon Kindle. Feedback always welcome!

  17. #16
    boop, got your nose stevie lee's Avatar
    Join Date
    Sep 2007
    Location
    southport
    Posts
    2,670
    Thanks
    414
    Thanked
    422 times in 318 posts
    • stevie lee's system
      • Motherboard:
      • ASUS ROG STRIX B450-F Gaming
      • CPU:
      • Ryzen 3600 (stock cooler)
      • Memory:
      • 16 GB Corsair 3600 MHZ Cas 18
      • Storage:
      • Crucial 250GB BX500, Hitachi 640GB + 1TB. WDblue 2TB, Crucial M500 240GB
      • Graphics card(s):
      • sapphire 7700
      • PSU:
      • bluestorm II 400w
      • Case:
      • Xigmatech Midgard
      • Operating System:
      • Win 10 Home
      • Monitor(s):
      • 22" Samsung SM2233BW
      • Internet:
      • 20mb plusnet unlimited!

    Re: 10Gbps home network, on the cheap. Sort of.

    Quote Originally Posted by pmcx9 View Post
    can i ask why you have done this? Looks a lot of time, research and money spent to achieve something which may save you the odd second?
    erm, you do realise what hexus is dont you? its a place where nutters and tech enthusiasts get together ad do wierd things that serve no purpose other than to say "ive got shiny things, look at the shiny shiny".

    and the seconds shad saves at home, he can then spend trying to shave a few seconds off his lap times, which saves a few seconds off his lap times, which mans at the end of the day he can fit another lap in. or something like that

Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •