Building a Xeon system for a ZFS "Server" - Need your advice please!

[bsodmike]

Hi all,

Over the years folks here have given me solid advice, and once again I look forward to receiving the same You've saved my bacon in the past so here goes.

I'm looking to setup a Xeon X99 build, and want to check if this configuration makes sense, or if there is a cheaper mainboard you could potentially suggest.

The challenge however, is that any and all parts need to be purchasable from B&H USA (http://www.bhphotovideo.com/) as their DHL shipping is cheap for me.

1. CPU: Intel Xeon E5-2620 v4 2.1 GHz Eight-Core LGA 2011-3 Processor (https://www.bhphotovideo.com/c/produ...20_v4_2_1.html)

2. ASUS X99-E WS/USB 3.1 mainboard (https://www.bhphotovideo.com/c/produ...on.html?sts=pi)

At first it seemed like the above motherboard did not support v4 Xeons, but having checked the Asus site it states "E5-2620 V4(2.10GHz,85W,L3:20M,8C,HT)" is supported in the latest BIOS.

Thoughts on this combo? Also, thoughts on a decent cooler - should I go with say a Hyper 212 EVO or an AIO loop?

Thanks again chaps!

[jim]

If you're not going to overclock I doubt the cooler will make much difference.

On the motherboard, if it's the latest BIOS only then your only risk is that the board doesn't ship with the latest BIOS - at least I've seen that issue before, where you need a budget CPU just to kick things off.

I guess you're aware that AMD Ryzen is potentially the better value option, and that Intel and AMD's next gen professional kit is meant to be on the way very shortly? It doesn't feel like a good time to buy X99, although it's never a good time to buy PC kit in some ways.

Why did you want the X99-E WS in particular?

[bsodmike]

On the motherboard, if it's the latest BIOS only then your only risk is that the board doesn't ship with the latest BIOS - at least I've seen that issue before, where you need a budget CPU just to kick things off.

Asus has a nifty flash from USB feature without needing a CPU - I believe this one has that feature as well. Good point though!

Why did you want the X99-E WS in particular?

I'd admit that I'm partial to Asus given I'm running a regular X99/Z170 here and their UEFI is easy enough to get around; at least, I've not had any particularly painful issues.

The X99-E WS is the only compatible X99 with ECC that I could find on B&H; it also has the handy LED post-code read outs; plenty of RAM expansion, enough PCIe slots, at least 1x M.2; the Dual GigE ports is a plus too.

Certainly open to alternatives that may be cheaper, if they have most of those features?

I guess you're aware that AMD Ryzen is potentially the better value option

Yes, however, pairing ECC and getting it to work is somewhat hit and miss. In my case, I'm pretty much buying blind, and have to actually test the combo once it arrives. RMAs and reordering/taxes make it an expensive affair to get it 'wrong'.

Another complication of the Asus board is the CEB form-factor. The case I was planning to put this in probably won't be compatible (Basic ATX).

[b0redom]

As with everything what's the use case? If you just want a server to export ZFS then you don't need anything like that grunt. How many users etc?

Depending on the OS you're looking you might be better off with some component choices than others. FreeNAS/BSD/IllumOS or Linux with user space ZFS etc.

[bsodmike]

As with everything what's the use case? If you just want a server to export ZFS then you don't need anything like that grunt. How many users etc?

Depending on the OS you're looking you might be better off with some component choices than others. FreeNAS/BSD/IllumOS or Linux with user space ZFS etc.

Not many users at all, but it'll be the primary ZFS store and be on 24/7. I'm planning either debian/CentOS or Ubuntu LTS running headless in any case.

@b0redom, let me know what you have in mind for 'less grunt' please. I'm curious.

I found a decent case that is CEB compatible, and can hold a decent number of drives as well - Antec P380 Full-Tower (https://www.bhphotovideo.com/c/produ..._computer.html)

[scaryjim]

What are essential specs, and what's optional/WIBNI?

i.e. how many SATA, how many M.2, how much RAM, how many PCIe etc.? There's a range of bare-bones servers on B&H that are probably worth looking at, as they'll basically do a lot of the compatibility work for you, but it's hard to know which ones to suggest without a better idea of the minimal specs you're looking for...

[Biscuit]

Honestly for your use case I would probably suggest looking for more "off the shelf" products with a single overall warranty rather than a custom build. Something from Dell, Lenovo, HP, etc...

[bsodmike]

Hey Jim,

I just went over the specs on the Asus X99-E WS, those are pretty much what I'm after. 8 Sata 6Gb/s, Dual gigE, at least 1x M.2 etc. I'm leaning towards this configuration for the time-being - thanks!

[bsodmike]

Honestly for your use case I would probably suggest looking for more "off the shelf" products with a single overall warranty rather than a custom build. Something from Dell, Lenovo, HP, etc...

Thing is I hate having to deal with customer support; I usually fix my stuff 99.99% of the time. No, 100% of the time.

[scaryjim]

... 8 Sata 6Gb/s, Dual gigE, at least 1x M.2 etc. ...

You mentioned "enough" PCIe slots - how many is that? What PCIe cards are going in it?

I get the feeling from your earlier posts that this is for your place of employment? i.e. it' snot for personal use, and you don't actually outright own the company? If that is the case, please get something pre-built with a service contract! The fact that you like fixing all your own stuff is great, but what does the company do if you're not there? Unless you like being on 24/7 callout, that is, but personally I wouldn't...

[b0redom]

Not many users at all, but it'll be the primary ZFS store and be on 24/7. I'm planning either debian/CentOS or Ubuntu LTS running headless in any case.

@b0redom, let me know what you have in mind for 'less grunt' please. I'm curious.

I'm not sure I'd want to run ZFS on a Linux variant, but that's down to your own risk profile. The reason I run FreeNAS is that it's baked into the OS - there's no conflict of license types etc. Anyway, for reference, my ZFS server is running on a LeNovo TS140 with a Xeon E3-1225 and 16GB of non-ECC RAM. It's running 5x4TB disks in RAID-Z1, and a Plex server (inside a jail) which does transcoding to Amazon Fire sticks/Roku boxes/LG TV.

Looking back over the last few weeks, I've never seen CPU usage over 20%.

ZFS really cares about RAM rather than CPU unless you're doing data de-dup (which you shouldn't) etc.

You've still not really given me enough to give you a more complete answer.

• How many users is 'not many users at all'?
• How much I/O are you expecting? A SMB share usually eats way less I/O than an iSCSI boot device etc.
• Is this in an office with a required uptime?
• (How) are you planning on backing up etc?
• What level of RAID are you planning etc.
• Will the server be doing anything else apart from ZFS filesharing?

I'd also echo what ScaryJim says, if this is for corporate use, buy something you can get support on. Whilst ZFS is great in that you can just lift and shift the disks to another chassis, import them and you're golden, if this is business critical, I'd be looking at something a bit more bullet proof.

[bsodmike]

Thanks scaryjim/b0redom,

I really appreciate the advice; now I understand the concern.
This setup is meant to operate as an extension to the 2x Synology NASes I'm running at home. There is a component of client data that is critical to my 'personal' work/business.

Your recommendations for pre-built with service contracts are spot on, but something I'd only do if this was for a 3rd-party/large organisation.

The reason I run FreeNAS is that it's baked into the OS - there's no conflict of license types etc. Anyway, for reference, my ZFS server is running on a LeNovo TS140 with a Xeon E3-1225 and 16GB of non-ECC RAM. It's running 5x4TB disks in RAID-Z1, and a Plex server (inside a jail) which does transcoding to Amazon Fire sticks/Roku boxes/LG TV.

Thanks b0redom, that's actually the sort of use case I'm after.
- Users: 1-5 max
- SMB shares over GigE.
- Uptime: useful to have. I've managed to run my hand-built systems 24/7 for over a year now. My Synology NAS (2x units) have been running 24/7 since December 2014.
- Raidz2 with 6 disks (to start).
- Given there's free memory, I may run some docker containers.

[b0redom]

I would seriously consider FreeNAS as an OS then. The new version has support for VMs and containers. Be aware though that ZFS is very memory hungry.

[bsodmike]

I would seriously consider FreeNAS as an OS then. The new version has support for VMs and containers. Be aware though that ZFS is very memory hungry.

Thanks mate. I ordered this 32GB RDIMM kit from crucial
http://www.crucial.com/usa/en/ct2k16g4rfd4213

I can always add a second kit to expand to 64GB RAM if needed.