Supermicro Dual CPU motherboard

[Biscuit]

Hi everyone!

So I recently decided to upgrade my file server to enable me to offload a lot of virtual machine tasks to it. Instead of spending fortunes on new and modern server hardware or run the 'risk' of using regular consumer hardware, I decided to do a bit of ebay-ing.

I ended up with a LGA1356 motherboard, 2 x E5-2428L CPUs (60w 6C/12T), 24GB ECC registered Samsung RAM all for under £300.... happy days.

The problem I seem to be having is getting everything working. I was hoping (as always) for plug and play, but whenever I run dual CPUs I hit issues with memory errors. I have checked the RAM, tested with RAM out of my old server and even some none-ecc stuff and I only ever have issues running under dual CPU's.

Has anyone else got experience with Supermicro motherboards, dual CPU motherboards or really any advise on things to look out for.... hell, anyone just feeling super helpful and interested in me getting my 12 core overkill beast running and want to join in on the journey?

[shaithis]

Mostly with those boards, you need to put identical sets in each CPUs memory bank.

I have also found them to be VERY picky about mixing RAM, even when in separate banks.

How many RAM slots are there, whats the configuration of them and what DIMMs do you have?

[Biscuit]

Mostly with those boards, you need to put identical sets in each CPUs memory bank.

I have also found them to be VERY picky about mixing RAM, even when in separate banks.

How many RAM slots are there, whats the configuration of them and what DIMMs do you have?

This is the motherboard:
http://www.supermicro.com/products/m...0/X9DBL-iF.cfm

6 RAM slots (3 per CPU) and all of the Dimms I have are the same 4GB Samsung ECC Registered stuff (M392B5170EM1-CH9).

Unfortunately, they are not in pairs anymore as I have got them all jumbled up, is it important to keep the pairs matched? I figured as each CPU is tri channel, there is one channel per dimm anyway so it was irrelevant.

[DDY]

Have you tried forcing the RAM to run at lower frequencies, say at 1066MHz or even 800MHz?

[Biscuit]

Have you tried forcing the RAM to run at lower frequencies, say at 1066MHz or even 800MHz?

Not yet, but I have discovered that the system is infact not even stable with a single CPU.

Memtest runs ok, but I thought I would try and run more substancial tests on it and the only way I know how to do that is with windows. The windows installer just hangs and throws errors back at me. Going to try Ubuntu now just to see if I can get any kind of stability out of the thing.

Earlier on in the life cycle of this build I had stuff running and was transferring all my files but I had a few crashes and I noticed the PCH was mega hot. I'm starting to think that perhaps this overheating has killed it and the motherboard is now a dud :'(

[Biscuit]

With one stick of RAM and one CPU in there I managed to get Ubuntu installed and running ok. Even ran some stress tests just to see what would happen although Im not entirely convinced they are doing the same thing as something like prime...

I just started experimenting with the IMPI thing though and im wondering if this can be used to help me troubleshoot when something is wrong.

[DDY]

I'm not familiar with Supermicro's IPMI, but I'd imagine there would be a system event log somewhere, maybe that'll offer some clues.

[Biscuit]

Well I stuck a second stick of RAM in and once again ubuntu was OK but unRAID just collapsed



Loads of memory error messages there :/

[DDY]

Does the error log show any memory errors at DIMM1A?

What happens when;
a) You swap the two sticks around?
b) A third stick of RAM is installed?

[Biscuit]

I have took the second stick out and booted back into unRAID. The thing that seemed to trigger the critical error was running the Emby media server and scanning my library. It would always give up at about 62%.

I have cleared that container from the docker and re-added it, ran through the library update and it was ok.

I then added the second stick, got to 94% and boom... same problem again.

I swapped the sticks around and now it ran through without issues

[Biscuit]

OK I deleted and re-added the folders to force a thorough rescann and I got the error again. This time IPMI just went straight to CATERR

[DDY]

It is at this stage where running out of ideas I often remember the first rule of troubleshooting;

Try the easiest things first.

So, errm. Have you updated the BIOS?

[Biscuit]

It is at this stage where running out of ideas I often remember the first rule of troubleshooting;

Try the easiest things first.

So, errm. Have you updated the BIOS?

I actually have but I forgot to post it in the thread as it had no positive impact

That was learning experience in itself. To do it you have to either create a bootable dos image and run an application through that OR.....

Pay Supermicro for a product key to unlock updating via IMPI. I mean, seriously SMC?!?! Why would you make simple BIOS updating a pay option?!

There are no other options so I went the DOS route and used freedos.

I have swapped the RAM out for the two sticks of crucial non-registered stuff out of my microserver and its been running all night though.

Another thing I didnt mention is that I have pulled the whole thing apart. Its now in test bench bench style propped up on cardboard on top of the case to allow me to access all the slots and jumpers easier, the case isn't the easiest to work with.
What that has also enabled me to do is swap the CPUs round so I now know that this is either a motherboard issue or a RAM issue as the problem doesnt seem to follow one of them. I'm starting to veer towards a simple RAM incompatibility, or perhaps some incorrect 'auto' settings in the BIOS that are causing the RAM to be instable, as this non registered stuff is working OK.

Im adding a huge chunk of files to the Emby library again now to be doubly sure. I guess the operation on the server which is causing the CATERR is probably lots of small IOPS as it pulls loads of images and test data from websites.

Would this be a common thing to cause memory errors and can anyone recomend a more reliable method of pushing the RAM in this way to ensure I'm stable?

Next step is adding the second CPU back in and seeing if I can get that to run with Ubuntu before switching back to unRAID

[Biscuit]

Well after being reasonably satisfied that single CPU with 2 chips of RAM was stable, I have booted Memtest86 v6 and it crashed... strangely, there isnt any errors in IMPI event log.

Memtest runs fine on a single CPU, but the minute I set it to run dual CPUs I run into issues. Nothing reported in IMPI still

[Biscuit]

After a bit of reading it seems there is some issues with memtest in multicore mode with certain setups. Seeing as im using an unusual configuration I guess I probably fall into that group.

Decided to do a windows installation and run some tests im familiar with in there to test the stability of the system.

Ran OCCT and at first I hit a few issues with incorrect sensor readings (probably nothing to be concerned about?) but then once I disabled monitoring of them it ran for a couple of hours without problem.
Honestly though, im not hugely comfortable with this tool so I've gone for a P95 run.

If you are wondering what that's like with 24 logical cores and the very limited GPU power on a server motherboard




helpful

Hopefully all will still be good in the morning.

[Bluecube]

Fingers crossed. Can't help I'm afraid but I'm following with interest.