Supermicro Dual CPU motherboard
Hi everyone!
So I recently decided to upgrade my file server to enable me to offload a lot of virtual machine tasks to it. Instead of spending fortunes on new and modern server hardware or run the 'risk' of using regular consumer hardware, I decided to do a bit of ebay-ing.
I ended up with a LGA1356 motherboard, 2 x E5-2428L CPUs (60w 6C/12T), 24GB ECC registered Samsung RAM all for under £300.... happy days.
The problem I seem to be having is getting everything working. I was hoping (as always) for plug and play, but whenever I run dual CPUs I hit issues with memory errors. I have checked the RAM, tested with RAM out of my old server and even some none-ecc stuff and I only ever have issues running under dual CPU's.
Has anyone else got experience with Supermicro motherboards, dual CPU motherboards or really any advise on things to look out for.... hell, anyone just feeling super helpful and interested in me getting my 12 core overkill beast running and want to join in on the journey?
Re: Supermicro Dual CPU motherboard
Mostly with those boards, you need to put identical sets in each CPUs memory bank.
I have also found them to be VERY picky about mixing RAM, even when in separate banks.
How many RAM slots are there, whats the configuration of them and what DIMMs do you have?
Re: Supermicro Dual CPU motherboard
Quote:
Originally Posted by
shaithis
Mostly with those boards, you need to put identical sets in each CPUs memory bank.
I have also found them to be VERY picky about mixing RAM, even when in separate banks.
How many RAM slots are there, whats the configuration of them and what DIMMs do you have?
This is the motherboard:
http://www.supermicro.com/products/m...0/X9DBL-iF.cfm
6 RAM slots (3 per CPU) and all of the Dimms I have are the same 4GB Samsung ECC Registered stuff (M392B5170EM1-CH9).
Unfortunately, they are not in pairs anymore as I have got them all jumbled up, is it important to keep the pairs matched? I figured as each CPU is tri channel, there is one channel per dimm anyway so it was irrelevant.
Re: Supermicro Dual CPU motherboard
Have you tried forcing the RAM to run at lower frequencies, say at 1066MHz or even 800MHz?
Re: Supermicro Dual CPU motherboard
Quote:
Originally Posted by
DDY
Have you tried forcing the RAM to run at lower frequencies, say at 1066MHz or even 800MHz?
Not yet, but I have discovered that the system is infact not even stable with a single CPU.
Memtest runs ok, but I thought I would try and run more substancial tests on it and the only way I know how to do that is with windows. The windows installer just hangs and throws errors back at me. Going to try Ubuntu now just to see if I can get any kind of stability out of the thing.
Earlier on in the life cycle of this build I had stuff running and was transferring all my files but I had a few crashes and I noticed the PCH was mega hot. I'm starting to think that perhaps this overheating has killed it and the motherboard is now a dud :'(
Re: Supermicro Dual CPU motherboard
With one stick of RAM and one CPU in there I managed to get Ubuntu installed and running ok. Even ran some stress tests just to see what would happen although Im not entirely convinced they are doing the same thing as something like prime...
I just started experimenting with the IMPI thing though and im wondering if this can be used to help me troubleshoot when something is wrong.
Re: Supermicro Dual CPU motherboard
I'm not familiar with Supermicro's IPMI, but I'd imagine there would be a system event log somewhere, maybe that'll offer some clues.
Re: Supermicro Dual CPU motherboard
Well I stuck a second stick of RAM in and once again ubuntu was OK but unRAID just collapsed
http://s32.postimg.org/3otmf2tp1/Capture.png
Loads of memory error messages there :/
Re: Supermicro Dual CPU motherboard
Does the error log show any memory errors at DIMM1A?
What happens when;
a) You swap the two sticks around?
b) A third stick of RAM is installed?
Re: Supermicro Dual CPU motherboard
I have took the second stick out and booted back into unRAID. The thing that seemed to trigger the critical error was running the Emby media server and scanning my library. It would always give up at about 62%.
I have cleared that container from the docker and re-added it, ran through the library update and it was ok.
I then added the second stick, got to 94% and boom... same problem again.
I swapped the sticks around and now it ran through without issues :wallbash:
Re: Supermicro Dual CPU motherboard
OK I deleted and re-added the folders to force a thorough rescann and I got the error again. This time IPMI just went straight to CATERR
Re: Supermicro Dual CPU motherboard
It is at this stage where running out of ideas I often remember the first rule of troubleshooting;
Try the easiest things first.
So, errm. Have you updated the BIOS? :D
Re: Supermicro Dual CPU motherboard
Quote:
Originally Posted by
DDY
It is at this stage where running out of ideas I often remember the first rule of troubleshooting;
Try the easiest things first.
So, errm. Have you updated the BIOS? :D
I actually have but I forgot to post it in the thread as it had no positive impact :(
That was learning experience in itself. To do it you have to either create a bootable dos image and run an application through that OR.....
Pay Supermicro for a product key to unlock updating via IMPI. I mean, seriously SMC?!?! Why would you make simple BIOS updating a pay option?!
There are no other options so I went the DOS route and used freedos.
I have swapped the RAM out for the two sticks of crucial non-registered stuff out of my microserver and its been running all night though.
Another thing I didnt mention is that I have pulled the whole thing apart. Its now in test bench bench style propped up on cardboard on top of the case to allow me to access all the slots and jumpers easier, the case isn't the easiest to work with.
What that has also enabled me to do is swap the CPUs round so I now know that this is either a motherboard issue or a RAM issue as the problem doesnt seem to follow one of them. I'm starting to veer towards a simple RAM incompatibility, or perhaps some incorrect 'auto' settings in the BIOS that are causing the RAM to be instable, as this non registered stuff is working OK.
Im adding a huge chunk of files to the Emby library again now to be doubly sure. I guess the operation on the server which is causing the CATERR is probably lots of small IOPS as it pulls loads of images and test data from websites.
Would this be a common thing to cause memory errors and can anyone recomend a more reliable method of pushing the RAM in this way to ensure I'm stable?
Next step is adding the second CPU back in and seeing if I can get that to run with Ubuntu before switching back to unRAID
Re: Supermicro Dual CPU motherboard
Well after being reasonably satisfied that single CPU with 2 chips of RAM was stable, I have booted Memtest86 v6 and it crashed... strangely, there isnt any errors in IMPI event log.
Memtest runs fine on a single CPU, but the minute I set it to run dual CPUs I run into issues. Nothing reported in IMPI still :(
Re: Supermicro Dual CPU motherboard
After a bit of reading it seems there is some issues with memtest in multicore mode with certain setups. Seeing as im using an unusual configuration I guess I probably fall into that group.
Decided to do a windows installation and run some tests im familiar with in there to test the stability of the system.
Ran OCCT and at first I hit a few issues with incorrect sensor readings (probably nothing to be concerned about?) but then once I disabled monitoring of them it ran for a couple of hours without problem.
Honestly though, im not hugely comfortable with this tool so I've gone for a P95 run.
If you are wondering what that's like with 24 logical cores and the very limited GPU power on a server motherboard
http://s32.postimg.org/5h77cph05/Prime_lol.png
:lol: helpful
Hopefully all will still be good in the morning.
Re: Supermicro Dual CPU motherboard
Fingers crossed. Can't help I'm afraid but I'm following with interest.