Clustering commodity PC hardware - A web log

Hi all,

I recently bought myself a small cluster of x86 machines for rebuilding into a fully operating and useful system, for a personal project, and David and I agree that it's primetime material for a thread at HEXUS, so I'm going to republish my cluster 'blog here to spark some discussion about clustering. So that includes the easiest way to do it, the interconnect to choose, what hardware can be used, why you'd do it, how much it costs to run, software you'd run on one and anything else related to its construction.

So if you're interested in my cluster, or building your own, post away and I'll endeavour to help. There's more than just myself on these forums that have experience building and running them, so I hope the thread attracts those folks in (you know who you are!).

What follows is the first few posts to my cluster 'blog over the last week or so, and I'll cross publish in this thread whenever I update it in the future.

Hello!

If you read my main weblog you'll know that I recently bought a small cluster. I've setup this web log to document my adventures in rebuilding it from scratch and getting it up and running. It's not a high performance cluster by today's standards, but it's competent enough to do useful work and it'll be something cool to put on my CV. I'll start from the beginning (it's still in pieces in my living room) and hopefully by the end, this 'blog will be a useful reference document for anyone looking to build their own.

Posted by Rys at January 16, 2005 12:57 PM

Hardware and initial setup work

Cluster Hardware

The cluster houses around 11GHz of compute power across eight compute nodes, using Intel Pentium III processors. Each node is a IBM Netfinity 5100 server in a 5U rack chassis and has a pair of 667MHz Slot 1 Coppermine P3s, 1GB of PC133 ECC memory, 9GB SCSI and a Myrinet 1000 card for inter-compute node communications. Onboard 100Mbit Ethernet provides connectivity to the management node. The management (head) node is another Netfinity 5100 in the desktop chassis with the same specification as the compute nodes, with an extra 40GB IDE RAID volume, sans Myrinet card.

A Cisco C2916M-XL Ethernet switch provides the TCP/IP interconnect and a Myricom M2L-SW16 is the Myrinet interconnect switch.

Initial Setup

Initial setup has consisted of reclaiming the configuration on the Cisco switch, following this document to reset the master password and let me reconfigure the VLAN1 and first eight Ethernet ports to match my existing home network setup. The next step is to check the configuration on the M2L-SW16 for obvious weirdness that would cause problems with the operating system loaded onto the compute nodes. I imagine the original switch configuration will be fine, but I'd like to check, even if it's just to make sure the switch is actually alive and working.

Software wise, Rocks looks like being my weapon of choice. There's a custom RedHat 7.2 installation on the cluster at the moment, although it's unclear how usable it is, although it'll be useful to boot just to see how much of the hardware is working (hopefully 100%). I'm planning on booting it for the first time this week, after sorting out power (it'll require 11 power points for the hardware) in my office.

Posted by Rys at January 16, 2005 05:44 PM

Myrinet switch is alive

I can successfully give the Myrinet switch an IP address using RARP, which allows it to wake up on the Ethernet management port and respond to simple ICMP pings and certain SNMP traffic. The switch listens on port 4003 for SNMP and uses the community string public. Using some freeware SNMP tools I can obtain very limited information (finding an OID list for the switch seems impossible without contacting Myricom), so it looks like the switch is alive. Actual functional testing looks like being "plug it in and see" though. There's some scope for testing doing TCP/IP over GM though, outside of operation in the cluster, if I fancy pulling a couple of cards from two nodes.

It's completely silent when running, compared to the Cisco, although it might get noisier when loaded up with traffic to process.

Posted by Rys at January 17, 2005 12:26 AM

Early cluster pictures

While Alex was on my PC surfing the web earlier, I decided to not just sit there and twiddle my thumbs, so I brought the cluster upstairs to my office. She didn't give me a hand, so I now feel like death. The individual nodes are very heavy and taking them up two flights of stairs wasn't fun. The result is a few new pics though.

Find those here.

Posted by Rys at January 18, 2005 12:14 AM

Power provision was purchased

I grabbed a couple of Belkin Surgemasters for the cluster. Power provision will still be on the "unplug some other stuff first" side of the fence for the time being, until I rejig my office, but I can start the cluster rebuild at the weekend.

Posted by Rys at January 20, 2005 09:54 PM

Management node is setup

The ethereal, mysterious 'Tim', and Ben, came round earlier to help me do the initial setup and cabling. We booted the management system and the first compute node with their original operating system installs (RedHat 7.3 on the management node, 6.2 on the compute node) to check they'd boot and no hardware was misconfigured. Then we set about following the configuration guide for setting up a front-end server with Rocks. Install went fine and we ended up calling the cluster superchode (thanks, Alex!). There looks to be some problems with Ganglia but given that no compute nodes are connected to the switch, it's understandable.

We also set the cabling up for power to the two switches, front-end and four compute nodes (and a monitor we won't use too much until we can SSH in with confidence) ready for attaching nodes to the cluster.

Tomorrow's task is to bring up as many compute nodes as possible using the basic Rocks install and then possibly experiment with adding Roll packages to the nodes for further functionality. 'Tim' will assist again I think and I'm sure Ben will pop over too. It's better than doing it solo, although the cluster's blinking lights on the Netfinity chassis' are mesmerising and give it the appearance of limited intelligence on its own, to keep me company, like some second-rate console panel from the first series of Star Trek.

More tomorrow as we bring the nodes up. There's some new pictures from today's endeavours in the image gallery. If it's got a cable attached in the picture, it's something we did earlier.

Posted by Rys at January 22, 2005 09:52 PM

Superchode and four baby chodenodes

Tim and I rebuilt superchode's front-end system this afternoon to correct a misconfiguration with the Ethernet interfaces. The box has three Ethernet adapters and the Rocks installer numbers them as it finds them on a bus probe, so the on-board AMD controller was being brought up first. We wanted to ignore that and just use the Intel 100Mbit cards instead so a quick trip into the BIOS (IBM BIOS implementations really do go the extra ten miles to give you information and system setup) and a reinstall later and superchode's head was alive again.

Then attaching the first half of the compute nodes to the cluster was fairly simple. Nodes 0, 1 and 2 attached without issue, although node 1 has a dead stick of memory that'll need replacing to give it the full 1GB. The BIOS disables the DIMM slot, so it's purring along with 768MB for the time being. Node 3 wouldn't boot from the CD drive so we swapped in a DVD drive to test and that wouldn't be booted from either. An IDE ribbon swap and everything was alive. It also has a slightly sensitive chassis intrusion switch that stops you booting it unless the chassis cover is on properly.

So an hour or so after that node had kick-started itself across the network, feeding from the front-end, we had 10 active processors, just under 5GB of memory and the cluster alive. The cluster monitoring software, Ganglia, showed us this when it was done.

The next step is to boostrap the final four nodes and register them into the cluster database. Then the Myrinet interconnect can be brought up and cluster jobs started over that. I might benchmark it before and after, just to see how the Myrinet affects I/O-bound performance.

So everything's coming along nicely. Rocks makes it relatively simple to get compute nodes added and while it'd be nice to say it took us a massive amount of time and effort to bring up this first half of the cluster, it wasn't too bad and hardware issues kicked our asses more than the software did.

Apache doesn't stay up for long at the moment however, but that's no doubt fixable. I should have the entire cluster connected (at least with Ethernet) sometime this week, time pending.

Posted by Rys at January 23, 2005 11:24 PM

So the initial stages have gone pretty well, bar the odd bit of hardware in the compute nodes not cooperating. That was all easily fixed or ignored for the time being, so things are going well. 10 processors connected is a pretty good start and while that could have been all 18 yesterday, we were running late due to the hardware problems.

It's definitely been a learning experience. The cool this is that you don't need a high-speed interconnect like Myrinet to build one, you can do it over Ethernet just as easily. Rocks has support for x86-64 and you can co-cluster that architecture with regular x86 if you're careful about the jobs you run.

There's loads more to it since all I've done is bring up the hardware so far. So there's talk on the job scheduler, process accounting, security and a myriad of others things to cover over time. I'll get there though (with your help!).

Have a look at Rocks, ClusterWorld, Grid Computing and SourceForge's clustering foundry for more information.

Two 386 PCs on a 10Mbit hub can be a cluster, so don't think it's out of your league in terms of building one for fun, enjoyment and your own education.

Enjoy!

Rys

Brilliant,

Going to be keeping a close eye on this thread.

Great project :D

my mind is now BOGGLED. I have absolutely no idea what most of that is.

Starting at base point A, I shall ask : Are these computers, individually, similar to a home PC in as much as having a CPU, some ram, a mobo, a vid card, a hdd, and a PSU?

I have no idea about clustering but this sure looks interesting :D

Zak: This should help you there :)

The only thing i'll say is that while building a cluster may be fun and a learning experience most of us have little or no use for it, and sadly i don't view UD or SETi as valuable uses of that amount of processing power.

I was lucky enough to get taught at uni by one of the experts in the country on distributed computing and still remember a stack load of that information. The real interesting thing about a cluster is the ability to use rather different setups of software.

Something like Occam to program your own capabilities on the cluster is something that is imminently doable and could be used to do some rather interesting maths problems.

Interested to hear the uses you find for it Rys :)

TiG

Quote:

---

Originally Posted by Zak33

my mind is now BOGGLED. I have absolutely no idea what most of that is.

Starting at base point A, I shall ask : Are these computers, individually, similar to a home PC in as much as having a CPU, some ram, a mobo, a vid card, a hdd, and a PSU?

---

Yeah, they're just indivdual PCs connected together. Each node is an IBM Netfinity 5100 server, which is just a server with a pair of P3s, on-board graphics, memory, mainboard, HDD and the PSU, exactly like any normal PC.

They're in rack-mount cases, rather than your usual desktop ATX fare, but that's the only difference :cool:

Software then links them together and lets you run a program across all the PCs connected to the cluster. So my collection of 8 systems in one just appears to any software as a single PC, albeit one with 16 processors and 8GB of memory.

Clustering came about as a way to migrate away from the massive shared-memory systems peddled by the likes of Sun, SGI and Cray (to name but three) and create supercomputers from basic PCs, all linked together.

Think of it as something like 8 Shuttles linked together on a basic network switch, just with some special software to make them appear as one PC.

My cluster is a bit more involved that than, but not by much.

In terms of what I'll do with it, there's a mound of research software I could put to use, including fluid dynamics simulations, protein folding, number crunching, distributed webserving and loads more. Picking a task for it is definitely the hard part!

Rys

That's also another thing I was wandering, what can you do with it? :)

*edit*
I was a bit late with the reply :P

How about using it as the webserver for hexus? :D

Some comparison stats against a standard pc for something like folding etc would be good :)

get it to run pifast :)

right...getting it now(yeah...1% ;) ) So its RAID but..er......RAIC... (if I'm thefirst person to think of that I'd like royalties pls)

IS the software that runs it self-written, or is their a defacto set of software developers who create it?

And in either case IS IT a network....CAT5 cable and stuff? Or does it slot together like interconnectabe motherboards? Or is it a kinda "blade" that slots like a daughterboard into a big motherboard?

I'm well intrigued :) sounds very cool indeed.

Trying to take me on at my own game I see Rys, geting people invovled with a project and getting them posting, ala the HEXUS living document.

Rest assured, I shall dig deep and rise to the challenge.

:D

Seriously though mate, must be enough trouble to do, let alone make posts telling everyone about it. Nice work. :)

Quote:

---

Originally Posted by Zak33

right...getting it now(yeah...1% ;) ) So its RAID but..er......RAIC... (if I'm thefirst person to think of that I'd like royalties pls)

IS the software that runs it self-written, or is their a defacto set of software developers who create it?

And in either case IS IT a network....CAT5 cable and stuff? Or does it slot together like interconnectabe motherboards? Or is it a kinda "blade" that slots like a daughterboard into a big motherboard?

I'm well intrigued :) sounds very cool indeed.

---

The software can be self written if you know what you're doing and in many cases existing software can be adapted to a clustered computing environment without much bother, too. There are companies and software teams out there that specialise creating applications just for cluster setups, but that doesn't have to be the case.

Then you can take things like video encoding, something that lends itself well to distribution in a clustered system and that's something that most people can understand. Video encoding can go lots faster on a cluster, distributing the workload to all the systems in the cluster to get your work done faster.

As far as connecting them goes, my cluster is currently only connected via regular network cable (CAT5) into a 100Mbit Ethernet switch. And the cluster will run just fine over that. It's hardware most people have already.

I also have a Myrinet interconnect for the compute nodes too, which is much (orders of magnitude!) lower latency than any Ethernet variant, allowing the nodes to talk to each other faster, and at higher bandwidth (Myrinet 1000, which I have, is 1.28Gbit/sec) than even gigabit Ethernet. So my compute nodes, when configured with the Myrinet, will allow them to talk to each other faster and at a higher rate.

RIAC is a great term to use for home clustering! Cobble together a bunch of older machines, installing an operating system on them that supports clustering and run the applications that can take advantage of it. It doesn't have to be an expensive cluster (like mine, which when first deployed in 1990 cost the wrong side of £50,000), it can just be a bunch of EPIAs or something.

The thing to remember is that in terms of hardware, there's many ways to create a cluster. Blade servers with a common backplane and chassis, separate machines connected via CAT5 cable and Ethernet, or Myrinet (or Infiniband and loads of other interconnect standards), or what have you.

Just connect machines really :devilish:

It's then the software on top that's the difficult part, which I'll cover in due course with my own cluster.

Rys

I'm just disappointed at the lack of the word "beowulf". I guess I spend far too much time on slashdot.

Keep up the good work Rys :)

Quote:

---

As far as connecting them goes, my cluster is currently only connected via regular network cable (CAT5) into a 100Mbit Ethernet switch .... I also have a Myrinet interconnect for the compute nodes too, which is much (orders of magnitude!) lower latency than any Ethernet variant, allowing the nodes to talk to each other faster, and at higher bandwidth

---

So are all they connected by ethernet now and will be using Myrinet when you have set it up? Or are just some of the connections by Myrinet, and the rest ethernet?

Quote:

---

Originally Posted by Mblaster

So are all they connected by ethernet now and will be using Myrinet when you have set it up? Or are just some of the connections by Myrinet, and the rest ethernet?

---

You were right the first time :) So...

They're all connected via Ethernet atm, which will be a permanent fixture since the front-end has no Myrinet and has to issue jobs and the like over Ethernet. However the compute nodes have Myrinet, which when configured will be the interface they use to talk to each other when jobs are running. The Myrinet is idle just now though, unconfigured so far.

So just now, if I send out a job, it goes out over Ethernet, is computed using Ethernet as the transport, and comes back to the front-end using Ethernet.

In the future it'll be out over Ethernet, compute over Myrinet, back over Ethernet when the job is done.

Rys

The final four chodenodes are built

The final four chodenodes, configured in Rocks as a separate rack cabinet since they're in two piles of four, got built tonight. There was an issue with the front-end dropping the connection that the chodenodes pulled their install images from, but I got there in the end.

I'll boot the full cluster with all nodes attached at lunchtime tomorrow (today, since it's 3.26am), so expect another update soon after with an obligatory Ganglia screenshot to celebrate bringing all CPUs online.

What I haven't done so far is detail what software is being used, how it was setup on the front-end and how you bootstrap the compute nodes over the network using the front-end, so I'll do a series of posts sometime this week detailing that. It'll be much what you'll find in the Rocks base install guide, with insight to match my particular cluster setup.

So that's milestone number one reached: the front-end machine and all the compute nodes are setup and can talk to each other correctly over Ethernet.

Milestone two will be to have them running simultaneously, with milestone three being able successfully run jobs over configured Myrinet.

Posted by Rys at January 25, 2005 03:24 AM

Nice work Rys! I'm good friends with The Tim and have experience with Win2k clusters but very little with Linux - I'll be hawking this thread closely :D

Milestone two reached; all 18 CPUs connected

I switched the front-end on during lunch to have a quick look at the Myrinet configuration (the kernel isn't loading the Myrinet module yet) before turning on the two banks of nodes. Everything came up just fine and the end result brought a smile to my face. All eighteeen processors (sixteen for the compute nodes and two for the front-end) can be seen and the graphs at the top of that Ganglia display show you just how I booted it.

If you look at the memory graph which I've overlayed with a few labels, you can see me bring up the front-end which registers itself and says hello to Ganglia, then I bring up the first 'cabinet' of compute nodes a short while after, when I was done playing with the Myrinet config, then I bring up the second cabinet of nodes after the first four are showing in Ganglia.

That information is shown properly in the load graph too, I just used the memory size graph since it had less metrics to look at. So in that graph, CPU count starts at 2, rises to 10, then to 18 (red metric). You can see the same trend in the node count metric (green). Notice how the process count spikes as each node is switched on, as they talk to the front-end to say hello and join the cluster.

Myrinet configuration is next, sometime this week or at the weekend.

Posted by Rys at January 25, 2005 02:09 PM

99% of the way to milestone three, Myrinet is up (just)

Tim and I successfully brought up the Myrinet interconnect on superchode tonight. Rocks gets you most of the way there on a default 3.3.0 install, but not quite. Instead of quickly fixing the default GM (Myricom's message passing layer for Myrinet) install supplied with Rocks, I decided to self-upgrade to 2.0.17 instead, which is the current release.

I did the initial testing and bootstrap of the GM mapper - software which controls the Myrinet routing amongst other things - on chodenode-0-0 (the first one in cabinet 0) which went fine, so Tim and I deployed it out onto a cabinet each, using NFS stores for common data. I'll document that process in due course. chodenode-1-0 has a slight cabling issue (masses of CRC errors logged by the hardware and no route to the mapper running on 0-1) which seems to be fixed now, but I'll be investigating getting some spare cables, just incase.

It all looks good now anyway. In terms of milestones, we didn't quite get round to pushing a job out over the cluster when the Myrinet was up, due to the problems with 1-0 and the time it took (it's now nearly 2am and Alex will kill me when I get to bed), so milestone three isn't officially met, but I'm sure we'll get there tomorrow when I bring it back up either at lunchtime if I'm not too busy (I probably will be) or at night.

In terms of performance, GM's benchmarking tools showed very low latency (data being moved in periods measured in the low numberrs of microseconds) across seven hosts earlier, and an excess of 2Gbit/sec of bandwidth both reading and writing, per node. Inter-node bandwidth with all nodes should therefore be in the region of 16Gbit/sec.

Posted by Rys at January 26, 2005 01:45 AM

Rebuild success; milestone three reached

Tim and I spent the day rebuilding superchode to add some new Rolls to the cluster (Rolls are bundles of files and configuration data used to add functionality to a Rocks cluster). It's now in much better shape to be used for useful work, with clustered Java, C, C++ and Fortran implementations available for coding with, and their requisite libs for precompiled stuff.

Sun's Grid Engine is the default scheduler now too. You submit jobs to the SGE queue and it sends them out to the required nodes for processing. There's a bunch of queue tools for monitoring, and there's a web queue monitor if we're feeling lazy and can't be bothered SSH'ing in to the front-end machine.

The rebuild was made painful by an automount issue on the front-end (automount is the daemon that manages NFS shares for the home directories of users on the cluster) and the requirement for more swap space than we'd originally allocated, meaning that the first attempted rebuild fell over during the configuration phase.

However it's all up and running again, Myrinet is up and jobs can be scheduled over GM on the compute nodes. In short, the cluster is fully operational and better equipped to do some real work.

Milestone three was reached with a successful run of Linpack over Myrinet and two compute nodes, just before I shut it down for the night.

Milestone four will be the completion of useful work using the cluster. More on that after I rope Tom into things.

Posted by Rys at February 7, 2005 01:16 AM

pity ur selling it

is there like a simple way to get that sort of power shared? Like a rendering farm

Quote:

---

Originally Posted by Matt1eD

is there like a simple way to get that sort of power shared? Like a rendering farm

---

what do you mean by "shared", and what's your understanding of the words "rendering farm"?

Quote:

---

Originally Posted by directhex

what do you mean by "shared", and what's your understanding of the words "rendering farm"?

---

shared - like my computer connected to loads of others over some sort of network switch do do more work for me.

rendering farm - a computer cluser thing that like has everything split up (like images of a film) between the processors of the farm's other computers. No doubt am wrong but a big version of what a dual processor mobo does

well, that's what confused me - a cluster is designed for precisely that!

you connect to a "front end" or "master" node, and submit a "job" (usually a script detailing work to be done) using a scheduler - Sun GridEngine is a common scheduler, as is PBSPro or Torque. The scheduler then sends your job script to X number of machines in a cluster (as specified when you ran the submission), and lets the job run to either competion, or until a time limit has been reached.

What you need though are programs designed to be run on a cluster - more specifically, they need to use MPI (Message Passing Interface) to send messages to each other. In this case, SuperChode has a dedicated Myrinet (low latency) network interconnect to do message passing, as well as regular ethernet for file transfer et al. Using a myrinet-capable application (or compiling one on the master node) means you can pslit the job up at will

Alternatively, you could submit sixteen 1-cpu jobs with their own submissions scripts and no need to interoperate. the scheduler would take care of that.

Thanks, that seems quite complicated (And clever)

I don't use 3dsmax or whatever it is but quite a few users 'split' it for that, are the scripts to 'split' it built in.

So say how could you run folding@home or similar across loads of say 233MHz comps?

something like BOINC you wouldn't schedule, you'd just run it - at worst, you'd use a parallel shell (gives a linux shell, but commands are run on several computers instead of just 1)

as for how 3DS does split jobs, i have no idea. I just work with stuff like GROMACS

Cheers for the explanation. Is there like a (no doubt expensive) book or something about this - it sounds really interesting.

Yes, this is all very interesting, i was going to read up and try this some months back. However i dont like linux much at the minute, is there software in place to cluster with windows? Or is it far simpler to learn to love linux?

Quote:

---

Originally Posted by Scarlet Infidel

Yes, this is all very interesting, i was going to read up and try this some months back. However i dont like linux much at the minute, is there software in place to cluster with windows? Or is it far simpler to learn to love linux?

---

nobody -SERIOUS- runs Windows on a cluster.

for one thing, you want CPU power going onto computation, not opengl screensavers (and there's no Windows for Supercomputers yet)

that said, there -ARE- drivers for using Myrinet kit under windows. i seem to recall 3 of the top 500 supercomputers in the world run it.

oh, and don't forget you'd need a windows license for every compute node

that said, there are some efforts in the GRID community to Cycle Scavenge (like BOINC), and that's particularly Windows-friendly. it's not proper parallel computing, but it IS Windows-friendly

actually, i lie, i think the university of southampton runs a windows cluster - they advertise the fact heavily, suggesting that MS paid for it & told them "tell everyone how great windows is for supercomputing"

Im sure they came up with some sort of modified version of ms server or whatever they call it. Ive never found linux to be fast anyway (flame away if you must) tho thats probably my lack of experience. And to be honest who owns seperate copies of windows for every pc, however your legal point is taken.

would it be at all possible to use windows as my workstation with a linux server and lunx nodes, then i wouldnt have to deal with it much. I think i need to do some serious reading and research. My knowledge of networks and processing is limited at best, theres only one way to learn.

Quote:

---

Originally Posted by Scarlet Infidel

Im sure they came up with some sort of modified version of ms server or whatever they call it.

---

unreleased as of yet (Microsoft Windows Server 2003 Compute Cluster x64 Edition)

Quote:

---

Ive never found linux to be fast anyway (flame away if you must) tho thats probably my lack of experience.

---

yep, it is. also note the availability of software - almost all HPC applications come for Linux, Solaris or AIX. a few for MacOSX (since it's UNIXy). almost none for Windows.

Quote:

---

And to be honest who owns seperate copies of windows for every pc, however your legal point is taken.

---

running a 256 node cluster without licenses is a fast-track to seven figure fines at best

Quote:

---

would it be at all possible to use windows as my workstation with a linux server and lunx nodes, then i wouldnt have to deal with it much.

---

that's the idea. most people contact a head node with PuTTY from Windows machines.

Quote:

---

I think i need to do some serious reading and research. My knowledge of networks and processing is limited at best, theres only one way to learn.

---

http://bofh.be/clusterknoppix/

or buy Rys' cluster, which is the real deal (i.e. has real HPC interconnect)

Cheers directhex, this has been great help and interesting. Want to read loads more now!

Thanks for the help, very useful. Id just like to make clear im only interested in trying it for a bit of fun and gaining experience with maybe 4 or 5 very basic nodes. Nothing big and expensive.

clusterknoppix is probably appropriate for that level of interest, though an old *real* cluster is much closer to the *real* thing

Oh cluster.

I'll miss you so. :(