Cluster Design

From Cluster Documentation Project

Clusters are all about design. Indeed, clusters are about designing machines around problems. When designing a cluster the following basic questions may be helpful. If you need help with answers, try asking on the Beowulf Mailing List or submit a question to the Cluster Questions section.

Consider your budget

• Do you need help with software and support? If so, reserve some funds for this part of the project. While clusters can be easy to set up initially, continued operation can be time consuming.
• Do you plan on assembling the cluster yourself? If yes, at what level? Starting from motherboards, cases, memory, processors, etc. can be very time consuming. Buy quality parts.
• If you plan on using a vendor, provide them with a detailed specification and don't forget to run benchmarks.

How many and what type of processors do I need?

• This depends on your application. How well does you application scale. Have you tested it on a real cluster? If it is a common application, have other run it on a cluster?
• It also depends on how you want to arrange your processors. The introduction dual-core CPUs make this a more interesting question.

What Type of Interconnect?

• Again, this depends on your application. If you can use Gigabit Ethernet, then you can probably buy more nodes. If you need a faster interconnect, then you may have to sacrifice nodes for interconnects to stay in your budget. Benchmarking your code(s) is important.
• Remember Gigabit Ethernet performance will vary by chip-set, some are better than others.

What type of software?

• This depends on your needs. There are several preconfigured distributions, but which one is best depends upon your needs.

Do I need disk drives on my nodes?

• This decision also depends on on your needs. In some cases nodes can be run without hard drives, in others local swap or scratch space is needed.

What type of storage?

• do you need a parallel file system?
• Will a basic NFS setup be enough? Will it scale to the nodes you need?
• How do I construct a diskless cluster?
  • Can I circumvent the local swap dilemma?

There are many other questions that need to be answered, but these should give you a start. The book below (freely available) is also a good place to start.

Books

• Engineering a Beowulf-Style Compute Cluster by Robert G. Brown - A freely available book on building/designing Beowulf style clusters.

• See the Cluster Monkey Cluster Books Page

Tools

• The Cluster Design Rules - A tool to help design a cluster tuned for an application