Creating Corwin

Building a 64-bit Dual-Opteron Linux Workstation

by Scott Robert Ladd
3 April 2004
 

As I write this, I've just finished building a custom dual Opteron workstation. This is not a gaming computer, or an office system — it is a software development workstation, with support for various programming, scientific, and mathematical packages. This article explains the choices I made and describes the process of getting the system up and running.

From Here to There

While I have my passions, my goal is to be as agnostic and pragmatic as possible when it comes to issues of hardware and software. My last major hardware acquisition was Tycho, a 2.8GHz Pentium 4 system. Delivered at the beginning of 2003, Tycho is the system I use the most, for everything from web work (this page!) and programming to graphics and writing. As such it has performed yoeman's service, it will continue to be a useful machine for some time to come.

In terms of my work, Tycho has one major limitation — it is a uniprocessor computer. Even with Intel's HyperThreading (HT) enabled, the machine is limited as a development tool for parallel programming. Using HT improves system performance by around 15% on computationally-intensive applications; while HT appears to be two processors on an operating system level, it is not as effective as having two physical processors working in parallel.

I do have a dual processor workstation, Copernicus, a venerable IBM IntelliStation M-Pro with a pair of 600MHz Pentium 3 processors. I bought the system on eBay in 2001, upgrading the memory and processor to make it a viable system. While it has served me well, Copernicus is too slow for many tasks.

Those tasks include writing parallel code, and supporting my customer's increasing interest in 64-bit software. Intel's solutions are either as-yet unavailable (64-bit enabled Prescott Xeons and Pentiums) or prohibitively expensive (Itanium). I would very much like to try a dual PowerPC G5 from Apple, but none of my customers use Macintosh systems. So the choice became clearly into focus: I needed an dual Opteron system.

Hardware Choices

My first consideration was fitting the new system into a budget of $2000. With a wife, kids, and several animals to feed, I don't have an infinite amount of money to spend; any computers I buy must justify their cost in business terms. As such, I purchased the following componets (plus a case and other peripheral items):

Corwin (Homemade)
Gentoo AMD64 2004.0 GNU/Linux, Kernel 2.6 SMP (custom)
Dual Opteron 240s, at 1.4GHz
Tyan K8W 2885 motherboard
120GB Maxtor 7200 RPM ATA-133 HD
2GB PC2700 DRAM (1GB per processor)
Radeon 9200 Pro, 128MB, NEC FE990

Why the slowest Opterons? It came down to a matter of money &mdash less than $500 for the 1.4 GHz 240s as opposed to more than $1500 for 2.0GHz 246s. My plan was to buy a system that I could expand as funds became available. I can use the Opteron 240s in new single-processor machines for my daughters and wife when money allows me to upgrade Corwin.

My tasks are compute-intensive, so there was some trepidation on my part at buying 1.4GHz processors, when my existing Pentium system runs at 2.8GHz. As you'll see, that concern was unfounded.

While I wanted Serial ATA disk drives, those are something I can add later.

The ability to install per-CPU memory was one reason I chose the Tyan motherboard. I want to experiment with tuning programs for NUMA (Non-Uniform Memory Architecture). As of this writing, Intel has been vague about whether or not their 64-bit "enhanced" processors will include memory controllers. SMP systems route all CPU memory access through a single bus; NUMA machines scale better, in that each CPU has it's own "local" memory that is accessed directly. Multiple Opterons only share a memory bus when accessing memory that belongs to another CPU; if you want the best performance from a multi-Opteron system, you need a motherboard like the K8W.

The new system arrived, and there were the usual small annoyances and problems that crop up when building any system. The drive bays, for example, block installation of one of the AMD-supplied processor heatsinks; I ended up buying a low-profile cooler that fit in the available space.

Of note is my error in ordering an ATX power supply when the Tyan 2885 requires an EPS12V. Asking for an EPS12V at the local CompUSA drew blank stares. This part of Florida isn;t exactly a center of high technology, so I was forced to order the proper PS online, delaying my joy for a day.

I made another small mistake when ordering the memory — I bought PC2700 based on the original K8W specifications, and the latest BIOS upgrade supports PC3200. This will make a small difference in system speed, but not very much.

Oh, and I did buy some very pretty blue lights for the case, since it had came with window; my total cost for "tricking out" Corwin came to under $40. I'm rather sick of beige, gray, and black, so a metallic blue computer with lights is a pleasant change. What I didn't realize was that I was purchasing a very expensive bug-zapper — I've watched several flies and moths (this is Florida, land of insects) climb into the acryllic case fan, from which they are expelled in pieces. Brrrzzzppp...!

Those complications aside, the system worked as expected when I turned it on: It POSTed and let me into the BIOS to change a few minor settings.

Now it was time to install Linux.

Choosing and Installing a Linux

Both Tycho and Copernicus run Debian 'sid' (unstable) Linux; I've used Debian for a long time, and find the apt packaging system quite comfortable. Other machines in my office run Solaris, Windows 2000, and Windows XP. For the most part, I use Windows systems for compatibility testing, user interface development, and gaming. For hardcore development work, however, Linux is my preference. And in the case of the 64-bit Opteron, Microsoft's Windows for AMD64 is still in beta testing, with an expected release date sometime in late 2004.

So which Linux? Debian is still working on their "pure64" implementation; Red Hat Fedora is in beta testing; SuSE was an option, but, in early March, it looked to be a bit dated in its offerings. And so I settled on what appeared to be the most complete Linux for the new system: Gentoo's AMD64 port, which hit its first official release at the end of February.

Using Tycho, I downloaded the LiveCD for gentoo-amd64 and burned it to disk. Previously, I'd only installed Linux from binary packages (Debian and Red Hat) before. But then again, I remember (vaguely) starting up an Altair 8800 by entering the bootstrap loader through its front panel switches — how hard could installing from source be? Yes, I'm dating myself... ;)

Gentoo's web site has step-by-step instructions. Installation isn't a simple matter of booting from a CD and clicking on "OK" dozens of times; getting Gentoo working requires a both consideration and effort. Much to my own surprise, everything went rather well; the compiling was somewhat tedious at times, but I could have avoided some of it by installing ("emerging") binary packages. And I build my own kernels for the Debian boxes, so configuring Linux itself was easy enough.

It took twelve hours to get Gentoo installed such that I could boot into Gnome. During that time, I was logged into the #gentoo-amd64 IRC channel, via irc.freenode.net, receiving help from some very kind people (Thanks guys!). I've found that the best tool for installing a computer is another computer -- while Corwin was being configured, I was searching the web and chatting on IRC through Tycho. I don't think the installation would have gone as well if I hadn't had a place to ask questions (and get answers!)

Software Packages

I've been very satisfied with apt-get and dselect for package management on Debian systems. Now that I'm familiar with Gentoo's Portage system, I find it equally pleasant to use. Portage is based around to concept of ebuilds which describe how a package is built from its original source. Whereas most apt packages have Debian-supplied patches, Portage uses original source, sometimes downloading source tarballs directly from the originator's web site. A simple emerge package, and the source code is retrieved, built, and installed. After installation, the package's source code is retained in usr/portage/distfiles, keeping a record of exactly what you've installed.

One option I would like is the ability to run the built-in self-checks and santity tests that come with many packages. This is especially important for scientific and numerical libraries (e.g. LAPACK, Atlas), where correctness of generated code is essential. An emerge --check package or emerge --test package option could provide this capability.

While gentoo-amd64 is an official release, it is very much a core release, with many packages "masked" such that they will not install with a simple emerge command. In several instances (e.g., Abiword), I installed masked packages via commands of the form ACCEPT_KEYWORDS="~amd64" emerge package. Some packages, such as OpenOffice, are only available in 32-bit form. I like AbiWord and Gnumeric, so the lack of OpenOffice is a minor point for my work. If I need to work on Word, PowerPoint, or Excel documents, I use a Windows-based computer running MS Office.

One glitch remains: The Radeon video card is not behaving as I'd expected. I have a similar card in the Pentium 4 system, and it works flawlessly. I don't use my Linux boxes for gaming, so I just need a reliable card that is supported by XFree86. Under 32-bit Linux, I quite happily use the "radeon" driver provided by XFree86 4.3.0; binary drivers can be obtained from ATI, but I prefer to avoid closed-source software on my Linux machines.

Under gentoo-amd64, I've been less successful using the same driver and version of XFree86. When I close a window, I can see it being erased, from top-to-bottom; when I drag a window, the display is afflicted with white, horizontal "lightning." Glxgears runs at about 1400 fps at default resolution, indicating that direct-rendering works. I suspect that I need to tune my XF86Config, or perhaps change a kernel setting. The problem, for the moment, is a minor annoyance, and if I solve the problem, I'll update this article.

Impressions and Benchmarks

I've been running Corwin for two weeks, and couldn't be more pleased. The system has been rock-stable, fast, and pleasant to use.

Comparing Corwin to Tycho is a matter of apples and oranges -- different Linux distributions, memory architectures, cache sizes, and processor designs all make a direct comparison problematic. I can say this much: When running serial programs (i.e., using only one of Corwin's processors), its 1.4GHz Opterons perform as well or better than the Pentium 4 running at twice the GHz. On a few tests (certain matrix operations, for example) the Pentium 4 is much faster.

I'm in the midst of creating a new benchmark suite, in conjunction with my Acovea project. Here's a taste of the results I'm seeing from a pre-release version of the benchmark suite:

Corwin (serial code running on dual Opteron 240s at 1.4GHz)

     alma time:   25.7
      evo time:   30.9
      fft time:   31.1
     huff time:   20.7
      lin time:   23.5
     mat1 time:   19.3
     tree time:   24.8
--------------
total run time:  176.0

Tycho (serial code running on HT-enabled Pentium 4 at 2.8GHz)

     alma time:   26.2
      evo time:   51.8
      fft time:   34.8
     huff time:   10.8
      lin time:   24.5
     mat1 time:    5.6
     tree time:   17.2
--------------
total run time:  170.9

The benchmark code can be downloaded here. The tarball contains the scripts I used in generating the executable images fopr the timings above; the programs contain comments describing their purpose. This is a preliminary version of the benchmarks, and should not be taken as the final word on anything.

As always, I look forward to considered comments.

-- Scott