Building one, …nah… six, enterprise servers/workstations on about $1.5k

I have started building a relatively sophisticated new server. The objective was to have an extremely powerful, reliable, and expandable machine that could handle a broad cross section of tasks. All this without breaking the bank.

I first split up the tasks into a few broad categories. Included but not limited to:

• Personal file storage (media, photos, isos, school and other historical files, etc) and sharing (nfs, samba, potentially streaming audio/video)
• Data storage (scraped news in database, time series flat files, time series data files) and data processing (cpu intensive easily parallelized computations through compiled and interpreted code)
• Development (programming + ide, svn, web server and testing environments) and deployment
• Online backups (mozy — doesn’t support linux)
• Personal computing (web browsing, Photoshop, video editing, audio/music editing and sequencing), although not much gaming
• Testing random crap that I find online without screwing up the previous tasks via spyware, viruses, and general bloatware

Computers fail easily. They’re hard to upgrade, and it takes an enormous time to reconfigure a machine. This is still the case even if you migrate files over seamlessly, or maintain drive images. Because I didn’t want to deal with those kinds of issues, I opted to set up my server not only on new hardware, but do so entirely through virtual machines. The setup is roughly as follows:

• Application/Persistent Server VM: http, svn, and other low CPU apps that need to be on 24/7. However, all files are NFSed from the:
• File Server VM: Massive file server that has multiple virtual hard drives for various file types (arranged by category and access pattern; e.g. frequently used data files is one type, low use and bandwidth media files is another, svn repository is another).
• Computation Server VM: Empty shell of a server to ssh into and run jobs.
• Primary Linux Workstation VM: With the IDEs and apps I need like Eclipse
• Primary Windows Workstation VM: Office, Photoshop, Mozy backups, etc
• Scratch/Test Workstation VM: Well secured WinXP VM

I’m still in the process of setting all those machines up on my old desktop with VMware 6.  There are a couple roadblocks I ran into:

1. 64bit machines don’t always run nicely in a VMware, and require you to tweak with the processor Execution Mode. Often it helps to set the mode to binary translation and disable acceleration. I probably wont have a problem when on the actual server computer.
2. SVN hates the repository sitting on an NFS share. The solution, found after an hour of banging around, is to add “nolock” as an option for the nfs mount in /etc/fstab . Also, it’s important that usernames and ids are equivalent over the machines (e.g. the subversion user and group). Other than that, the guide at https://help.ubuntu.com/community/Subversion works nicely.

The next step is to deploy it on the hardware, which is arriving in the next week or so. The setup is as follows:

• ASUS Z8NA-D6C Dual LGA 1366 Intel 5500 ATX Dual Intel Xeon 5500 Series Server/Workstation Motherboard
• Intel Xeon E5504 Nehalem 2.0GHz 4 x 256KB L2 Cache 4MB L3 Cache LGA 1366 80W Quad-Core Server Processor (Starting with one, upgrading to 2)
• Intel BXSTS100C Passive/active combination heat sink with removable fan (the cpu does not come with a fan!)
• PQI 6GB (3 x 2GB) 240-Pin DDR3 SDRAM DDR3 1333 Desktop Memory Model MFC46GUOE-X3 (upgrading to 12GB soon. The board supports up to 6x8GB, but those 8GB chips mad expensive right now)
• HITACHI 0A38016 1TB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5″ Internal Hard Drive (Starting with four of these in two 1TB Raid1 configuration. The host machine will run software raid under Ubuntu LTS server)
• Athena Power BP-SATA3051B 3 x 5.25″ Bays to 5 x 3.5″ HD (SATA) Backplane (hot swap hard drive bays that also cool them down)
• Also, a cheapo case+power supply, adapter/cables for eSata, and a CyberPower Home CP550SL 550VA 330Watts UPS.

Total: $1277. Add another processor for $230 to bring it to about $1500. Add 6GB more DDR3 ram to bring it to about $1600.

So we have…

• Speed, upgradable: The Xeon processor is in the same league as the i7, and costs about the same as the i920. With the option for two on this board, you can get up to 8 cores (not hyperthreaded with the E5504. that requires the 5520). Seeing as the LGA1366 socket seems to be replacing LGA775, we might very well see an 8 core processor coming down the line soon. In that case, the computer can be upgraded to 16 cores. If that isn’t enough, the processing VM can be offloaded to another machine (or a cluster).
• Disk space, backed up and upgradeable: The VMs are easy to resize and easy to relocate onto a new file server if needed. By sitting on a RAID that’s also offsite backed up, the data is pretty safe.
• Stability and simplicity, since the various server functionality has been put into multiple VMs that are relatively easy to maintain independently.

There are some cons. First, the setup relies on proprietary VMware software, but other alternatives are available. Secondly, the performance is not going to be exactly like native code, and certainly graphics will be worse on this machine than one that has a native graphics card. Third, this setup is not built for perfect uptime. In fact, if the host fails, all underlying machines will fail. Luckily, data should be preserved, but this is not meant for production software/services. Finally, this setup takes quite a lot of time to get up and running… definitely not the right solution if a single low end non-vm dell/HP/etc server will satisfy your needs.

So if anybody is interested, I can keep you posted on how it goes.