Thursday, September 19, 2024

Home Data Center 4.0

 Like many long-time software developers, I've managed a small "home data center" which has evolved through many generations. (I somewhat arbitrarily numbered the latest one 4.0 though there have been many incarnations.)  This post is largely a follow-up of a similar Home Data Center blog post by my colleague Dan McDonald.  I found Dan's post very helpful -- Thanks, Dan!  So here's my update:

Requirements:

  • relatively low power, heat, and noise (will be in a living space)
  • at least 6TB of bulk storage (modest performance is OK, mostly for backups)
  • a smaller amount (at least 256GB) of high-speed storage (SSD)
  • CPU + RAM sufficient to run a few VMs and occasional software builds
  • "Headless" server with full-featured IPMI remote access

Bill of Materials (BOM)

Build, burn-in, etc.

Assembly

The build with the Fractal Designs tower case was nice and easy, except that I broke the little plastic fingers off one of the top covers by pulling too hard on the wrong side before figuring out one was meant to pull the other side first.  I'm sure that was in the instructions that I only read _after_ I broke it. (What can I say.  Who needs instructions?!? :)

One time when I was installing the main board into the case (I had it out a couple times) I failed to notice that the little push button switch that protrudes through the back of the case was caught on the edge of the plate it's supposed to go through, so the button was constantly pressed.  That caused a ton of errors during boot of the OS which I initially thought were a h/w fault, but ended up being just an assembly problem.  Careful there! :)

Burn-in

I configured the OS (I'm running OminoOS on this) with a boot/root pool on about half of the SSD space and the rest of the SSD configured as a ZFS intent log. The four SATA disks are configured as two mirror pairs.  I downloaded illumos and ran some builds on the SATA disks, and observed good performance (about 20 min. for a full build). However, I got overheat notifications from the IPMI.  The IPMI (browser-based inteface) was showing "main board" over temperature. It turned out to be the X540 I/O chip that  was overheating.  As delivered, the main board has a modest sized passive heat sync. on that chip. I found that with this case design, there was not enough airflow over that heat sync. to keep the chip in operating range while the disks are busy.  My solution was to get another small fan (see "Fan for I/O chip" above) and mount that in a position so it blows air across the X540 I/O chip. With that in place, I can run builds all day long and the chip stays cool enough to prevent faults.

Here's a picture of the I/O chip fan installation.

I installed two threaded rods (3/16 in.) with some locking nuts through some spare holes in the "bottom" of the case (the panel on which the main board is attached) and then bent them over a bit and attached the fan to those rods with some small zip ties. The locking nuts are placed to keep the fan from sliding up or down.  I adjusted the angle by just bending over the rods a bit (they're soft metal, easy to bend) so that fan axis (red arrow in picture) blows on the I/O chip (red circle in the picture).

There were lots of fan connectors on this MB so I just plugged the extra fan into one of those.
The two case fans are the in normal "fan 1" and "fan 2" connectors, and the small fan for the I/O chip is in the "fan 3" connector.  I left the IPMI fan speed programs and the default and the fan speeds seem fine.

Notes, updates, etc.

Note 1: The disks I chose (Seagate Constellation ES.3 ST4000NM0033) turned out to have a higher failure rate than some other choices.  I've had two disk failures in 18 months.  Dan McD. pointed me to this useful compilation of disk failure statistics.  (Thanks again, Dan!)

Note 2: The I/O chip on this (and apparently other X540-based board) seems to run a bit hot (I'm guessing somewhere around 5 to 10 watts). It needs either very good airflow from the case design (see my solution above) or an active heat sync with its own fan.  I looked for an active heat sync that would fit, as that would have been a much cleaner solution, but I could not find one.  A suggestion for AsRock perhaps?

The IMPI features on this board are very nice for "headless server" operation (the way I want to use the system).  The browser-based tools allow access to the keyboard/display through the ASPEED AST2500 chip providing complete remote control.

Overall, I'm quite happy with this setup.

Thanks again to Dan McDonald for publishing the details of his home data center 3.0 from which this build derives heavily.

I did this build in early 2023, so some of the parts I list above may be unavailable.  When I was working from Dan's list, the X470 was no longer available, and the X540 appeared to be the suggested replacement.  Hopefully this helps you find similar parts for your build.


Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)