Re-deploying My Old Server Supermicro X7DBE

So, I was originally moving from "real" servers to NAS units, in order to try and cut down on my power consumption a little bit and also to make managing dumb storage a lot easier, so I originally purchased a Buffalo LinkStation 441e 4-bay diskless NAS unit, with the original intention of plugging in four 6 TB HGST SATA 6 Gbps 7200 rpm HDDs in it, and when that didn't work, said NAS unit was relegated to only using four 3 TB drives instead.

Fast forward three years, and I guess that I just got tired of the fact that the Buffalo LinkStation 441e couldn't read/write the data at anything more than 20 MB/s with my Windows clients. So I decided that I am going to re-deploy my server as an actual server for dumb file storage and data serving tasks.

Hardware specs:

Supermicro SC826 12-bay, 2U, SATA rackmount chassis

Supermicro X7DBE dual Socket 771 motherboard

2x Intel Xeon E5310 (4-cores, 1.6 GHz stock, no HyperThreading available)

8x 2 GB DDR2-533 ECC Registered RAM

2x LSI MegaRAID SAS 12 Gbps 9240-8i (SAS3008)

SIMLP (it came with it, but I think that the IPMI card is dead)

4x HGST 3TB SATA 6 Gbps 7200 rpm HDDs

OSes that I tried:

1) TrueNAS Core 12.0 U1.1

2) Solaris 10 1/13 (U11)

3) CentOS 7.7.1908

4) TrueNAS Core 12.0 U1.1

I first tested the server using four HGST 1 TB SATA 3 Gbps 7200 HDDs in order to test the resiliency of ZFS in TrueNAS Core 12 (FreeBSD) by randomly yanking and plugging the drives back in into random locations to see what ZFS would do about it.

Of course, with a raidz pool, it was only fault tolerant to one drive (out of the four), which meant that, as expected, pulling out two drives killed the pool (took the pool permanently offline such that even if I plugged the drives back in, it would still report the pool as having failed).

I copied a little bit of data onto the pool to see what it would do.

And then I figured "well, since I was going to use ZFS, why not use the OS that started the whole ZFS thing in the first place?" - Solaris.

After anywhere from a day-and-a-half to three days, I finally got Solaris onto the system.

But then I wasn't able to get Samba up and running. (It still surprises me that the native samba package that ships with Solaris 10 - they NEVER got that working out-of-the-box, even with all of the updates to the Solaris 10 release cadence.) I found a samba package from OpenCSW, but that was only SMB 1.0, which is disabled in Windows 7 by default due to security vulnerabilities.

So, I wasn't able to get that up and running and quickly abandoned it.

Next, I tried to use CentOS.

I use CentOS for my micro cluster and also the micro cluster headnode, so I have some experience with it, but again, because of how my backplane was wired up where three of the 3 TB drives was connected to one of the HW RAID HBAs and the fourth drive was connected to the second one, it meant that I would have needed to use mdadm to make it work, which I was a little bit weary of, on the off chance that said md array would fail.

At least with ZFS, there appears to be more resources available for help, should I need it, as that seems to be all the rage in the Linux storage subspace right now, despite the fact that for years, I have been telling people that it's not my favourite filesystem to work with due to the fact that there are ZERO external data recovery tools for data that resides on hard drives that belonged to a ZFS pool. (A point which is still true today. i.e. if your ZFS pool dies, you can't do a bit read on the drive in order to salvage whatever information you can off the platters themselves and try and reconstruct the data/your files whereas with (at least a single NTFS drive, you CAN do a bit read on the drive and try and pick up/pick off whatever data you can with that).)

So, CentOS didn't really work like I thought it would've/could've.

So that sent me back to TrueNAS Core 12.0 because at least, it has a nice, web-based GUI (I'm so done and tired of looking up commands to copy-and-paste into a command prompt or terminal or over ssh to get the system setup).

I did consider UNraid, but the problem with UNraid is that it will fill up one drive and then the next and then the next, which means that the write speed of a single drive can quickly become a bottleneck for your entire server.

It's too bad that Qnap doesn't publish/sell their QTS software so that you can install it on any hardware because I really like Qnap's software. And it's also too bad that the only way that you can get the Qnap software is if you buy their hardware as well, which can be quite expensive for the hardware that you are getting and with that, it doesn't even necessarily support some of the other nice-to-haves that I would want, perhaps in the future, from my future storage server(s) (like being able to install a Mellanox/Nvidia (I'm old school, I still call it Mellanox because that's what it is) ConnectX-4 100 Gbps Infiniband network card.

 

So TrueNAS Core 12.0 became the selected candidate, and I proceed to get everything all set up and up and running.

 

As it stands, CIFS/SMB and NFS is running, although I AM running into a permissions issue between the two differing protocols right now (Windows clients connect over CIFS/SMB and my Qnap Linux based NAS units connect to it over NFS because for some reason, it fails to log in with CIFS/SMB to the TrueNAS Core 12.0 system). I posted the question in the forums, and it seems that nobody has identified a cause nor a fix for this yet.

Luckily, most of the time, it's the Windows clients that uses this newly re-deployed server rather than my Qnap NAS units.

But this partially documents and chronicles my journey with TrueNAS Core 12.0 and also my old Supermicro server.

It took almost like 4 days to get the server back up and running. But it's humming quite nicely now.

The only downside is that I think it's consuming somewhere around like 205 W of power or something like that vs. the Buffalo LinkStation 441e only had a 90 W AC adapter.

And the weight of having to move/lug the server around before I finally put it into the rack. (It's just sitting on top of stuff. I don't know if I actually have rails for it.)

Oracle Solaris Is A Self-Terminating Ecosystem

So, recently, I was trying to re-deploy one of my servers to take over dummy fileserving duties from a NAS unit that really just wasn't up to the task (more on that later).

Here is what I was looking to be able to at least try and do (here is a list of my performance/technical requirements, in no particular order):

1) Actually be able to hit and/or sustain gigabit ethernet (1 GbE) line speeds (e.g. ~ 100 MB/s sustained transfer rates).

2) Be able to create a share folder that both Windows and Linux clients can use, simultaneous. (So, using a combination of CIFS/SMB for the Windows side and NFS (mostly) for the *nix side.)

3) Due to the way that my old server was setup and how the SATA backplane was wired up, it meant that three of the drives in the top row of a 12-bay, 2U server was attached to hardware RAID controller #1 whilst the fourth drive (also in the top row of said 12-bay, 2U server) was attached to hardware RAID controller #2. What this meant was that I couldn't create a single logical drive/RAID array consisting of all four drives because said all four drives weren't attached to a single controller. And I didn't want to rewire the backplane.

Therefore; as a result of this, I needed a software-based solution that would be able to create a RAID5 array across the two controllers, and my preference was for ZFS.

So here are the OS candidates:

1) FreeBSD (TrueNAS Core 12.0 U1.1)

2) Solaris 10 1/13 (U11)

3) Linux (CentOS 7.7.1908)

I've previously used Solaris 10 before for a similar kind of dumb file serving duties and being that, for example, ZFS on Linux on root JUST became a "thing" as of like Ubuntu 20.04, whilst ZFS on root for Solaris has been a thing since Solaris 10 10/08 (U6) (Source: https://en.wikipedia.org/wiki/Oracle_Solaris#Version_history), I figured that I would give ZFS another go.

(Previously, I had sworn off of ZFS because I had some major problems with it - problems that even after purchasing a premium support subscription directly from Sun Microsystems themselves, the creators of ZFS, they weren't able to help me fix/resolve my problem/issue with ZFS. So I avoided ZFS like the plague.)

As it turned out, trying to get Solaris 10 1/13 onto my old server become quite the challenge.

First off, the server only sports a Supermicro X7DBE motherboard, which only comes with two USB 2.0 ports in the back and none in the front. (I think the chassis that I am using is a Supermicro SC826, but I'm not 100% sure). The motherboard itself has ports for PS/2 mouse and keyboard, but as such, I don't really have nor use PS/2 mouse nor keyboards anymore (I've finally "modernised" and moved up to USB mice and keyboards) which meant that I was short for USB ports.

So, how do I plug in a mouse, keyboard, AND install Solaris?

Well, at first, I tried installing with an USB slim, external DVD drive, but for some reason, the graphical installer had a problem loading/reading off from the DVD+RW disk that I use pretty much for all of my OS installs that needs to do so from an actual DVD.

And then I tried to use a USB flash drive, but that had problems as well.

So, I remember years ago, deploying a PXE install server so that I can get Solaris onto the system.

So that's what I set off to do.

And oh boy, what a journey (in a bad way) that was.

First off, to be able to deploy Solaris over PXE, pretty much all of the official Oracle documentation asssumes that you have another system already running Oracle Solaris.

(I did try to deploy it using a Linux system (Ubuntu), but that didn't really work out the way it should've, based on people who's written about it on other blogs previously.)

So, deploying it using another Oracle Solaris system it is.

I fire up Oracle VirtualBox, and get to installing Solaris 10 1/13 on said VM and I didn't really have much of a problem with that.

(Maybe perhaps in a little bit of irony, I was running the Solaris VM off an Intel NUC. I'm actually finding myself using my Intel NUC for a BUNCH of VMs now whereas before, it was only relegated to the task of being a license server for some of my applications, i.e. it wasn't really doing much.)

I get the VM all set up, running Java Desktop 3 (yes, that was a thing), per usual. Nothing special to write about there.

Oracle Solaris 10 1/13 Java Desktop System 3. Yes, it's a thing.

After that, I then proceed to follow the instructions, as published by Oracle, for Oracle Solaris 10 1/13 Installation Guide: Network-Based Installations.

The instructions for creating an install server with DVD media (for x86 systems (as opposed for SPARC systems)) you follow the instructions here and there is nothing special to write about that.

Now, it says, in the instructions, that if you are going to be using a DHCP (which I was going to be because apparently, my system is so old that the PXE boot for the server can ONLY happen with DHCP, i.e. you cannot assign a static IP address to the onboard NICs themselves), so it said that you didn't need to create a boot server, but I did it anyways.

Now this is the part where it gets messed up.

According to the Oracle documentation, the next step in the process is "Adding Systems to Be Installed From the Network with a DVD Image - How to Add Systems to Be Installed From the Network With add_install_client (DVD)".

Step 3 states:

"3. Add the client to the install server's /etc/ethers file.

a. On the client, find the ethers address. The /etc/ethers map is taken from the local file.

# ifconfig -a grep ether

ether 8:0:20:b3:39:1d"

 

Here is the problem with this:

 

If you have a system that you are trying to deploy Oracle Solaris onto, you're NOT going to be able to run "ifconfig -a grep ether" on the client!

 

This is a really stupid step/line in the instructions.

Think about it. You're trying to "jumpstart" the system. The client isn't up and running yet because this is what (and why) you are setting up the PXE server for a network installation, and therefore; NOTHING is running on the target/client.

So, instead, I had to pull the MAC address by powering on the server, setting the BIOS so that it will boot from PXE, and then waiting/letting the PXE request to time out so that it will show me the MAC address for both interfaces (because I think that it might have actually tried to ennumerate thes second NIC first, which meant that I had to wait until that failed before the primary NIC's MAC address will show up).

And then took a picture with my phone so that I would then be able to go back to my desk so that I can keep going with the instructions.

That was a really, really stupid step in the instructions on the part of Oracle.

(But this isn't why I think that the entire Oracle Solaris ecosystem is a self-terminating one though.)

Okay, so besides that, you keep going with the instructions after you get the MAC address off the NIC from the server.

Note here: If you are running these steps back to back, make sure you get out of the directory of where the DVD is mounted to (e.g. just type cd to get back to your root, home directory) so that you can then unmount the DVD before adding your client.

I made this mistake a bunch of times.

And THEN go to the directory where you copied the contents of the DVD to.

Proceed with the next step.

At the very bottom of that page is a one liner that says that if you are going to be using DHCP, then you need to set up the DHCP server.

Now, I thought that this was going to be like Linux installations where you just set up some dummy DHCP server, tell it what's the IP address range that you want said DHCP server to dish out/serve up, and it's already going to have a TFTP server running, and away you go.

[buzzer]

 

I couldn't be more wrong.

Here's what it takes to ACTUALLY set up the DHCP for a DHCP based, PXE/network installation of Solaris 10 1/13, in Solaris 10 1/13:

(Refer to the Oracle Solaris documentation, "Preconfiguring System Configuration Information with the DHCP Service (Tasks)")

Chapter 14. Configuring the DHCP Service (Tasks), in Oracle Solaris Administration: IP Services.

You have to go through the steps of running

 

# /usr/sadm/admin/bin/dhcpmgr

 

And then you get this:

 

 

Once you're done with that, then you got to create the macro for the client, BY MAC ADDRESS, itself.

 

Oracle Solaris 10 1/13 DHCP Manager Macros BY CLIENT MAC ADDRESS

And THIS is where I think that Oracle Solaris is a self-terminating ecosystem.

In order for you to be able to install the system, especially over PXE, according to their own instructions, you apparently have to set up and permit each system, BY CLIENT MAC ADDRESS, to be able to boot from and pull the Oracle Solaris 10 install image.

(The rest of the installation ran fine, by the way. It wasn't the fastest install, but at least it worked. And it IS technicaly faster than USB 2.0, or at least faster than what an USB slim, external DVD drive can do at 4x speeds.)

In other words, you can just deploy Solaris 10 on ANY system that you want. It's almost like you have to "register" it with the DHCP manager before this will work.

That's incredibly dumb/stupid.

 

Now, it is entirely possible that you don't have to "register" the client's MAC address to get this to work, and that would be nice, but according to the Oracle Solaris official documentation, that's NOT the example that they've shown here.

 

I have NO idea why they chose to make this more complicated way (but it DOES work) as the example that they use to show/tell/teach people how to deploy an installation over your local area network, but that's precisely what they've done.

Also apparently, and I don't know exactly when this happened, but apparently, as of this writing (27 July 2021), Oracle deleted the Oracle Solaris page off from Facebook.

Sounds to me like the days for Oracle Solaris may be numbered, even if that number may still be years, possibly a decade-or-more away.

(Also, by the way, the reason why I went BACK to Solaris 10 instead of using the latest Solaris 11 is because I currently have Solaris 11 deployed on another Intel NUC of mine, that exists to perform some basic web serving tasks and for some strange, stupid reason, after a while, one of the NICs that has been assigned to it (it too, also runs in a VM), which has a static IP assigned to it, stops working. And when I try to switch it from static IP to DHCP, it fails to pick up an IPv4 address from the router's DHCP server. So I ended up just adding a second NIC to the VM and left that as being a DHCP managed interface and haven't had any issues with that NIC since, but I can't give it a static IPv4 address, of course.)

Looks like what we are seeing is the beginning of the end for Oracle Solaris.

And that's a shame/pity. I used to really like Solaris.