The database that's behind one of my Photoprism instances took a dump.

 This is for all the Linux fanbois out there and whilst this isn't actually, technically, the fault of Linux -- considering the fact that said Linux fanbois will bitch about software that runs on Windows as if that's a Windows problem, therefore; this post is going to be similarly in that vain.

So, I've got a bunch of separate Photoprism instances running, where in the background, in the same docker-compose.yml file, it uses the MariaDB as the database backend. (Which is like some form of MySQL using the InnoDB engine. I don't really know much about it, so I'll just leave it at that.)

Anyways, for some inexplicable reason, the database took a dump and Photoprim stopped being able to communicate with it.

Found out that the database itself was corrupt, and then tried restarting the mariadb part of the docker-compose.yml file with --innodb-force-recovery=2 option (because when I tried to run it with --innodb-force-recovery=1, it was still producing a fatal 11 error). I also had to add --skip-grant-table to the command that's starting the mariadb because without it, not even the root user, from inside the container, can administer the database, which is very strange because normally, a root user should be able to do anything and everything.

When I tried to run mariadb-check --all-databases, it says:

photoprism.photos

Warrning: InnoDB: Index `idx_photos_checked_at` contains 1832435 entries, should be 1832246.

error: corrupt

And then I tried to run mariadb-check -f --all-databases and it said:

(for all of the photoprism.* tables)

note: The storage engine for the table doesn't support repair.

Anyways, long story short -- the database back end took a dump, so I ended up deleting the whole thing and my system is in the process of re-indexing anything so that it can rebuild the database.

Linux sucks.

*edit*

Yes, I did try to backup the database, drop the database, re-create the database, and then importing everything from backup.

The new error that I got with that was:

[Warning] failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table `mysql.gtid_slave_pos` doesn't exist in engine

I tried to quickly google how I can re-create it, but the SQL statements seemed awfully specific to the specific person who was asking that question, and so, I assumed that that may NOT necessarily be how that table needs to be defined for use by Photoprism. So, I just ended up deleting the entire database, and let it re-build said database from scratch, all over again. We'll see how that goes. What a colossal PITA it is, to try and fix/repair said database though.

Vastly differing results in WSL2 between 5950X and 12900K in Windows 10 21H2

A little while ago, I came across this video which was talking about how you can run Linux graphical applications natively in Windows (more specifically, in Windows 11).

However, when at the time when I watched said original video, I didn't have any hardware that could actually really run that probably. My "newest" system that I had was an Intel Core i7-6700K and as far as I know, it didn't have the Trusted Platform Module (TPM) anywhere (whether it is as an external add-on dongle) or integrated into the motherboard firmware/BIOS.

So, I didn't really make much of it back then.

But since then, I've built both my AMD Ryzen 9 5950X system and also my Intel Core i9-12900K system and I figured that with some of the work that I needed the systems to be doing over with, I had a little bit of time with the system to do some more testing with it.

So I grabbed two extra HGST 1 TB SATA 6 Gbps 7200 rpm HDDs (one per system), threw Windows 10 21H2 on it, and proceeded with the instructions on how to install and configure Windows Subsytem for Linux 2 (WSL2). I installed Ubuntu 20.04 LTS (which really, turned out to be 20.04.4 LTS), and proceeded to try and install the graphical layer/elements to it.

So that's all fine and dandy. (Well, not really because in both instances, neither of the systems was able to start the display and I can't tell if it is because I have older video cards in the system (Nvidia GeForce GTX 980 and a GTX 660 respectively - because as a CentOS 7.7.1908 system, it didn't really matter what I had in there since I was going to remote in over VNC anyways).)

But, since I had it installed, AND by some miracle, Windows 10 picked up on the Mellanox ConnectX-4 dual port VPI 100 Gbps Infiniband cards automatically, I just had to manually give each card in each system an IPv4 address so that it can talk to my cluster headnode (which was still running CentOS along with the OpenSM), and connect up to the network shares that I had set up. (SELinux is a PITA. But I got Samba going on said CentOS system so that on the Linux side, it can connect up to the RAID arrays using NFS-over-RDMA whilst in Windows, it's just through "normal" Samba (i.e. NOT SMB Direct).)

So, I might as well benchmark the systems to see how fast it would be able to write and read a 1024*1024*10240 byte file.

And for fun, I also installed Cygwin on both of the systems as well, so that I can compare the two together.

Being that both systems was able to pick up the Mellanox ConnectX-4 card right away (I didn't have to do anything special, install the Mellanox drivers, etc.), I was able to connect up to my cluster headnode and the Samba shares were visible immediately. As a result of that, I was able to right-click on both of those shared folders and map it to a network drive directly and automatically.

Now, in WSL2, I had to mount the mapped network drive using the command:

$ sudo mount -t drvfs V: /mnt/V

(Source: https://superuser.com/questions/1128634/how-to-access-mounted-network-drive-on-windows-linux-subsystem)

And then once that was done, I was able to run the follow commands in both Ubuntu on WSL2 and also in Cygwin:

Write test:
$ time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240

Read test:
$ time -p dd if=10Gfile of=/dev/null bs=1024k

Here are the results:

Huh. Interrresting.

I have absolutely NO clue why WSL2 on the 5950X is so much slower compared to WSL2 on the 12900K.

But what is interesting though is that the speeds are close, with the 5950X being a little bit faster under Cygwin than the 12900K, also under Cygwin.

I decided to blog about this because there is a potential possibility that for those that might be working with WSL2, the hardware that you pick MAY have an adverse performance impact.

I'm not sure who, if anybody, has done a cross-platform comparison like this before but to be honest, I haven't really bothered to look for it either because you might have reasonably expected that this significant performance difference wouldn't/doesn't exist, but the results clearly show that there's a difference. And a rather significant difference in performance at that.

Please be aware and you should do your own testing for your workload/case/circumstance if you get a chance to be able to do so.

Getting the latest and greatest hardware running in Linux is sometimes, a bit of a nightmare

Just prior to the holidays, I decided to upgrade one of three of my systems and consolidate it down to two. My old Supermicro Big Twin^2 Pro micro cluster server and two HP Z420 workstations (that I was using in lieu of the Supermicro because the Supermicro was "too loud") were getting replaced by an AMD system, built on the Ryzen 9 5950X CPU and an Intel system, built on the latest and greatest that Intel had to offer - the Core i9-12900K.

So, I speced out all of the rest of the hardware, which really, consisted of the motherboard, RAM, and the CPU heatsink and fan assembly whilst I was able to reuse some of my older, existing components as well. (I did have to buy an extra power supply though because I had originally miscalculated how many power supplies that I would need.)

So that's all fine and dandy. All of the hardware arrived just before the start of the Christmas break for me, so I started to set up the AMD system. Install the CPU, the RAM, the CPU HSF, plug everything in, check and double check all of the connections - everything is good to go. I used Rufus USB to write the CentOS 7.7.1908 installed onto a USB drive, plug in the keyboard, mouse, and flip the switch on the power supply and off I go right?

[buzzer]

Nope!

Near instant kernel panic. Nice.

 

As you can see from the picture above, less than 3 seconds into the boot sequence from the USB drive - Linux has a kernel panic.

Great.

So now I get the "fun" [/sarcasm] job of trying to sort this kernel panic out. Try it a few more times, the same thing happens.

So, ok. Now I'm thinking that the hardware is too new for this older Linux distro and version (and kernel). So, I take out my Intel Core i7-3930K system (one of them that I use to run my tape backup system), and I plug the hard drive into that system, along with the video card back in, and run through the boot and installation process (which worked without any issues of course), power down the 3930K, take the hard drive back out, and plug it into the 5950X system. Power it on. (I set the BIOS to power on after AC loss so that I can turn on the system even when it isn't inside a case and I don't have a power button connected to it.)

The official CentOS forums state that they only support CentOS 7.9.2009, so I try that as well, still to no avail.

Eventually, I end up using a spare Intel 545 series 512 GB SATA 6 Gbps SSD that I had laying around so that I could try installing and re-installing, trying different drivers, kernel modules, kernels, etc. a LOT faster than I was able to with a 7,200 rpm HDD.

End net result: I filed a bug report with kernel.org because the mainline kernel 5.15.11 kept producing kernel panics with the Mellanox 100 Gbps Infiniband network card installed. And it didn't matter whether I tried to use the "inbox" CentOS Infiniband drivers or the "official" Mellanox OFED Infiniband drivers.

Yet another Linux kernel panic.

Interestingly enough, the mainline kernel 5.14.15 works with the Infiniband NIC just fine. So that's what I landed on/with. 

The other major problem that I ran into was that the Asus X570 TUF Gaming Pro (WiFi) used the Intel I225-V 2.5 GbE NIC. Unbeknownst to me when I originally purchased the motherboard, I didn't realise that Intel does NOT have a Linux driver (even on Intel's website) for said Intel I225-V 2.5 GbE NIC. And what was weird was that when I migrating the SSD over during the testing and trying to find/figure out a configuration that worked, said Intel onboard 2.5 GbE NIC would work initially, but then it would eventually and periodically drop out and so that was quite the puzzle because if there wasn't a driver for it, then how was it that it was able to work when I moved the drive over?

As a result of that, that took up a couple of days where I would be trying to clone the disk image over from the Intel SSD over onto the HGST HDD using dd and in the end, that didn't work either.

So, what did I end up with?

This is the hardware specs that I ended up with on the AMD system:

CPU: AMD Ryzen 9 5950X (16-core, 3.4 GHz stock base clock, 4.9 GHz max boost clock, SMT enabled)

Motherboard: Asus X570 TUF Gaming Pro (WiFi)

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 980

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 5.14.15-1-el7.elrepo.x86_64

 

I ended up adding the Intel Gigabit CT Desktop NIC because a) it was an extra Intel GbE NIC AIC that I had also laying around, and b) it proved to be able to provide a vastly more reliable connection than the onboard Intel I225-V 2.5 GbE due to the driver issue.

Now that I have the system set up and running, there is a  higher probabilty that the igc kernel module probably works more reliably now than it did when I was originally setting up the system, but being that it was not reliable when I was doing the initial setup and testing, I am less likely to use said onboard NIC, which is a pity. Brand spankin' new motherboard and I can't even use nor trust the reliability of the onboard NIC. And I can't even blame Asus for it because it is an Intel NIC. (Sidebar: Ironically, the Asus Z690 Prime-P D4 motherboard that I also purchased uses a Realtek RTL8125 2.5 GbE NIC, which I WAS able to find a driver for that and it has been working flawlessly with it.)

That took probably on the order of around 10 days, from beginning to end, to get the AMD system up and running.

The Intel system was a little bit easier to set up.

The kernel panic issue with the mainline 5.15.11 kernel and Infiniband was also present on the Intel platform as well.

Interestingly and ironically enough, the newer kernel kept crashing or had severe stability issues. It turns out that I did NOT install the RAM correctly (i.e. in the DIMM_A2 and DIMM_B2 slots), so since then, I've corrected that.

Keen readers might note that I have stated that I have 4 sticks of RAM, except that one of the sticks arrived DOA, and is currently being sent back to Crucial under RMA, so when it comes back, then I will be able to install the extra stick that is currently not installed and the stick that is due back from the RMA exchange.

I might try the newer kernels again later, but for now, at least the system is up and running so that I can start making it do the work that I need it to be doing.

The system stability issues due to the error that I made when installing (and uninstalling) the RAM (because I was testing the stick of RAM that wouldn't POST that ended up getting RMA'd back to Crucial), I ended up with a RAM installation configuration that wasn't correct and the resulting system stability issues ate up a few more days.

So, in the end, it took me almost the entire Christmas holiday to get both of these systems up and running.

(This is also a really good reason why traditionally, I have stuck with workstation and server hardware because on my old Supermicro micro cluster, I can deploy all four nodes in 2 hours or less. It's a pity that the system is too loud.)

This is the hardware that I ended up with on the Intel system:

CPU: Intel Core i9-12900K (16 cores (8P + 8E), 3.2 GHz/2.4 GHz base clock speed, 5.2 GHz/3.9 GHz max boost clock, HTT enabled)

Motherboard: Asus Z690 Prime-P D4

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 660

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 3.10.0-1127.el7.x86_64

AMD Ryzen 9 5950X is faster than the Intel Core i9-12900K for mining Raptoreum

The results speak for themselves.

 

The AMD Ryzen 9 5950X is faster at mining Raptoreum than Intel's latest and greatest 12th gen Core i9-12900K.

 

System/hardware specs:

AMD:

CPU: AMD Ryzen 9 5950X (16-core, 3.4 GHz stock base clock, 4.9 GHz max boost clock, SMT enabled)

Motherboard: Asus X570 TUF Gaming Pro (WiFi)

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 980

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 5.14.15-1-el7.elrepo.x86_64

Intel:

CPU: Intel Core i9-12900K (16 cores (8P + 8E), 3.2 GHz/2.4 GHz base clock speed, 5.2 GHz/3.9 GHz max boost clock, HTT enabled)

Motherboard: Asus Z690 Prime-P D4

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 660

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 3.10.0-1127.el7.x86_64

Configuration notes:

I had about two weeks over the Christmas break 2021 to receive all of the hardware, assemble the systems, and get the systems set up and up and running. And that was quite the endeavour because with the latest and greatest hardware, the older versions of CentOS (7.7.1908) and the older kernel didn't work with all of the features and functions with this level of hardware.

As a result, I had to "jumpstart" both systems by first installing the OS using my Intel Core i7-3930K system (Asus X79 Sabertooth motherboard, 4x Crucial 8 GB DDR3-1600 unbuffered, non-ECC RAM, Mellanox MCX456A-ECAT, GTX 660) first, and then update the systems (at least in part) before I can transplant the hard drive with the OS install into their respective systems and finish setting the systems up. (I will write more about that "journey"/clusterf in another blog post here shortly because it was quite the journey to jumpstart both of these systems simultaneously, which took pretty much the full two weeks that I had.)

You will find out how and why I ended up with the respective hardware choices in that blog post.

I am using cpuminer-gr-1.2.4.1-x86_64_linux from here (https://github.com/WyvernTKC/cpuminer-gr-avx2/releases/tag/1.2.4.1).

For the Intel system, because of the existence of the combination of (P)erformance Cores and (E)fficiency Cores and HyperThreading, this resulted in more combinations that I had to test in order to find the setting that had the highest Raptoreum hash rate as reported from their benchmarking tool. Each time the CPU configuration changed, I ran a full tune again, which you might well imagine, took quite some time to do. In the cases where the efficient cores were disabled, I also tested and re-ran the full tune for both AVX2 and AVX512.

The AVX512 runs (both times I ran it, i.e. with and without HyperThreading), resulted in thermal throttling with about a 23-24 C ambient at the time.

For the AMD system, testing it was a lot simpler because it was either only SMT on or off.

Results:

The results speak for themselves.

The AMD Ryzen 9 5950X with SMT enabled produces the highest hash rate (3953.64 hashes/second). Compare that with the run where SMT was disabled, enabling SMT results in about a 10.3% increase in the hash rate performance result.

The Intel Core i9-12900K results are an interesting case. Despite the plethora of benchmarks talking about how great and how fast the latest and greatest from Intel is (and there ARE some things that said latest and greatest from Intel are great at), unfortunately, for Raptoreum mining, this is not one of them.

At best, the 5950X with SMT enabled compared to the 12900K with all 16 cores AND HyperThreading enabled puts the 5950X at about 80.9% faster in Raptoreum hash rate performance vs. the 12900K.

Comparing like-for-like thread count, whether it is 8P+8E without HyperThreading, the 16 core/16 thread of the 5950X is still approximately 64.2% faster than the 12900K. Without the efficiency cores, but turning HyperThreading back on (i.e. 8P+0E with HyperThreading), the 12900K again is bested by the 5950X by a 71.8% margin.

Unfortunately, running this in Linux meant that I didn't have or didn't know of a tool like Hardware Info 64 to be able to report power consumption figures/values. Maybe I might get around to re-running this test again in Windows, but for now, this might be helpful to those who might be interested in looking for guidance if mining Raptoreum is on your mind.

Apparently, GlusterFS No Longer Supports RDMA And You Can't Use It Across Ramdrives Anymore

Back in the day, I used to use CentOS 7.6.1810 with GlusterFS 3.7 and I was able to create ramdrives in said CentOS and then tie a bunch of ramdrives together with GlusterFS.

Apparently, that's not the case anymore and it hasn't been since Version 5.0 as RDMA was deprecated.

Bummer.

Here's why this was important (and useful):

SSDs, regardless of whether it's consumer-grade or enterprise-grade, the NAND flash memory cells/chips/modules that's used in them all have a finite number of program/erase cycles.

Therefore; as a result, ALL SSDs are consumer wear components (like brake pads on a car) where they are designed to be replaced after a few years due to said wear. (This is a point that unfortunately, I don't think that the SSD as an industry, as a whole, spends enough time focusing on because a LOT of people were and are using SSDs as a boot drive, and as a boot drive, because it has a finite number of program/erase cycles, this means that it is only a matter of time before the system will fail, but I'm going to write/rant about that some other time/day.)

But for now, the key takeaway is that SSDs have a finite number of erase/program cycles and that can cause SSDs to fail.

So, in the HPC space, where I am running simulations, I can produce a LOT of data over the course of a run, sometimes, into the PB/run territory.

Therefore; if I have a large amount of data that needs to be read and written, but I don't need to keep all of the transient data that the solver produces the course of a simulation, then I want it to be as fast as possible, but also NOT have it be a money pit where I am constantly pouring money into replacing SSDs (again, regardless of whether it's consumer grade SATA SSDs or enterprise grade U.2 NVMe SSDs).

So, this was where the idea came from - what if I were to create a bunch of ramdrives, and then tie them together somehow?

Originally GlusterFS was able to to do this with gluster version 3.7.

I would be able to create a tmpfs partition/mount point, make that a GlusterFS brick, and then create a GlusterFS volume with those bricks and then export the GlusterFS volume onto the Infiniband network as a NFSoRDMA file system.

And it worked ok for the most part.

I think that I was getting somewhere around like maybe 30 Gbps write speeds on it (for the distributed stripped volume).

Lately, I wanted to try and deploy that again, but for creating plots for the chia cryptocurrency.

Apparently, that wasn't possible/capable anymore.

And that just makes me sad because it had so much potential.

You can create the tmpfs.

Gluster will make you think that you can create the Gluster bricks and volume.

Gluster lies (which you only find out when you attempt to mount the gluster volume that it never really created the bricks (on tmpfs) to begin with).

And then Gluster-hell-breaks-loose because it thinks that the bricks are a part of a gluster volume already which locks the bricks and volume together, and nowhere in the Gluster documentation does it tell you how to dissociate a brick from a volume or vice versa.

And that's too bad that because GlusterFS had so much potential.

Re-deploying My Old Server Supermicro X7DBE

So, I was originally moving from "real" servers to NAS units, in order to try and cut down on my power consumption a little bit and also to make managing dumb storage a lot easier, so I originally purchased a Buffalo LinkStation 441e 4-bay diskless NAS unit, with the original intention of plugging in four 6 TB HGST SATA 6 Gbps 7200 rpm HDDs in it, and when that didn't work, said NAS unit was relegated to only using four 3 TB drives instead.

Fast forward three years, and I guess that I just got tired of the fact that the Buffalo LinkStation 441e couldn't read/write the data at anything more than 20 MB/s with my Windows clients. So I decided that I am going to re-deploy my server as an actual server for dumb file storage and data serving tasks.

Hardware specs:

Supermicro SC826 12-bay, 2U, SATA rackmount chassis

Supermicro X7DBE dual Socket 771 motherboard

2x Intel Xeon E5310 (4-cores, 1.6 GHz stock, no HyperThreading available)

8x 2 GB DDR2-533 ECC Registered RAM

2x LSI MegaRAID SAS 12 Gbps 9240-8i (SAS3008)

SIMLP (it came with it, but I think that the IPMI card is dead)

4x HGST 3TB SATA 6 Gbps 7200 rpm HDDs

OSes that I tried:

1) TrueNAS Core 12.0 U1.1

2) Solaris 10 1/13 (U11)

3) CentOS 7.7.1908

4) TrueNAS Core 12.0 U1.1

I first tested the server using four HGST 1 TB SATA 3 Gbps 7200 HDDs in order to test the resiliency of ZFS in TrueNAS Core 12 (FreeBSD) by randomly yanking and plugging the drives back in into random locations to see what ZFS would do about it.

Of course, with a raidz pool, it was only fault tolerant to one drive (out of the four), which meant that, as expected, pulling out two drives killed the pool (took the pool permanently offline such that even if I plugged the drives back in, it would still report the pool as having failed).

I copied a little bit of data onto the pool to see what it would do.

And then I figured "well, since I was going to use ZFS, why not use the OS that started the whole ZFS thing in the first place?" - Solaris.

After anywhere from a day-and-a-half to three days, I finally got Solaris onto the system.

But then I wasn't able to get Samba up and running. (It still surprises me that the native samba package that ships with Solaris 10 - they NEVER got that working out-of-the-box, even with all of the updates to the Solaris 10 release cadence.) I found a samba package from OpenCSW, but that was only SMB 1.0, which is disabled in Windows 7 by default due to security vulnerabilities.

So, I wasn't able to get that up and running and quickly abandoned it.

Next, I tried to use CentOS.

I use CentOS for my micro cluster and also the micro cluster headnode, so I have some experience with it, but again, because of how my backplane was wired up where three of the 3 TB drives was connected to one of the HW RAID HBAs and the fourth drive was connected to the second one, it meant that I would have needed to use mdadm to make it work, which I was a little bit weary of, on the off chance that said md array would fail.

At least with ZFS, there appears to be more resources available for help, should I need it, as that seems to be all the rage in the Linux storage subspace right now, despite the fact that for years, I have been telling people that it's not my favourite filesystem to work with due to the fact that there are ZERO external data recovery tools for data that resides on hard drives that belonged to a ZFS pool. (A point which is still true today. i.e. if your ZFS pool dies, you can't do a bit read on the drive in order to salvage whatever information you can off the platters themselves and try and reconstruct the data/your files whereas with (at least a single NTFS drive, you CAN do a bit read on the drive and try and pick up/pick off whatever data you can with that).)

So, CentOS didn't really work like I thought it would've/could've.

So that sent me back to TrueNAS Core 12.0 because at least, it has a nice, web-based GUI (I'm so done and tired of looking up commands to copy-and-paste into a command prompt or terminal or over ssh to get the system setup).

I did consider UNraid, but the problem with UNraid is that it will fill up one drive and then the next and then the next, which means that the write speed of a single drive can quickly become a bottleneck for your entire server.

It's too bad that Qnap doesn't publish/sell their QTS software so that you can install it on any hardware because I really like Qnap's software. And it's also too bad that the only way that you can get the Qnap software is if you buy their hardware as well, which can be quite expensive for the hardware that you are getting and with that, it doesn't even necessarily support some of the other nice-to-haves that I would want, perhaps in the future, from my future storage server(s) (like being able to install a Mellanox/Nvidia (I'm old school, I still call it Mellanox because that's what it is) ConnectX-4 100 Gbps Infiniband network card.

 

So TrueNAS Core 12.0 became the selected candidate, and I proceed to get everything all set up and up and running.

 

As it stands, CIFS/SMB and NFS is running, although I AM running into a permissions issue between the two differing protocols right now (Windows clients connect over CIFS/SMB and my Qnap Linux based NAS units connect to it over NFS because for some reason, it fails to log in with CIFS/SMB to the TrueNAS Core 12.0 system). I posted the question in the forums, and it seems that nobody has identified a cause nor a fix for this yet.

Luckily, most of the time, it's the Windows clients that uses this newly re-deployed server rather than my Qnap NAS units.

But this partially documents and chronicles my journey with TrueNAS Core 12.0 and also my old Supermicro server.

It took almost like 4 days to get the server back up and running. But it's humming quite nicely now.

The only downside is that I think it's consuming somewhere around like 205 W of power or something like that vs. the Buffalo LinkStation 441e only had a 90 W AC adapter.

And the weight of having to move/lug the server around before I finally put it into the rack. (It's just sitting on top of stuff. I don't know if I actually have rails for it.)

Here Is The Security Problem With HiveOS' Diskless PXE Boot System

Here is the problem with the HiveOS diskless PXE boot system: 

If you have a HiveOS installation (this security flaw exists regardless of whether you are running the HiveOS' diskless PXE system/setup or whether you are using a disk of some sort for the HiveOS image), that is connected to the public internet (which is preferred so that your cryptocurrency shares that you are mining/farming will have the lowest latency back to the presumably, pool servers as much as possible), and that you have the SSH port open (port 22), then hackers and attackers can remotely take over your system, as laid out in Simon Bell's blog post Sneaky Malware Reconfigures Hive OS Wallet for Profit.

 

After you install HiveOS' diskless PXE system, if you look at what's inside /etc/shadow, you will notice that the salt and the key for user is the same, even if you update the image. (The more interesting question is whether the salt is the same across different users or not, but I am unable to personally test and verify this.)

This is important is because this is how the HiveOS diskless PXE is able to keep it so that the default password for the default user ("user") as "1".

(It uses the SHA-512 based password algorithm.)

The fact that the salt and the key is consistent through different HiveOS diskless PXE images, even if you update the image is one of the reasons why it is fundamentally and principally insecure.

(Ironically, if you do end up getting hit with this by hackers/attackers, one of the first things that you can do if you have a PXE installation is to power cycle your system/rig whereby you can take advantage of this fact that the HiveOS image that is served up by your PXE server will just reset and restore the default setup/configuration of your entire rig and since it will pull the flight sheet from your HiveOS account, you won't have a problem there.

In other words, there are pros and cons to using the HiveOS' diskless PXE setup/system because if you first notice that your crypto in your wallet or pool address isn't increasing as your system is mining and it is supposed to be, then you will be able to very quickly resolve and rectify this problem by just power cycling your rig given the fact that the HiveOS diskless PXE image lives in the local RAM of your rig, which means that turning your rig off -- any of the changes that the hacker/attacker has made will be gone/lost as far as the hacker/attacker is concerned since the image only lived in RAM and not on a non-volatile storage device like a HDD or SSD.

By the same token, the fact that the default password is the same for the default user ("user") also is a security flaw unto itself for HiveOS' diskless PXE system because this is what allows the hacker/attackers shell script to be able to run in the first place.

The HiveOS team, in their Telegram post, writes:

But as my testing and research has shown, you can't actually abide by their recommendation of using more secure passwords because any changes that you make to your machine/system/rig's HiveOS password will ONLY be temporarily changed until the next reboot or power cycle.

To test and prove this, you can either actually perform a reboot, or you can just extract any of the hiveramfs.tar.xz images (or any of the backup images in <<path_to_your_pxeserver_root>>/backup_fs) and then cat ./etc/shadow | grep user and you will see that in your current image and any of your backup images, the result is the same.

The other really big security flaw with this setup is also the fact that the HiveOS team DOES give you a command (hive-passwd) so that you can change the password locally on your system.

The problem with this command is that you have to pass in the new password as an input argument to that command, which means that the password will be typed in, in plain text.

This also means that by default, this system, which is based on the Ubuntu 18.04.5 LTS distribution, also uses bash as the default shell. As a virtue of the bash shell, if you type in the command: history, you will be able to read, in plain text, the password that the user has changed their system to.

HiveOS' diskless PXE setup does NOT clear your bash shell history with each login or with each log off.

And then, on top of all this, if a hacker/attacker is even mildly or remotely sophisticated, and your mining rig is directly connected to the public internet with port 22 open, it won't take them much to be able to find out whether you are running a diskless PXE setup or not. They can literally just read the config files, in plain text, to be able to determine and ascertain that. If you are running a PXE server, then you run the risk where they can start hacking into the rest of your network via your PXE server, or if they are only interested in hacking your mining rig and stealing your crypto, then all they would need to do is to run the hive-upgrade.sh script, and after your system is done updating, they would just need to unpack the PXE image into a temporary location that you aren't aware of on your PXE server, change the salt and the key for their new password and encode that into /etc/shadow, and then re-pack the hiveramfs.tar.xz file back up, and then pass the hacked version of hiveramfs.tar.xz as the "good" image (which as far as I can tell, HiveOS doesn't check), send the reboot signal to your rig so that it will pull the new, hacked image, and now you're hosed to the point where even a power cycle won't help you fix/restore your PXE image.

(And this also assumes that they haven't locked you out of the hive-upgrade.sh so that you can just run the update again, and undo those changes, which I can only imagine that a hacker/attacker that's slightly more sophisticated may probably be able to do that with what I can only assume, would amount to minimal effort (to lock you out of the hive-upgrade.sh command).

In fact, if they were smart, they would delete the hive-upgrade.sh command after they're done modifying your image, and also delete the pxe-setup.sh command as well so that you can't re-setup/re-download the image from the HiveOS server/repository which will undo said hacker/attacker's changes.)

And now your entire PXE setup is screwed until you rebuild the PXE server again, so that you can re-deploy the diskless PXE image again.

These are a bunch of inherent security flaws with the HiveOS' diskless PXE system.

I tried to warn the HiveOS team on the HiveOS forums, but thanks you @HaloGenius there, he decided to take it upon himself to just label this as fear, uncertainty, and doubt (FUD) when the statements that I have presented here are VERY easy to prove (because I've been able to prove them and I'm an idiot as far as computers and sysadmin is concerned). It doesn't take much.

In fact, I am such an idiot that I used the hive-passwd command to change the password to a more secure password and then I forgot what it was that I changed said password to, so I power-cycled my rig so that it would pull the HiveOS image from my PXE server again just to "reset" the password. (Because if you use the hive-passwd command and you mess it up, you don't really have a good way of recovering from that (or at least it's not an easy way to recover from that).) Whoopsies! (When I said that I'm an idiot when it comes to this stuff, I'm not kidding.)

So....what does this all mean? And what can you do?

The best thing that the HiveOS team can do at this point is to take a good, long, hard look at the security issues that I have presented here, building on top of the work that Simon Bell had presented almost three weeks ago (8 July 2021).

What I would LIKE the HiveOS team to do would be that when you run the pxe-setup.sh command for the first time, it will ask you what you want the default rig password to be (you WILL need to make sure that you don't screw this one up because if you toast it, then you will have to rerun the pxe-setup.sh command again to correct it). 

That way, it will download the original image with the default password for the default user, and then right away, you will get to change it and it will replace the line that's in /etc/shadow with the new password, at minimum, and preferably, with a new salt and password.

Right now, it will ask you what your farm hash is, what's the IP address of your PXE server, etc. The HiveOS team should be able to add another, new question, "what do you want the default password to be?" and then you can type it in (preferably echo off), and IF and/or anywhere where HiveOS might need to store that in plain text anywhere, that it will only do so temporarily and then it will clean up after itself so that it won't leave any reminents behind.

When I read Simon Bell's blog and I started thinking about what this means for a diskless PXE setup, I never realised how many failure modes there are that existed with the diskless PXE setup until I started really taking a look into this.

That's would be the minimum of what I would recommend the HiveOS team to take a look at, and implement those changes or something along those lines.

In the meantime, here is what you can do to protect yourself:

1) Secure your local area network. There are no shortage of videos on YouTube that you can look up and watch (or google it and read) where there are people who are dedicated to this topic, so follow their best practices.

2) Use a VPN. It won't really help secure what's inherently insecure, but at least it adds one more layer of protection. It will take your rig off the public internet, and therefore; will increase your latency times for shares, but I suppose that's better than getting hacked, right?

3) Close off port 22 on your router if you don't absolutely need SSH to remote into your network. I know that this isn't always possible, but if and where it is possible, you should probably close that off. (If you DO need port 22 to remain open, there are ways to do so in a more secure fashion as well. You can google and/or YouTube that as well.)

4) Change the password to a more secure password. Yes, I know that you have to pass in your new password in the hive-passwd command, but if you do it right after your system boots up (each time, preferably), then it will help safe guard your system. This is especially true if you've upgraded the HiveOS image on your PXE server.

Keep in mind that any password changes on your rigs are local and therefore; only temporary.

(I haven't checked to see if rig.conf stores the rig password in plain text, but I wouldn't put it past the HiveOS team to send the rig password, via rig.conf, in plain text as well. But I can't say for certain being that I don't use rig.conf files.)

5) After you have changed your rig's password, make sure you clear the bash history on your rig by running history -c. This is important because, again, since you have to pass in your new password, in plain text, as an input argument for the hive-passwd command, this will mean that the bash history will retain a record of it. Running history -c will clear said bash history.

I wished that the HiveOS diskless PXE system would do this automatically whenever you have ran the hive-passwd command by default, but right now, as it stands, it doesn't.

Stay safe.

Happy mining!

 

 

Credits and acknowledgements:

Credit goes to @HaloGenius from the HiveOS forums for deleting my thread and comments about these security issues/flaws with the HiveOS diskless PXE system/setup where instead of, or rather than addressing these security concerns, you took it upon yourself to delete the thread and comments under the guise of this being fear, uncertainty, and doubt (FUD).

As a direct result of your actions there, you inspired me to revive my blog that I haven't touched in years, and turn this blog into me blogging about engineering and IT stuff such as this so I want to acknowledge you, thank you, and give you credit for inspiring me to write this so that people will know what kind of security issues/flaws exists with the HiveOS diskless PXE system, which is sold as a "feature" on the HiveOS features page.

Apparently, security flaws with HiveOS diskless PXE system/setup is a "feature" of HiveOS.

More Reasons Why HiveOS' Diskless PXE Boot Is Such A Massive, Critical Problem

Here are more reasons why the HiveOS diskless PXE boot is such a massive, critical problem:

Per the HiveOS team's own Telegram post:

They tell you:

"How to avoid this kind of attack:

- use secure passwords"

The problem with this, of course, is that if you are using HiveOS that was installed on a hard drive, SSD, or a USB flash drive (which, by the way, if you got hacked/hijacked like this, you couldn't simply just reboot to wipe out the stuff that the hacker/attacker has done), then sure, you can change the password to your heart's content when you first boot up and everything will be a little bit more secure from that point forward.

But if you are using HiveOS' diskless PXE, the hacker/attacker hacks/hijacks your system, you reboot it to reset everything, which sounds great in theory, until you realise that it also restores your system to the default password "1".

So much for having a more secure password.

In other words, if you ACTUALLY want to abide by the HiveOS team's recommendations of using a secure password, a reboot and/or a power cycle on your rig/worker/system will wipe out said secure password and restore it back to the default unsecure password. Once that happens, then a hacker/attacker can hack/hijack/attack your system again. Rinse. Repeat.

 

This flaw exists REGARDLESS of whether you have port 22 open on your modem/router.

Even if you secured your network, the fact this massive, critical flaw that still exists with the HiveOS diskless PXE system means that your network and your rig/worker/system(s) are still at risk.

In other words, you want to actually abide by the HiveOS team's recommendation. But literally a reboot and/or a power cycle on your rig later, and your diskless PXE rig/worker/system would be just as unsecure as it was before you tried to use a secure password, pursuant to said HiveOS team's recommendations on how to avoid this type of an attack.

And in talking to at least the HiveOS forum moderator(s) and administrator(s), they are CLEARLY in denial about this massive, critical security flaw.

Again, the two things that I can think of that would make this vastly more secure would be:

1) If you actually WANT to abide by the HiveOS team's recommendations of using a more secure password, said HiveOS team would have to modify the pxe-setup.sh so that it will ask you want you want the default rig password to be (so that it won't just automatically deploy "1" as the default password to all of your rigs), and then it will deploy all of your diskless PXE rigs with the default password that you specified (that isn't stored in some config file somewhere, in plaintext).

2) An even better option, in addition to pxe-setup.sh asking you what you want the default password to be, would be if it asked you if you wanted to disable password authentication for ssh completely in the first place.

That way, you would have to worry a lot less about:

"How to avoid this kind of attack:

- use secure passwords"

 

BOTH times that I've written about this in the HiveOS forums and BOTH times, the post and the comments were deleted in regards to this. IF the HiveOS Team ISN'T able to or aren't willing to put the time in to actually make their diskless PXE more secure GIVEN these GLARING, massive, and critical security flaws with it, then they shouldn't advertise said diskless PXE as a feature on their HiveOS features page. That way, said HiveOS team won't have to worry about this gaping security flaw anymore.

HiveOS Has A Massive, Critical Security Flaw With Their Diskless PXE Boot System

TL;DR:

Don't use HiveOS diskless PXE because there's a massive, critical security flaw with it.

 

The issue: 

 

For some inexplicable reason, the moderators and the forum administrators over at the HiveOS forum has decided to take it upon themselves to delete the topic (https://forum.hiveos.farm/t/massive-significant-substantial-pxe-security-risk/43187/1) that I had posted where I talk about the fact that the HiveOS diskless PXE boot system currently has a massive security flaw in it right now. (*edit/update 2021-07-14 18:08 UTC*: I've also written extensively about it in the HiveOS diskless PXE forum thread as well whereby the moderators and administrators of the HiveOS forums are also fully and completely in denial about this MASSIVE security flaw in the HiveOS diskless PXE deployment environment.)

As reported by RedPandaMining in his video, he talks about the current security flaw with HiveOS where if your system/worker/rig is open/exposed to the public internet and the port for ssh (port 22) is open as well, that hackers can then hack into your mining system, change the configuration, change the password and hijack your system to mine for said hacker. 

 

So, my thought was that if you run a diskless PXE setup/configuration/deployment, that shouldn't be a problem right? 

 

Well, yes, and no. 

 

(Note: For the purposes of this blog post, I am going to be focusing in on the security of the HiveOS diskless PXE deployment rather than the actual network security side of things. This isn't to say that one area is more important than any other. As such, you can google how to secure your home network as there are plenty of resources that talks about this. From my research, there is little, if anything, written about the security of HiveOS diskless PXE setup/deployment/system so this is the area that I am going to focus on specifically, in this blog post. In fact, there is so little that's written about this topic and that may have something to do with the fact that the HiveOS forum moderators and administrators will delete any thread or posts talking about this topic, which is why I am writing it on my own personal blog instead since it is quite obvious, this is an area that they don't want to talk about.)

 

See, the thing with a diskless PXE setup is that whilst yes, you will be able to "fix" the hack simply just by either issuing a reboot command to your mining rig/worker/system via HiveOS' online management interface, or, if you can't get to it because the hackers have hijacked your rig, you will have to be able to physically power cycle the system, then it will just go back to booting like it normally would off of your PXE server, and that would undo all of those changes that the hacker was able to implement.

But that's not the end of it.

The way the hacked worked, according to Simon Bell, it installs basically a loader script that will connect to the hacker/hijacker/attacker's web server address that will then reconfigure your system (mostly to lock you out and prevent itself from being removed) and then it will download the rest of the payload and supporting files that will then change your actual configuration files specific to mining/the miner itself.

So, with that, again, the thought process could be that if you have a diskless PXE system/deployment, all you would need to do is power cycle your rig/worker and that will restore everything back to its default and that would be the end of the hack/hijack.

But that's not the case.

The truth of the matter is that some of the recommendations that Simon Bell makes are to change your password, both through the HiveOS online management interface, and if that fails/doesn't work (as it happened to be in my case when I tried it), you can always ssh into your rig/worker/system and run the command:

hive-passwd

to change the password locally on the system and that way you can set a stronger, more secure password.

Another mitigating action that you can take is also to disable password authentication for ssh altogether and authenticate only using keys instead which would make it more secure.

And you can google the steps on how to do that or follow the link from Simon Bell's post in regards to the specific instructions on how to do that.

But here is the problem with both of those actions: the moment you power cycle and/or reboot your rig, and it goes through the diskless PXE boot process all over again, it will (re-)download the diskless PXE image (hiveramfs.tar.xz) from your PXE server and then start to unpack it and ALL of those changes that you've just made to secure your rig is now, once again, NOT secure again, as the defaults are restored.

I know because I accidentally tested this.

I tried to change the password for my rig/worker/system via the HiveOS online management interface and that didn't seem to work so that ssh into my rig/worker/system, and ran:

hive-passwd

and changed my password to something more secure. I log out, and when I tried to log back in, it had failed. Maybe my password was too secure to transmit over ssh? Who knows.

Either way, I couldn't log back into my own system again.

 

So, what do I do to "fix" this?

 

I power cycle the system and as I mentioned above, it resumed the normal PXE booting process, downloaded the hiveramfs.tar.xz file from my PXE server and proceeded to boot up the rig/worker/system like normal.

But when went to log back into the system, instead of it being the password that I had changed it to, it reverted back to the default password.

That's a HUGE problem.

The default password is built into the hiveramfs.tar.xz image that it creates when you run the pxe-setup.sh initially PXE configuration script as well as the hive-upgrade.sh script.

This means that by default, the diskless PXE boot image is NOT secure because if you change the password and/or if you disable password authentication for ssh, all you would have to do to undo those changes is just to simply reboot the rig/worker/system.

And if your rig/worker/system is still wide open to the public internet and port 22 for ssh is still open on your modem/router, then the attacker/hijacker can attack your rig/worker/system repeatedly even if you were using a diskless PXE setup/server/deployment.

What's worse - if the attacker was more sophisticated, they can actually hack and hijack your rig/worker/system to a point where it will try and see if you are using/running a diskless PXE system/server, find out the IP address of the PXE server, and then try and implement the changes in the hiveramfs.tar.xz file directly so that even if you tried to reboot your diskless PXE rig/worker/system to try and "restore to default", it would be downloading a hacked/hijacked boot image where the hack/hijack would then be built into the hiveramfs.tar.xz image and then this would mean that your HiveOS diskless PXE server is toast (along with your rig).

So....where do we go from here?

 

Well, for starters, the HiveOS development team should change/modify the pxe-setup.sh script  such that it asks you what you want the default rig/worker password to be rather than just cascading or setting up the default password for everything, and making/leaving it all to be the same (and that it doesn't change unless you either change it via the HiveOS online management interface and/or by ssh'ing into your system individually and changing them one at a time).

Secondly, I think that the HiveOS development team should also ask you whether you want to disable password authentication for ssh by default when run pxe-setup.sh script in order to make it more secure and therefore; you DEFINITELY won't have this default user/default password problem that hackers/attackers can take advantage of.

Again, at the per rig level, if you are running HiveOS on a disk of some sort, then these actions can go a long way to help mitigating this type of an attack. However, if you're running a diskless PXE setup, then these mitigations will be erased/cleared out/reset the moment you reboot the system because the inherent hiveramfs.tar.xz file will NOT have these changes implemented and for that to happen, the pxe-setup.sh deployment script will need to be modified by the HiveOS team so that these mitigations will be implemented by default, so that even if you reboot the system, it won't wipe out these security mitigations and worse, put your system back into a state where it can be attacked again.