Since I built my Ryzen 5950X system, and the 12900K system, and then had to completely disassemble the 12900K system, and then built another Ryzen 5950X system whilst arguing with Asus, I was in the middle of a data consolidation effort for all of my engineering data from the various projects that I've worked on over the years.
Today marks the day where the first pass of this data consolidation effort has completed and I ended up saving almost 14 TB of storage space.
It feels nice, and I get a sense of accomplishment as the data is being written to tape right now.
I can't believe that it's taken me like about 6 months to finish this data consolidation effort.
At some points during the process of unpacking, packing, and then re-packing the data, both Ryzen 5950Xs and also the Intel Core i7-4930K that's in the headnode was oversubscribed 3:1 when it was processing the data. That just seems pretty crazy to me because that's also a little bit of an indication as to how much work the CPUs had to do to process and re-process the data.
Not to mention, my poor, poor hard drives, that have been working so hard throughout all of this.
15 June 2022
Engineering data consolidation efforts
13 June 2022
Welp....this is a problem.
Let me begin with the problem statement:
What you see above is the results from the 100 Gbps Infiniband network bandwidth test that are between my two AMD Ryzen 5950X systems. Both of them has a discrete GPU in the primary PCIe slot, and then the Mellanox ConnectX-4 dual port, 100 Gbps Infiniband NIC is in the next available PCIe slot.
I can't really tell from the motherboard manual for the Asus ROG Strix X570-E Gaming WiFi II motherboard what speed the second PCIe slot is supposed to be when there is a discrete GPU plugged into the primary PCIe slot.
The Mellanox ConnectX-4 card is a PCIe 3.0 x16 card, which means that the slot itself is supposed to support upto 128 Gbps (and the ports themselves is supposed to go up to a maximum of 100 Gbps out of the 128 Gbps that's theorectically available). If the slots were running as PCIe 3.0 x4, it should be capable of 32 Gbps.
As the results show, clearly, that is not the case.
I'll have to see if I can run both of those systems without the discrete GPU, so that I can plug the Mellanox cards into the primary PCIe slot.
*Update 2022-06-14*:
So I took out the discrete GPUs from both systems and put the Mellanox card into the primary PCIe slot and this is what I get from the bandwidth test results:
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x0c QPN 0x008c PSN 0x5ccdd5
remote address: LID 0x05 QPN 0x010a PSN 0x178491
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 100000 0.000000 0.066552 4.159479
4 100000 0.00 0.11 3.529205
8 100000 0.00 0.27 4.225857
16 100000 0.00 0.54 4.254547
32 100000 0.00 1.09 4.254549
64 100000 0.00 2.19 4.276291
128 100000 0.00 4.51 4.408332
256 100000 0.00 9.21 4.498839
512 100000 0.00 18.60 4.540925
1024 100000 0.00 36.74 4.485289
2048 100000 0.00 75.76 4.623960
4096 100000 0.00 96.55 2.946372
8192 100000 0.00 96.57 1.473530
16384 100000 0.00 96.58 0.736823
32768 100000 0.00 96.58 0.368421
65536 100000 0.00 96.58 0.184218
131072 100000 0.00 96.58 0.092109
262144 100000 0.00 96.58 0.046055
524288 100000 0.00 96.58 0.023027
1048576 100000 0.00 96.58 0.011514
2097152 100000 0.00 96.58 0.005757
4194304 100000 0.00 96.58 0.002878
8388608 100000 0.00 96.58 0.001439
---------------------------------------------------------------------------------------
Ahhhh.....much better. That's more like it.
23 February 2022
Vastly differing results in WSL2 between 5950X and 12900K in Windows 10 21H2
A little while ago, I came across this video which was talking about how you can run Linux graphical applications natively in Windows (more specifically, in Windows 11).
However, when at the time when I watched said original video, I didn't have any hardware that could actually really run that probably. My "newest" system that I had was an Intel Core i7-6700K and as far as I know, it didn't have the Trusted Platform Module (TPM) anywhere (whether it is as an external add-on dongle) or integrated into the motherboard firmware/BIOS.
So, I didn't really make much of it back then.
But since then, I've built both my AMD Ryzen 9 5950X system and also my Intel Core i9-12900K system and I figured that with some of the work that I needed the systems to be doing over with, I had a little bit of time with the system to do some more testing with it.
So I grabbed two extra HGST 1 TB SATA 6 Gbps 7200 rpm HDDs (one per system), threw Windows 10 21H2 on it, and proceeded with the instructions on how to install and configure Windows Subsytem for Linux 2 (WSL2). I installed Ubuntu 20.04 LTS (which really, turned out to be 20.04.4 LTS), and proceeded to try and install the graphical layer/elements to it.
So that's all fine and dandy. (Well, not really because in both instances, neither of the systems was able to start the display and I can't tell if it is because I have older video cards in the system (Nvidia GeForce GTX 980 and a GTX 660 respectively - because as a CentOS 7.7.1908 system, it didn't really matter what I had in there since I was going to remote in over VNC anyways).)
But, since I had it installed, AND by some miracle, Windows 10 picked up on the Mellanox ConnectX-4 dual port VPI 100 Gbps Infiniband cards automatically, I just had to manually give each card in each system an IPv4 address so that it can talk to my cluster headnode (which was still running CentOS along with the OpenSM), and connect up to the network shares that I had set up. (SELinux is a PITA. But I got Samba going on said CentOS system so that on the Linux side, it can connect up to the RAID arrays using NFS-over-RDMA whilst in Windows, it's just through "normal" Samba (i.e. NOT SMB Direct).)
So, I might as well benchmark the systems to see how fast it would be able to write and read a 1024*1024*10240 byte file.
And for fun, I also installed Cygwin on both of the systems as well, so that I can compare the two together.
Being that both systems was able to pick up the Mellanox ConnectX-4 card right away (I didn't have to do anything special, install the Mellanox drivers, etc.), I was able to connect up to my cluster headnode and the Samba shares were visible immediately. As a result of that, I was able to right-click on both of those shared folders and map it to a network drive directly and automatically.
Now, in WSL2, I had to mount the mapped network drive using the command:
$ sudo mount -t drvfs V: /mnt/V
(Source: https://superuser.com/questions/1128634/how-to-access-mounted-network-drive-on-windows-linux-subsystem)
And then once that was done, I was able to run the follow commands in both Ubuntu on WSL2 and also in Cygwin:
Write test:
$ time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240
Read test:
$ time -p dd if=10Gfile of=/dev/null bs=1024k
Here are the results:
 |
| Huh. Interrresting. |
But what is interesting though is that the speeds are close, with the 5950X being a little bit faster under Cygwin than the 12900K, also under Cygwin.
I decided to blog about this because there is a potential possibility that for those that might be working with WSL2, the hardware that you pick MAY have an adverse performance impact.
I'm not sure who, if anybody, has done a cross-platform comparison like this before but to be honest, I haven't really bothered to look for it either because you might have reasonably expected that this significant performance difference wouldn't/doesn't exist, but the results clearly show that there's a difference. And a rather significant difference in performance at that.
Please be aware and you should do your own testing for your workload/case/circumstance if you get a chance to be able to do so.
06 January 2022
Getting the latest and greatest hardware running in Linux is sometimes, a bit of a nightmare
Just prior to the holidays, I decided to upgrade one of three of my systems and consolidate it down to two. My old Supermicro Big Twin^2 Pro micro cluster server and two HP Z420 workstations (that I was using in lieu of the Supermicro because the Supermicro was "too loud") were getting replaced by an AMD system, built on the Ryzen 9 5950X CPU and an Intel system, built on the latest and greatest that Intel had to offer - the Core i9-12900K.
So, I speced out all of the rest of the hardware, which really, consisted of the motherboard, RAM, and the CPU heatsink and fan assembly whilst I was able to reuse some of my older, existing components as well. (I did have to buy an extra power supply though because I had originally miscalculated how many power supplies that I would need.)
So that's all fine and dandy. All of the hardware arrived just before the start of the Christmas break for me, so I started to set up the AMD system. Install the CPU, the RAM, the CPU HSF, plug everything in, check and double check all of the connections - everything is good to go. I used Rufus USB to write the CentOS 7.7.1908 installed onto a USB drive, plug in the keyboard, mouse, and flip the switch on the power supply and off I go right?
[buzzer]
Nope!
 |
| Near instant kernel panic. Nice. |
As you can see from the picture above, less than 3 seconds into the boot sequence from the USB drive - Linux has a kernel panic.
Great.
So now I get the "fun" [/sarcasm] job of trying to sort this kernel panic out. Try it a few more times, the same thing happens.
So, ok. Now I'm thinking that the hardware is too new for this older Linux distro and version (and kernel). So, I take out my Intel Core i7-3930K system (one of them that I use to run my tape backup system), and I plug the hard drive into that system, along with the video card back in, and run through the boot and installation process (which worked without any issues of course), power down the 3930K, take the hard drive back out, and plug it into the 5950X system. Power it on. (I set the BIOS to power on after AC loss so that I can turn on the system even when it isn't inside a case and I don't have a power button connected to it.)
The official CentOS forums state that they only support CentOS 7.9.2009, so I try that as well, still to no avail.
Eventually, I end up using a spare Intel 545 series 512 GB SATA 6 Gbps SSD that I had laying around so that I could try installing and re-installing, trying different drivers, kernel modules, kernels, etc. a LOT faster than I was able to with a 7,200 rpm HDD.
End net result: I filed a bug report with kernel.org because the mainline kernel 5.15.11 kept producing kernel panics with the Mellanox 100 Gbps Infiniband network card installed. And it didn't matter whether I tried to use the "inbox" CentOS Infiniband drivers or the "official" Mellanox OFED Infiniband drivers.
 |
| Yet another Linux kernel panic. |
Interestingly enough, the mainline kernel 5.14.15 works with the Infiniband NIC just fine. So that's what I landed on/with.
The other major problem that I ran into was that the Asus X570 TUF Gaming Pro (WiFi) used the Intel I225-V 2.5 GbE NIC. Unbeknownst to me when I originally purchased the motherboard, I didn't realise that Intel does NOT have a Linux driver (even on Intel's website) for said Intel I225-V 2.5 GbE NIC. And what was weird was that when I migrating the SSD over during the testing and trying to find/figure out a configuration that worked, said Intel onboard 2.5 GbE NIC would work initially, but then it would eventually and periodically drop out and so that was quite the puzzle because if there wasn't a driver for it, then how was it that it was able to work when I moved the drive over?
As a result of that, that took up a couple of days where I would be trying to clone the disk image over from the Intel SSD over onto the HGST HDD using dd and in the end, that didn't work either.
So, what did I end up with?
This is the hardware specs that I ended up with on the AMD system:
CPU: AMD Ryzen 9 5950X (16-core, 3.4 GHz stock base clock, 4.9 GHz max boost clock, SMT enabled)
Motherboard: Asus X570 TUF Gaming Pro (WiFi)
RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)
CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan
Video card: EVGA GeForce GTX 980
Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD
NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)
NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)
Power Supply: Corsair CX750M
OS: CentOS 7.7.1908 kernel 5.14.15-1-el7.elrepo.x86_64
I ended up adding the Intel Gigabit CT Desktop NIC because a) it was an extra Intel GbE NIC AIC that I had also laying around, and b) it proved to be able to provide a vastly more reliable connection than the onboard Intel I225-V 2.5 GbE due to the driver issue.
Now that I have the system set up and running, there is a higher probabilty that the igc kernel module probably works more reliably now than it did when I was originally setting up the system, but being that it was not reliable when I was doing the initial setup and testing, I am less likely to use said onboard NIC, which is a pity. Brand spankin' new motherboard and I can't even use nor trust the reliability of the onboard NIC. And I can't even blame Asus for it because it is an Intel NIC. (Sidebar: Ironically, the Asus Z690 Prime-P D4 motherboard that I also purchased uses a Realtek RTL8125 2.5 GbE NIC, which I WAS able to find a driver for that and it has been working flawlessly with it.)
That took probably on the order of around 10 days, from beginning to end, to get the AMD system up and running.
The Intel system was a little bit easier to set up.
The kernel panic issue with the mainline 5.15.11 kernel and Infiniband was also present on the Intel platform as well.
Interestingly and ironically enough, the newer kernel kept crashing or had severe stability issues. It turns out that I did NOT install the RAM correctly (i.e. in the DIMM_A2 and DIMM_B2 slots), so since then, I've corrected that.
Keen readers might note that I have stated that I have 4 sticks of RAM, except that one of the sticks arrived DOA, and is currently being sent back to Crucial under RMA, so when it comes back, then I will be able to install the extra stick that is currently not installed and the stick that is due back from the RMA exchange.
I might try the newer kernels again later, but for now, at least the system is up and running so that I can start making it do the work that I need it to be doing.
The system stability issues due to the error that I made when installing (and uninstalling) the RAM (because I was testing the stick of RAM that wouldn't POST that ended up getting RMA'd back to Crucial), I ended up with a RAM installation configuration that wasn't correct and the resulting system stability issues ate up a few more days.
So, in the end, it took me almost the entire Christmas holiday to get both of these systems up and running.
(This is also a really good reason why traditionally, I have stuck with workstation and server hardware because on my old Supermicro micro cluster, I can deploy all four nodes in 2 hours or less. It's a pity that the system is too loud.)
This is the hardware that I ended up with on the Intel system:
CPU: Intel Core i9-12900K (16 cores (8P + 8E), 3.2 GHz/2.4 GHz base clock speed, 5.2 GHz/3.9 GHz max boost clock, HTT enabled)
Motherboard: Asus Z690 Prime-P D4
RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)
CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan
Video card: EVGA GeForce GTX 660
Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD
NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)
NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)
Power Supply: Corsair CX750M
OS: CentOS 7.7.1908 kernel 3.10.0-1127.el7.x86_64
30 July 2021
Mellanox Lies (and Why It's Important To Speak Up)
 |
| Page 4 from the Mellanox IB Adapter Product Brochure |
But when I actually purchased said Mellanox ConnectX-4 dual 100 Gbps VPI port cards (MCX456A-ECAT), and I tried to use Mellanox's OFED drivers for Linux (MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64.iso), said NFSoRDMA wasn't working, so I posted on the Mellanox community forums to try and get some help (because sometimes, it can very well be that either I didn't read nor execute the installation instructions correctly or that I am not understanding something or that there was another step that I needed to do that wasn't documented in the installation instructions).
After fiddling around with it for a little while, I ended up reverting back to the "inbox" driver, i.e. the driver that ships with the OS (originally in my case, CentOS 7.6, and then I eventually moved up to CentOS 7.7.1908), because for some strange reason, NFSoRDMA was working with the "inbox" driver, but not the Mellanox OFED driver.
Turns out, the NFSoRDMA feature was disabled in that version of the Linux driver (and may have been actually disabled in some versions prior to that as well).
|
Oh really? |
 |
| Source: https://docs.mellanox.com/display/MLNXOFEDv541030/General+Support |