Paradoxic connundrum...

I bought a new Core i7-13700K and installed into my Asus Z690 Prime P that I bought about a year ago.

Question: How do you update the BIOS for a system that won't POST (so that it would be able to recognise said new processor)?

Moral of the story: Do NOT buy from Asus. Intel is willing to offer a refund. Asus is not.

 As a follow-up to my previous blog post about the data corruption issue that I was experiencing with the Intel Core i9-12900K processor that was running on the Asus Z690 Prime-P D4 motherboard, Intel has offered a full refund on the defective unit whilst Asus has not.

So, moral of the story:

Don't buy from Asus.

I mean, clearly, if the interaction between the Intel Core i9-12900K and the Asus Z690 Prime-P D4 motherboard is causing the system to spontaneously reset itself when I attempted to run memtest86 a second time, using the memory that was from my AMD Ryzen 9 5950X (which was also using an Asus motherboard), which PASSED memtest86 on said Ryzen platform, and by putting those four DIMMs into the Asus Z690 Prime-P D4 motherboard, it results in the system spontaneously resetting itself; that's NOT a good sign of a reliable motherboard.

Asus was ONLY willing offer a RMA repair, and I told them that the CPU is in the process of being sent back, so even if they attempted to repair it, I would have no way of verifying whether the issue is still there or not because the CPU would've already been sent back and I'm not buying another Alder Lake CPU from Intel only to give it the chance for this problem to repeat itself.

So, moral of the story:

Don't buy from Asus.

Vastly differing results in WSL2 between 5950X and 12900K in Windows 10 21H2

A little while ago, I came across this video which was talking about how you can run Linux graphical applications natively in Windows (more specifically, in Windows 11).

However, when at the time when I watched said original video, I didn't have any hardware that could actually really run that probably. My "newest" system that I had was an Intel Core i7-6700K and as far as I know, it didn't have the Trusted Platform Module (TPM) anywhere (whether it is as an external add-on dongle) or integrated into the motherboard firmware/BIOS.

So, I didn't really make much of it back then.

But since then, I've built both my AMD Ryzen 9 5950X system and also my Intel Core i9-12900K system and I figured that with some of the work that I needed the systems to be doing over with, I had a little bit of time with the system to do some more testing with it.

So I grabbed two extra HGST 1 TB SATA 6 Gbps 7200 rpm HDDs (one per system), threw Windows 10 21H2 on it, and proceeded with the instructions on how to install and configure Windows Subsytem for Linux 2 (WSL2). I installed Ubuntu 20.04 LTS (which really, turned out to be 20.04.4 LTS), and proceeded to try and install the graphical layer/elements to it.

So that's all fine and dandy. (Well, not really because in both instances, neither of the systems was able to start the display and I can't tell if it is because I have older video cards in the system (Nvidia GeForce GTX 980 and a GTX 660 respectively - because as a CentOS 7.7.1908 system, it didn't really matter what I had in there since I was going to remote in over VNC anyways).)

But, since I had it installed, AND by some miracle, Windows 10 picked up on the Mellanox ConnectX-4 dual port VPI 100 Gbps Infiniband cards automatically, I just had to manually give each card in each system an IPv4 address so that it can talk to my cluster headnode (which was still running CentOS along with the OpenSM), and connect up to the network shares that I had set up. (SELinux is a PITA. But I got Samba going on said CentOS system so that on the Linux side, it can connect up to the RAID arrays using NFS-over-RDMA whilst in Windows, it's just through "normal" Samba (i.e. NOT SMB Direct).)

So, I might as well benchmark the systems to see how fast it would be able to write and read a 1024*1024*10240 byte file.

And for fun, I also installed Cygwin on both of the systems as well, so that I can compare the two together.

Being that both systems was able to pick up the Mellanox ConnectX-4 card right away (I didn't have to do anything special, install the Mellanox drivers, etc.), I was able to connect up to my cluster headnode and the Samba shares were visible immediately. As a result of that, I was able to right-click on both of those shared folders and map it to a network drive directly and automatically.

Now, in WSL2, I had to mount the mapped network drive using the command:

$ sudo mount -t drvfs V: /mnt/V

(Source: https://superuser.com/questions/1128634/how-to-access-mounted-network-drive-on-windows-linux-subsystem)

And then once that was done, I was able to run the follow commands in both Ubuntu on WSL2 and also in Cygwin:

Write test:
$ time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240

Read test:
$ time -p dd if=10Gfile of=/dev/null bs=1024k

Here are the results:

Huh. Interrresting.

I have absolutely NO clue why WSL2 on the 5950X is so much slower compared to WSL2 on the 12900K.

But what is interesting though is that the speeds are close, with the 5950X being a little bit faster under Cygwin than the 12900K, also under Cygwin.

I decided to blog about this because there is a potential possibility that for those that might be working with WSL2, the hardware that you pick MAY have an adverse performance impact.

I'm not sure who, if anybody, has done a cross-platform comparison like this before but to be honest, I haven't really bothered to look for it either because you might have reasonably expected that this significant performance difference wouldn't/doesn't exist, but the results clearly show that there's a difference. And a rather significant difference in performance at that.

Please be aware and you should do your own testing for your workload/case/circumstance if you get a chance to be able to do so.

Why is Intel keeping the overall physical dimensions of their Intel 670p Series 2 TB SSD a secret?

I recently submitted my order for a Minis Forum HX90 (specs) and being that I am looking to use it to replace my very hot Intel NUC that I had previously written about (it's back up to 100 C nominal now), and that I might also be offload all of the virtualisation duties as well from my Intel Core i7-6700K system and onto this new system instead. As such, I didn't know if said new system would support RAID0 with my two existing Samsung EVO 850 1 TB SATA 6 Gbps SSDs that are no longer currently deployed in a system, so I figured that I was going to get a 2 TB NVMe SSD just to be safe and I landed on this - an Intel 670p Series 2 TB NVMe 3.0 x4 SSD (specs).

Whilst browsing through YouTube, I stumbled my way upon a video where they were talking about NVMe SSD and putting heatsinks on them and how they would thermal throttle the performance if said NVMe SSD got too hot whilst it was being used/under load.

So, that got me thinking - should I start looking and seeing if I should be getting a NVMe SSD heatsink of my own for this drive?

So, I reached out to the customer support at Minis Forum (based out of Hong Kong, which is interesting because their first email back to me was written entirely in Traditional Chinese), so I asked them about a SSD heatsink (because some of the review units that they've sent to other tech YouTubers included a NVMe SSD with a heatsink pre-installed in the system) and they told me that the total height that the HX90 can take, INCLUDING the NVMe SSD is 7 mm.

So, ok. No problems, right? If I can find out what's the overall height of the Intel 670p Series 2 TB NVMe 3.0 x4 SSD, then I can figure out what's the maximum height of a heatsink the HX90 can accept, and then I can start to look into what are my purchasing options.

So, then I reached out to Intel's customer support, because of course, lo and behold, the overall height of the Intel 670p Series 2 TB NVMe 3.0 x4 SSD isn't listed on their spec page.

Huh. No overall physical dimensions listed on Intel's website.

So I reached out to Intel's customer service and asked them this basic question and also told them that it was because the manufacturer of the computer has told me what the maximum height of the combined SSD and heatsink can be so that I can properly size and purchase said heatsink. Their customer service rep said that they understand why I was asking for this information and would need to do further research on this topic/matter and that they would get back to me. Okay. Not a big deal.

Well earlier today, I got an email from said customer service rep stating quote:

Why would Intel keep the overall physical dimensions of their product under a NDA?

So, at this point, it seemed awfully suspicious.

I told them that I am not asking on behalf of the company where I work, and therefore; I have no idea if they have a signed NDA with Intel or not. (And frankly, that shouldn't matter because a customer should be able to ask for the overall physical dimensions of their product (and not the overall dimensions of the box/packaging that their product gets shipped in either).)

I then told them that I will just measure my drive when it arrives and that as such, I will not be signing a NDA in regards to this.

Well, about 3 hours later, my drive arrived.

So, for those that are interested in knowing, the overall physical dimensions of the Intel 670p Series 2 TB NVMe 3.0 x4 SSD are:

Overall length: 80.12 mm
Overall width: 22.05 mm
Overall height: 2.0525 mm (average of 2.09 mm, 2.06 mm, 1.97 mm, and 2.09 mm)

So, in case you're out trying to shop for a NVMe heatsink, and you're trying to use it for a small form factor (SFF) or ultra compact form factor (UCFF) build, now you know the height of the NVMe heatsink you can get.

A friendly reminder to periodically clean your NUC

I have an Intel BOXNUC8i7BEH (specs) and I have been using it to run a VM and also as a host system/unit.

Lately, it's been having issues where even when I tried to run it without the chassis (i.e. running it in an "open case" configuration, the temps were still hitting a peak of 100 C whilst downloading something in the VM and also with 12 Firefox tabs open on the host itself.

So, given that it was still running so hot, even with it running out of the case/in the "open case" configuration, I figured that I would shut the unit down, wait for it to cool off a bit, and proceed with the further disassembling the unit.

Once I took the fan off, there was a LOT of dust that had been trapped where the inlet to the copper heatsink was, so I was able to clean that off with damp tissue paper.

And I also figured that since I had some Thermal Grizzly's Kyronaut sitting around, that I might as well also remove the plate that the heatpipes are connected to, clean off the old thermal paste that's on the CPU, and give it some new thermal paste whilst I'm at it.

Lo and below, the current system, still doing exactly what it was doing before (picking up from where it left off when I powered down the system) is now sitting at a cooler 85 C or so.

Yay!

Moral of the story: remember to periodically clean your NUC!

Getting the latest and greatest hardware running in Linux is sometimes, a bit of a nightmare

Just prior to the holidays, I decided to upgrade one of three of my systems and consolidate it down to two. My old Supermicro Big Twin^2 Pro micro cluster server and two HP Z420 workstations (that I was using in lieu of the Supermicro because the Supermicro was "too loud") were getting replaced by an AMD system, built on the Ryzen 9 5950X CPU and an Intel system, built on the latest and greatest that Intel had to offer - the Core i9-12900K.

So, I speced out all of the rest of the hardware, which really, consisted of the motherboard, RAM, and the CPU heatsink and fan assembly whilst I was able to reuse some of my older, existing components as well. (I did have to buy an extra power supply though because I had originally miscalculated how many power supplies that I would need.)

So that's all fine and dandy. All of the hardware arrived just before the start of the Christmas break for me, so I started to set up the AMD system. Install the CPU, the RAM, the CPU HSF, plug everything in, check and double check all of the connections - everything is good to go. I used Rufus USB to write the CentOS 7.7.1908 installed onto a USB drive, plug in the keyboard, mouse, and flip the switch on the power supply and off I go right?

[buzzer]

Nope!

Near instant kernel panic. Nice.

 

As you can see from the picture above, less than 3 seconds into the boot sequence from the USB drive - Linux has a kernel panic.

Great.

So now I get the "fun" [/sarcasm] job of trying to sort this kernel panic out. Try it a few more times, the same thing happens.

So, ok. Now I'm thinking that the hardware is too new for this older Linux distro and version (and kernel). So, I take out my Intel Core i7-3930K system (one of them that I use to run my tape backup system), and I plug the hard drive into that system, along with the video card back in, and run through the boot and installation process (which worked without any issues of course), power down the 3930K, take the hard drive back out, and plug it into the 5950X system. Power it on. (I set the BIOS to power on after AC loss so that I can turn on the system even when it isn't inside a case and I don't have a power button connected to it.)

The official CentOS forums state that they only support CentOS 7.9.2009, so I try that as well, still to no avail.

Eventually, I end up using a spare Intel 545 series 512 GB SATA 6 Gbps SSD that I had laying around so that I could try installing and re-installing, trying different drivers, kernel modules, kernels, etc. a LOT faster than I was able to with a 7,200 rpm HDD.

End net result: I filed a bug report with kernel.org because the mainline kernel 5.15.11 kept producing kernel panics with the Mellanox 100 Gbps Infiniband network card installed. And it didn't matter whether I tried to use the "inbox" CentOS Infiniband drivers or the "official" Mellanox OFED Infiniband drivers.

Yet another Linux kernel panic.

Interestingly enough, the mainline kernel 5.14.15 works with the Infiniband NIC just fine. So that's what I landed on/with. 

The other major problem that I ran into was that the Asus X570 TUF Gaming Pro (WiFi) used the Intel I225-V 2.5 GbE NIC. Unbeknownst to me when I originally purchased the motherboard, I didn't realise that Intel does NOT have a Linux driver (even on Intel's website) for said Intel I225-V 2.5 GbE NIC. And what was weird was that when I migrating the SSD over during the testing and trying to find/figure out a configuration that worked, said Intel onboard 2.5 GbE NIC would work initially, but then it would eventually and periodically drop out and so that was quite the puzzle because if there wasn't a driver for it, then how was it that it was able to work when I moved the drive over?

As a result of that, that took up a couple of days where I would be trying to clone the disk image over from the Intel SSD over onto the HGST HDD using dd and in the end, that didn't work either.

So, what did I end up with?

This is the hardware specs that I ended up with on the AMD system:

CPU: AMD Ryzen 9 5950X (16-core, 3.4 GHz stock base clock, 4.9 GHz max boost clock, SMT enabled)

Motherboard: Asus X570 TUF Gaming Pro (WiFi)

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 980

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 5.14.15-1-el7.elrepo.x86_64

 

I ended up adding the Intel Gigabit CT Desktop NIC because a) it was an extra Intel GbE NIC AIC that I had also laying around, and b) it proved to be able to provide a vastly more reliable connection than the onboard Intel I225-V 2.5 GbE due to the driver issue.

Now that I have the system set up and running, there is a  higher probabilty that the igc kernel module probably works more reliably now than it did when I was originally setting up the system, but being that it was not reliable when I was doing the initial setup and testing, I am less likely to use said onboard NIC, which is a pity. Brand spankin' new motherboard and I can't even use nor trust the reliability of the onboard NIC. And I can't even blame Asus for it because it is an Intel NIC. (Sidebar: Ironically, the Asus Z690 Prime-P D4 motherboard that I also purchased uses a Realtek RTL8125 2.5 GbE NIC, which I WAS able to find a driver for that and it has been working flawlessly with it.)

That took probably on the order of around 10 days, from beginning to end, to get the AMD system up and running.

The Intel system was a little bit easier to set up.

The kernel panic issue with the mainline 5.15.11 kernel and Infiniband was also present on the Intel platform as well.

Interestingly and ironically enough, the newer kernel kept crashing or had severe stability issues. It turns out that I did NOT install the RAM correctly (i.e. in the DIMM_A2 and DIMM_B2 slots), so since then, I've corrected that.

Keen readers might note that I have stated that I have 4 sticks of RAM, except that one of the sticks arrived DOA, and is currently being sent back to Crucial under RMA, so when it comes back, then I will be able to install the extra stick that is currently not installed and the stick that is due back from the RMA exchange.

I might try the newer kernels again later, but for now, at least the system is up and running so that I can start making it do the work that I need it to be doing.

The system stability issues due to the error that I made when installing (and uninstalling) the RAM (because I was testing the stick of RAM that wouldn't POST that ended up getting RMA'd back to Crucial), I ended up with a RAM installation configuration that wasn't correct and the resulting system stability issues ate up a few more days.

So, in the end, it took me almost the entire Christmas holiday to get both of these systems up and running.

(This is also a really good reason why traditionally, I have stuck with workstation and server hardware because on my old Supermicro micro cluster, I can deploy all four nodes in 2 hours or less. It's a pity that the system is too loud.)

This is the hardware that I ended up with on the Intel system:

CPU: Intel Core i9-12900K (16 cores (8P + 8E), 3.2 GHz/2.4 GHz base clock speed, 5.2 GHz/3.9 GHz max boost clock, HTT enabled)

Motherboard: Asus Z690 Prime-P D4

RAM: 4x Crucial 32 GB DDR4-3200 unbuffered, non-ECC RAM CL22 (128 GB total)

CPU HSF: Noctua NH-D15 with one stock 140 mm fan, and one NF-A14 industrialPPC 3000 PWM fan

Video card: EVGA GeForce GTX 660

Hard drive: 1x HGST 1 TB SATA 6 Gbps 7200 rpm HDD

NIC: Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband (MCX456A-ECAT)

NIC: Intel Gigabit CT Desktop 1 GbE NIC (Intel 82574L chipset)

Power Supply: Corsair CX750M

OS: CentOS 7.7.1908 kernel 3.10.0-1127.el7.x86_64