2022年2月12日 星期六

Incompatibilities and Compatibilities

NOTE: This article will be updated in the future when more compatibilities / incompatibilities are discovered. 

Incompatibilities

 

12-Feb-2022

  • OPNSense (and potentially pfSense as well) CARP protocol is not compatible with Arista (7060CX-32S, and potentially other models) switches.
  • [13-Feb-2022] OPNSense (and potentially pfSense as well) CARP protocol is not compatible with Windows Server 2022 Virtual Switches with SR-IOV enabled. Workaround is to NOT select SR-IOV option when creating the virtual switch.
  • Windows Server 2022 (Host) with Windows 11 Guest VM: Nested Virtualization is not compatible with Discreet Device Assignment (DDA) and SR-IOV. Once both are enabled, all passed-through devices (including SR-IOV virtual functions) will fail to be started with the same reason "Insufficient System Resources".
  • RME PCI-E Sounds Cards such as HDSPe RayDAT and HDSPe AIO are not compatible with Windows Discreet Device Assignment (DDA) and potentially all other device pass-through technologies. The device drivers can be installed on guest OS but it will not yield any sound.
  • Datapath VisionAV PCI-E Capture Cards are not compatible with Windows Discreet Device Assignment (DDA) and potentially all other device pass-through technologies. The device drivers cannot be started properly in guest OS.

2020年9月23日 星期三

易有大恒,是生兩檥:是 Vacuum 還是 Plenum?

或者, 其實兩者皆是。

中國上古的智慧其實也已經透露了一些端倪,不過這些一直被人們認爲生澀難懂或者帶有迷信色彩的道家學説很多時候都沒有跟科學聯係在一起。

西方科學的進步推翻了以前很多“被公認正確的假設”,例如,黑洞被證實不是傳説中的時空隧道,原子并不是“空”的。而最有趣的是量子學的進步,令人們對於物質的定義有了更深層次的認知。

物質就是能量,粒子就是波。這些都是 Duality,也都是“兩檥”的表現。

我們常説的真空 (Vacuum),真的是“空”的麽?假設一個空間,沒有任何的粒子,處於絕對零度,因此所有的活動都是靜止的。這個時候,我們看到的是真空,還只是沒有活動的“空間”?

就如同一個平靜的湖面,空無一物,但是湖的確是存在的。然後我們往胡裏扔一塊石頭,這個時候,湖就有了波瀾,我們就可以觀察到這些波瀾,這個時候,湖就不空了。但是不論是那種狀態,湖還是湖,一直都在。

把湖從二位面衍生到我們這個三維空間——空間一直都在。而我們現在能夠觀察到的,歸根究底,其實也只是這個空間的“波動”而已。

空間,就是“大恒”(太極)。

如果有人問,宇宙的年齡是多少,科學家能夠推算到的,也只是基於 Big Bang 這塊“石頭”落入我們這個“空間”所造成的波瀾的當前狀況。不過,“空間”存在了多久?以及“誰”投入這塊“石頭”?這些卻是我比較感興趣的問題。

2020年9月21日 星期一

超微 Supermicro X11DPi-NT 主板的注意事項

Supermicro X11DPi-NT / X11DPi-N 這款主板堪稱接近完美。它基本上把從 Xeon CPU 中可用的 PCI Express 鏈路都用上了,并且 CPU 和 CPU 之間也是使用 3 條 QPI 連接上的(這個是要點,有的主板只有兩條 QPI 的)。

但是!

如果你打算使用 X11DPi-NT 用作 VROC 用途,那麽你就要注意了。這款主板支持 Intel VROC,不過有以下的限制:

1、除了主板自帶的兩個 OCuLink 端口,就只有 Slot 6 這根插槽具有 VROC 功能。

2、必須把 Slot 6 設置成爲 x4x4x4x4 的 Bifurcation 模式才能夠啓用該插槽的 VMD 功能。這就是說我沒辦法使用 PCI Express Switch (PLX / PEX) 以鏈路更多的 NVME SSD。

3、好像 Slot 6 和主板的兩個 OCuLink 端口并不是同一個 CPU 的……

基本上,X11DPi-NT 的 VROC 功能極其有限。如果閣下有使用大量 NVME 作 RAID 的打算,那麽就不要重蹈本人的覆徹。可以考慮 X11DPH-T / X11DPH-i ——這款主板所有 PCIe 插槽都支援 VMD,包含兩個 M.2 插槽。并且應該支持任意的 Bifurcation 模式,這對於打算使用 PLX / PEX PCI-E Switch 的用戶來説是一個不錯選擇。

2020年9月15日 星期二

Intel Xeon Platinum 8269CY 處理器使用及兼容性報告

Intel Xeon Platinum 8269CY 這款 CPU 是“阿里雲” (Alibaba Cloud) 的定制版。在超微 Supermicro X11DPi-NT 主板上 (BIOS 3.1a)測試成功。不過,該 CPU 僅支援 2666MHz 的内存(2933MHz 的也運行於這個速度),這也是一個要注意的地方。

該處理器支援 Intel Speed Select 技術(型號中帶 'Y' 的 SKU 都有的),有兩個 Profile (預設方案):標準的 26 核,運行於 2.5 GHz 的基礎速度,全核運行最高速度 3.2 GHz;以及高性能的 20 核,運行於 3.1 GHz 的基礎速度,全核運行最高速度 3.5 GHz。兩個方案的單核最高速度均爲 3.8 GHz,兩個方案的 TDP 都是 205W。方案可以在 BIOS 裏面設定。


 

按照耗電量來説,除非這個定制版還有什麽特殊的能力,否則其功耗  205W 有點過高。不過勝於這個是步進 7 的正式版 CPU,穩定性和兼容性和其他零售版的 Platinum Xeon 相若,價錢合適的話也是個不錯的入場選項。

2020年8月22日 星期六

Farewell: Hard Disk Drives

It has been so many years already - my first HDD was a CONNER CFS-420A with capacity 420 MB. It should be around 1994. That was the exciting moment I can still remember - what an upgrade to 1.44MB floppy disks!

More than 25 years later, now I have made the decision to phase out all HDD sin my system. I have even bought a few 12TB Seagate Exos for my next project, but now they will be sold before being put into real use.

My experiences with HDDs could be quite a bit different from most of the people out there: I was very cautious about reliability and durability, rather than focusing mainly on capacity. If you have ever noticed about the specifications of the most decent HDDs, you might start noticing one thing: the Unrecoverable Error Rate has remained 10^14 (consumer parts) and 10^15 (server parts for many years without improvement.

Yes, right, who would be really caring about these numbers? They are just indicators, and new technology should be always more reliable, right? Unfortunately, my personal believe is, if there is a number published in that way, it MUST has a meaning. Consider other common storage technologies, such as SSD (10^17 for enterprise parts) and Tapes (10^17 - 10^19), what I can say is HDDs have a frightening LOW reliability nowadays.

Imagine a 1 TB HDD and a 10 TB HDD, they both have URE of 10^15. During a RAID rebuild, 10TB one will have 10 times higher chance to encounter an URE. While 10^15 URE means you *might* encounter a Non-Recoverable Read Error every 125 TB of data, this is not a lot for today's large capacity drives. If I am using 10x 12TB Exos HDDs to build a RAID 6 array, it will be so likely to get a Read Error during re-build - which is exactly the reason putting me off from proceeding with my original plan. I just don't want to put my data at risk.

That's also the reason why in the past my 16 HDDs RAID 60 array served me well, because each member was only 1TB in size. With 10^15 URE, the array was still relatively secured and it has proved itself. However, with 12TB, 10^15 URE is DANGEROUS and it is not making sense for traditional RAID techonology.

Also, think about the rebuild time - with my 16 HDDs RAID 60, even with a 1TB drive size, the rebuild time was somewhere around 4 hours without anyone using it. How about a 12 TB HDD? Well, if you are luck, you can have the array rebuilt in 2 days (given that nobody will be using it). Otherwise, it could be "weeks" - and the longer the time, the more likely another unrecoverable read error can happen. (And another one!)

You may say RAID is now out of date. But unfortunately it is the opposite - HDDs are now out of date. I now have an array of 2TB SSDs in RAID 6, and I am feeling way more comfortable with their 10^17 URE. The maximum size of SSD I can accept to be put into the array would be 15.36TB - if higher then I will need 10^18 URE.

HDD has a good characteristic: offline durability. This is especially true when it comes to offline data storage - think about data retention of SSD when it has been completely powered off - after the Intel 320 array incident happened to me in the past (data gone after powering off for > 1 month), I think you might expect data could be held in server class SSDs for 1 year max offline without problem, but don't expect more than that. HDD can do far better than SSD in this case since the storage of data is not depending on electricity stored in NAND cells that could leak, but on platter with magnetic recording that can last.

HOWEVER, how often are you going to put your HDDs offline? Possibly never - they are in RAID, and they are serving as nearline storage. This has defeated the purpose of this great characteristic. For nearline storage, I can easily use SSDs because they are ONLINE, with continuous power supply so they have the same data retention reliability as HDDs. They can run cooler and they have far better access performance than HDDs. So that's why my 12TB Exos are out of the picture for my project - I will be getting some 15.36 TB SSDs instead.

For archiving, an "old school" technology has somehow came to my mind - TAPES! They are offline storage, portable, power efficient and has a really high URE. They use magnetic recording as well so data retention is surprisingly good. If I need archiving, a tape library is better suited in this case.

With Tapes and SSDs, I have to say "farewell" to HDDs - unless URE has been increased to 10^16 or even 10^17, they won't be considered by me just because I want to have better sleep at night without worrying about RAID rebuild...

FAREWELL, MY LOVELY HDDS!

2020年8月1日 星期六

Windows Network Direct: Your better bet is with Windows Server 2019

I have been always struggling to get RDMA working inside a Windows virtual machine. I had tried Mellanox ConnectX-3, Mellanox ConnectX-5 and Chelsio T62100-LP-CR network adapters, with Windows Server 2012 or 2016, and even with Direct Device Assignment in Windows Server 2016, I could not get RDMA working flawlessly in any virtual machine.

Recently, I retried RDMA in a Windows VM (2019) on a Windows Server 2019 host, with a Chelsio T62100-LP-CR - and finally have RDMA (iWarp) working correctly (even without a switch - you can connect port 1 to port 2 to form a 100GbE link). It enabled SMB Direct between the VM and the host (and between VMs as well), and performance was acceptable (needs tuning).

If you are after any RDMA application inside a Windows VM, or simple just want to use SMB Direct in a VM, Windows Server 2019 or later is your better bet in this case.

Do note that you need the following:
  • SR-IOV support from BIOS - this sometimes means enabling the ASPM option in BIOS.
  • A network card that supports RDMA - I like iWarp because it is simpler (virtually no configuration needed). If you like RoCE then you may need DCB configured properly, or even need a 40GbE/100GbE switch.
  • Windows Server 2019 or higher - both host and VM. You may use Windows 10 (latest) - I didn't try that out but theoretically it should work.
  • Workstation / Server grade hardware - I have seen many times people complaining about not being able to enable SR-IOV due to missing implementation like IOMMU or ACS etc. with consumer grade hardware. Your CPU supports all these features doesn't mean your motherboard / BIOS has support of all features.

2016年8月28日 星期日

Windows Server 2016 DDA 直接設備授予虛擬機的一些要點

算是微軟的一些“突破”吧,反正 VMWare 一早就有這個技術。不論如何,至少直接把 PCI Express 設備,例如顯卡,聲卡,網卡或者 USB 3.0 端口授予虛擬機單獨訪問已經變得可能。然而并非所有設備都都能夠使用此功能。掃描了一下自己伺服器的硬件大概折這樣的:

1、老式 PCI 設備——別想了,放棄吧。
2、PCI Express 1.0 設備或者鏈路到 PCI Express 1.0 橋接芯片上的設備——也別想了,洗洗睡吧。
3、連路到 Intel PCH (就是現在有點想南橋的那個東東) 的設備——看運氣,就算你是 PCI Express 2.0,沒有 ACS (Access Control Service) 那也是別想了……什麽意思呢?就是板載的網卡,USB 3.0 口之類的,衹有是從 PCH 延伸出來的,就別想了……
4、有 ACS,但是是複合設備,那也別想了,至少我的 NVIDIA Quadro K4200 是不行的!

所以最終發現能夠 DDA 的就衹有直接連在 CPU 的根部 PCI Express 3.0 端口的設備,或者鏈路到支持 ACS 的 PCI Express 2.0 擴展盒的設備:
- Dell PERC H730P (別開玩笑了……把 100 個磁盤弄到 VM 裏面幹啥?)
- Mellanox ConnectX-3 (好吧,算是靠譜一點……)
- HP ioDrive DUO SLC 320GB (算不算 NVMe 直接授予!?-_-)

看來我那些 PCI Express 1.0 的 Expansion Backplanes / Enclosures 都沒啥用處了~

Incompatibilities and Compatibilities

NOTE: This article will be updated in the future when more compatibilities / incompatibilities are discovered.  Incompatibilities   12-Feb-...