| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Active Research
Forum: Mapping Cancer Markers Forum
Thread: Virtualization in BIOS (VT-x) effect on performance |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 8
|
|
| Author |
|
|
jives11
Cruncher United Kingdom Joined: Nov 22, 2023 Post Count: 14 Status: Offline Project Badges: 
|
Hi,
I recently added a pair of Dell Wyse 3040 thin client PC's to my boinc farm , they are based on the Intel Atom and feature 4 core CPU's which run on very littler power. They are similar in concept to the RPi4 but are able to run amd64 code. They've been installed with headless Debian, and have 2Gb memory and 8Gb SSD. They are or should be identical. I installed boinc from the debian repository , configured both to work on WCG Cancer Markers . Looking through my results , I notice that node1 takes roughly 22 hours per work unit while node 2 takes closer to 6. I compared their installs and the only thing I could find different was Node 1 (the slow one) had Virtualization (VT-x) disabled in the BIOS. I assumed that VT-x would only improve the performance of running a virtual container, is that how MCM projects work also ? Or am I looking in the wrong place ? Thanks |
||
|
|
William Albert
Cruncher Joined: Apr 5, 2020 Post Count: 41 Status: Offline Project Badges: 
|
MCM work units run as standalone binaries -- not VMs -- and the presence or absence of VT-x shouldn't make a difference.
Also, Intel CPUs require VT-x to run 64-bit code in virtual machines, so if you were relying on VT-x in some way, I would expect the consequence of not having it to be programs failing, not running slower than expected. A performance delta that large (assuming it's consistent) is more likely to be the CPU on node1 running much slower than expected, either because it's been downclocked in the BIOS, or is thermal throttling. |
||
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 865 Status: Offline Project Badges: 
|
What specific Intel Atom CPU is it? Maybe two of the cores are performance cores, and the other two are efficiency cores. That might explain the huge difference in speed. That and maybe the 2 GB of RAM is maxed out? I doubt it, especially headless Debian.
----------------------------------------[Edit: I checked one of my headless Debian quad core systems and 4 MCM1 tasks right now. It's using 324 MB RAM according to htop, and it's using 551 MB according to btop.] My gut feeling is that VT-x shouldn't be relevant here unless you're running Debian in a VM as opposed to baremetal. That 8 GB SSD might be eMMC or something, which might not last long compared to the flash memory used in a regular SSD drive. I'd be prepared for it to fail without warning, especially if you don't tweak BOINC to minimize writing to disk, caching, logging, etc. I bought a handful of 32 GB USB flash drives for $3.99 each from Microcenter, and one of my boxes runs on one. They're getting so cheap now. Anyway curious what CPU is in those Wyse thin clients.
[Edit 1 times, last edit by hchc at May 15, 2025 6:13:31 AM] |
||
|
|
jives11
Cruncher United Kingdom Joined: Nov 22, 2023 Post Count: 14 Status: Offline Project Badges:
|
Thank you for the replies.
----------------------------------------I set the VT-x flag enabled on node 1 and it's now returning work in 6-7 hours, not 22, so it seems as if the BIOS VT-x flag is the cause To confirm I have debian installed on bare metal, no VM's involved. It was a headless install basic os + ssh, and journaling is disabled . $ neofetch OS: Debian GNU/Linux 12 (bookworm) x86_64 Host: Wyse 3040 Thin Client Kernel: 6.1.0-34-amd64 Uptime: 1 day, 20 mins Packages: 729 (dpkg) Shell: bash 5.2.15 Terminal: /dev/pts/0 CPU: Intel Atom x5-Z8350 (4) @ 1.920GHz GPU: Intel Atom/Celeron/Pentium Processor x5-E800 Memory: 339MiB / 1879MiB Without boinc, it's running only 12 processes. There appears to be adequate free memory $ free -h total used free shared buff/cache available Mem: 1.8Gi 492Mi 1.3Gi 1.0Mi 208Mi 1.4Gi Swap: 975Mi 0B 975Mi Temperature hovers around 57 C under full load, so I don't think thermal throttling is the issue I picked these units up for circa £15 each so I'm not too worried if they fail due to over use of the internal storage. I disable journaling routinely anyway to minimise IO. sound is also disabled in the BIOS. These machines were also working on projects from gaia@home until it ran out of work a week ago. Looking through the results from then I see comparable delta between the two nodes (1 with VT-x enabled 1 disabled). The MCM data I was comparing was after work had dried up from gaia, so I don't think the problem was conflict between projects. This is Node 1, and Node 2 gives identical results $ sudo dmesg | grep MHz [ 0.000000] tsc: Detected 1440.000 MHz processor $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 76 model name : Intel(R) Atom(TM) x5-Z8350 CPU @ 1.44GHz stepping : 4 microcode : 0x411 cpu MHz : 1680.000 cache size : 1024 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown bogomips : 2880.00 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 76 model name : Intel(R) Atom(TM) x5-Z8350 CPU @ 1.44GHz stepping : 4 microcode : 0x411 cpu MHz : 1680.000 cache size : 1024 KB physical id : 0 siblings : 4 core id : 1 cpu cores : 4 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown bogomips : 2880.00 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 76 model name : Intel(R) Atom(TM) x5-Z8350 CPU @ 1.44GHz stepping : 4 microcode : 0x411 cpu MHz : 1680.000 cache size : 1024 KB physical id : 0 siblings : 4 core id : 2 cpu cores : 4 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown bogomips : 2880.00 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 76 model name : Intel(R) Atom(TM) x5-Z8350 CPU @ 1.44GHz stepping : 4 microcode : 0x411 cpu MHz : 1680.000 cache size : 1024 KB physical id : 0 siblings : 4 core id : 3 cpu cores : 4 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown bogomips : 2880.00 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [Edit 2 times, last edit by jives11 at May 15, 2025 7:50:04 AM] |
||
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 865 Status: Offline Project Badges:
|
Intel Ark page:
----------------------------------------https://www.intel.com/content/www/us/en/produ...2-ghz/specifications.html That's interesting that enabling VT-x somehow dramatically improved the performance of MCM1 tasks despite running baremetal. I don't have the answer to why. Those look like cheap little old thin clients. CPU came out early 2016 and is 2 Watt SDP. I imagine the whole device consumes 10 Watts from the wall at the most. Fascinating, since I'm looking for ways to affordably increase my contribution to WCG. I'm still kind of leaning towards 1 16c/32t machine instead of a farm of little quad cores -- less management overhead. Question since I'm a Linux beginner: How do you disable journaling in Debian headless? I wrote a beginner's tutorial in the Hardware forum: Beginner's Guide to installing and config...nance dedicated crunchbox and am looking to improve it, especially with reducing disk writes. Anyway sorry to not be of more help on VT-x. That's very odd. *shrug* - perhaps just leave it enabled. It doesn't answer your "why" question, however. Edited to Add: I noticed from the Intel Ark page that that Atom CPU has 2 MB total CPU cache (L3 cache?). I seem to remember each MCM1 task likes something like 3 MB of L3 cache each? I don't quite remember so I'd have to search this forum. I seem to remember people talking about CPUs with not enough L3 cache for MCM1 resulting in "cache misses" and having to page to system memory and it dramatically increasing the runtime of each task.
[Edit 1 times, last edit by hchc at May 16, 2025 2:33:11 AM] |
||
|
|
jives11
Cruncher United Kingdom Joined: Nov 22, 2023 Post Count: 14 Status: Offline Project Badges:
|
Many Thanks
----------------------------------------Your final point about cache could well be the issue. I got this reply over on the Debian forum Re: Debian on Dell Wyse 3040 - BIOS, virtualisation and boinc performance #2 Unread post by CwF » 2025-05-15 22:21 It's not that it does more than allow vm's to work, it's how it does it that applies to more than only vm's. It is a whole group of magic we'll call it that allows the redirects to happen in hardware with a re-address that formerly required a copy-to then read. Multi-core combined with internal memory controls per core can lead to a ‘block’ managed by one core needs access by another core. so instead of moving the block into the jurisdiction of the core that needs it, vt-x can simply remap it directly. Where it can't help is with actual mutlisocket system with numa zones. There in the same way, zone to zone movement can be costly but required. A different layer of control is needed to keep jobs local to a group of cores working the job. Some jobs can vary 30%+ when split vs confined. That's interesting. My natural instinct is always to disable stuff in the BIOS that I don't need, assuming less kernel modules will be loaded, and this works as expected when disabling Sound in the Wyse 3040 BIOS, meaning around 12 modules don't have to be loaded I run a menagerie of boinc servers including 6 Mac pros, 3 mac minis, 13 RPi's , a couple of Linux Pc's (one 32bit) and now a pair of these tiny thin client machines. The macs and Linux PC's only run at night when my electricity is cheap. I do something similar to your guide, which is excellent BTW, quite a few things for me to try. I currently manage each server via ssh individually. My master desktop is in fact mac, so I need to explore a unified management desktop for the farm that runs on mac, if such a thing exists. All the Linux machines are headless and just run ssh, no Window manager The Dell wyse 3040 seem very cheap, as they are roughly the power of an RPi4, but can run amd64 projects (MCM for example) and are very cheap on ebay. I paid £15 each but that includes case and PSU, which you have to buy separately with an RPi. If all you want is a silent blackbox boinc crunch server, they are great. I always seem to fail to disable swap during the debian net install, and it's actually very hard to disable subsequently I disable journaling using: $sudo systemctl disable --now systemd-journald.service systemd-journald-audit.socket systemd-journald-dev-log.socket systemd-journald.socket; sudo systemctl mask systemd-journald.service; [Edit 4 times, last edit by jives11 at May 16, 2025 3:08:11 AM] |
||
|
|
Tamagoch
Cruncher Ukraine Joined: Jul 21, 2007 Post Count: 35 Status: Offline Project Badges: 
|
Don't underestimate a possible external power unit issue. Dell runs their CPU at 800 MHz maximum if thinks that a power supply is not an original one or is malfunctioning. At least try to switch them between each other to see if the problem jumps as well.
----------------------------------------[Edit 1 times, last edit by Tamagoch at May 23, 2025 12:05:02 PM] |
||
|
|
catchercradle
Senior Cruncher England Joined: Jan 16, 2009 Post Count: 167 Status: Offline Project Badges: 
|
Interesting thread. Before I gave it away, I had a 2 core Atom netbook that used to take about 5 months to complete CPDN tasks!
|
||
|
|
|