Contents

• Why?
• What?
• Where?
• The Tests
  • The Test VM
  • Apache Cloudstack
  • Hyper-V
  • Bare-Metal
  • XCP-ng
  • Proxmox / KVM
• Results / Comparison Charts
• Conclusion
• Surprises

Foreword

I am fully aware, that as soon as you post something on the Internet, no matter what, someone will tell you and prove that you are wrong.
I am always happy to get constructive feedback to improve things, though. But it is unlikely that I will run this series of tests again soon, because including writing this article and trying to figure out how Hugo works, that whole adventure took 3 days off my 1 week holiday ;-)

There are other tests out there that use different workloads and they might come to different conclusions. If you want to conduct this particular test yourself to check if I measured bogus: All the Software / Code used during these test runs is free, so you can replicate this on your own hardware if you want.

Why?

Out of personal interest, but triggered by the fact that, at work, we are currently testing a Apache CloudStack (ACS) with KVM installation and found out that a pretty common task for us runs around 20% slower on ACS than on the current Hyper-V environment. My expectation was that the ACS setup should run considerably faster, because the Hyper-V Cluster runs on now nearly 5 year old hardware which was heavily in use while doing the testing and the ACS Cluster should be brand new Hardware which only had to handle the test load.

So I asked myself if we are dealing with a Hardware / Configuration issue here or if KVM is really running that much slower with the same workload (hosting Windows VMs, running our specific workload) than Hyper-V.
It would be nice to do a A/B Test with exactly the same Hardware to find out, right?

No better time for such testing than having a week off from work and doing some science in my Homelab ;-)

What?

The Hypervisors interesting to me from a business perspective are Apache CloudStack (ACS) and Hyper-V. But while I am on it, why not test others too I come across when homelabbing. So this is the list of Hypervisors I am conducting the same tests.

• Microsoft Hyper-V
• Apache Cloudstack 4.20 with KVM
• Proxmox VE with KVM
• XCP-ng with Xen

On top of that, I ran the same test Bare Metal.

Full disclosure: I have not much experience with KVM and Xen outside of my Homelab. I tried to apply the Hypervisor specific tweaks to make the VMs as quick as possible, but I may have missed some that might possibly help to speed things up. Please keep that in mind. If you are a KVM / Xen Guru and know some additional tweaks to speed up a Windows VM under these Hypervisors considerably, I would be happy if you let me know what they are :-)

The specific Workload I am talking about is the Test Next Major GitHub Action from Freddy Kristiansens AL-Go.
For those who are not familiar with it: This will basically download Build Artifacts for the Next Major Version of Microsoft Dynamics 365 Business Central, build a Docker Image, spin up a Container, compile the Sourcecode from the GitHub Repository the Action runs on and will destroy the whole environment again. From what I could see, during the run this puts quite some strain on the CPUs (sometimes maxing out all 6 Cores), RAM and Disks. To leave out the Internet connection / download from the Azure CDN in that equation, the first run of this Action will not count in these tests. After the first run, the Build Artifacts and Base Image for the Container are cached on Disk already. This should give pretty consistent results when running the task multiple times.

Regarding the Hypervisors:
When switching the Hypervisor, this means I wiped the Disk of the Testsystem and then installed the system hosting the next Hypervisor to be tested. There was nothing left or re-used from the previous test.
Also the Test VM has been rebuilt from scratch for each test.

Where?

As I said, this is out of personal interest and this means I ran these tests on my own (modest) spare hardware:

Dell OptiPlex 3070 Micro:

Intel Core i5-9500T @ 2.20GHz (6 Cores)
16 GB RAM
KXG60ZNV256G NVMe TOSHIBA 256GB

This is not the kind of system you would run that workload on in a production environment. But I want an A/B Test with different Hypervisors under the same circumstances, so it should do the trick… and it’s all I have, so there’s that.

The Tests

Every test was based on the same (AL) code hosted in a GitHub repository with AL-Go added. For the test runs I utilized the Action “Test Next Major”.
Since on the first run the Build Artifacts and base image get downloaded, this run never counts for this test. The step in this Action that puts the most stress on the hardware is the Build step, so I measured the time for that also. I think it gives a better idea of how good a Hypervisor performs, so I include both (total runtime + build runtime) in the results.

It is also worth nothing that during a run, memory consumption never went above 7 GB. This is important to know for the test that I also ran directly on the Test machine without a Hypervisor, because I was not able to limit the memory from 16 to 8 GB for that test.

The Test VM

On each Hypervisor I created a new VM, bootet from ISO, installed the OS and everything else needed for the tests.

VM Size:

• 6 Cores
• 8 GB RAM
• 60 GB HDD (always hosted on the NVMe)

OS:
Windows Server 2022 Standard Evaluation 21H2 with UEFI and Secure Boot enabled, fully patched with the latest Patchday Updates.

Additional Software installed:

• GitHub Self-Hosted Runner
• Docker CE/Moby 28.0.1
• PowerShell 7.5.0
• Git Client 2.48.1 x64
• Azure CLI 2.69.0 x64
• .NET Core SDK 8.0.406 x64
• .NET Core Runtime 8.0.13 x86 and x64
• WSL (wsl –install / wsl –update)

I always rebooted the VM before starting the test runs and made sure it settled down from all the updates / software Installations done previously.

Apache Cloudstack

Base OS: Ubuntu 24.04 LTS
ACS: Quick Install as in https://rohityadav.cloud/blog/cloudstack-kvm/, Version 4.20

Hypervisor specifics for the Test VM:

• KVM “cpuSpeed” set to 2.200
• VirtIO Drivers / Guest Tools v0.1.266
• machine = pc-q35-8.2
• cputype = host-passthrough
• Made sure the ovmf package is installed for the UEFI Firmwares
• /etc/cloudstack/agent/agent.properties added
  • guest.cpu.mode=host-passthrough
  • guest.nvram.template.secure=/usr/share/OVMF/OVMF_VARS_4M.ms.fd
  • guest.nvram.template.legacy=/usr/share/OVMF/OVMF_VARS_4M.fd
  • guest.loader.secure=/usr/share/OVMF/OVMF_CODE_4M.secboot.fd
  • guest.nvram.path=/var/lib/libvirt/qemu/nvram/
  • guest.loader.legacy=/usr/share/OVMF/OVMF_CODE_4M.fd

    xychart-beta
	  title "Apache Cloudstack / KVM Total Runtime (Bar=ACS, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [1142, 1183, 1144, 1170]
	  line [740, 750, 746, 767]

    xychart-beta
	  title "Apache Cloudstack / KVM Build Runtime (Bar=ACS, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [895, 898, 889, 885]
	  line [501, 498, 494, 509]

Hyper-V

Base OS: Windows Server 2022 21H2 Standard Eval
Hyper-V Server Role installed

Hypervisor specifics for the Test VM:

• Hyper-V VM Type is Generation 2

    xychart-beta
	  title "Hyper-V Total Runtime (Bar=Hyper-V, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [752, 800, 751, 745]
	  line [740, 750, 746, 767]

    xychart-beta
	  title "Hyper-V Build Runtime (Bar=Hyper-V, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [543, 574, 532, 530]
	  line [501, 498, 494, 509]

Bare-Metal

To see how much of a difference Bare Metal vs. Virtualization makes, I took the opportunity to also run a Test on the machine itself while Windows was installed. Since during the test runs within the VMs, memory consumption never went higher than 7 GB and the VMs had 8 GB, I can only assume that the extra memory the Host itself has available, should not make a big difference.

Test run specifics:

• 16 GB RAM available

    xychart-beta
	  title "Bare Metal Runtime (Bar=Full, Line=Build)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [752, 708, 735, 719]
	  line [514, 511, 520, 517]

XCP-ng

Base OS and Hypervisor: XCP-ng 8.3.0, running Linux xcp-ng 4.19.0+1

Hypervisor specifics for the Test VM:

• XenServer VM Tools for Windows 9.4.0

    xychart-beta
	  title "XCP-ng Runtime (Bar=Full, Line=Build)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [740, 750, 746, 767]
	  line [501, 498, 494, 509]

Proxmox / KVM

Base OS: Proxmox VE 8.3.0
Hypervisor: QEMU emulator version 9.0.2 (pve-qemu-kvm_9.0.2-4)

Hypervisor specifics for the Test VM:

• VirtIO Drivers / Guest Tools v0.1.266
• ACPI Support = Yes
• QEMU Guest Agent = Enabled
• Machine = pc-q35-9.0
• Processor Type = Host

    xychart-beta
	  title "Proxmox VE / KVM Total Runtime (Bar=Proxmox, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [1042, 1061, 1041, 1047]
	  line [740, 750, 746, 767]

    xychart-beta
	  title "Proxmox VE / KVM Build Runtime (Bar=Proxmox, Line=XCP-ng)"
	  x-axis "Run" [2nd, 3rd, 4th, 5th]
	  y-axis "Time (in s)" 0 --> 1500
	  bar [796, 810, 808, 825]
	  line [501, 498, 464, 509]

Results / Comparison Charts

Here are the comparison Charts for all the runs.

First we have the Total Runtime, which includes some tasks before and after the build process. They barely create any load on the system, but they take some time. Since in these parts Internet connectivity could play a role, the next chart is the build run only.

And here the pure build part of the runs. That is the part that makes the VMs sweat and I think is the better indicator of how each Hypervisor performs for that particular workload.

Conclusion

I had to put KVM twice in here to see if it performs like that when managed with ACS or if it does not make a difference. Configuring KVM within ACS, due to the numerous settings, is not as straight-forward as in Proxmox. So take the ACS results with a grain of salt. This might also be, why ACS and Proxmox are not as tightly together as one would assume, using the same Hypervisor. The performance difference between ACS and Proxmox, due to a probably not optimal configuration of ACS aside, seeing how far away they are away from all the others was a huge surprise for me. Maybe someone with more knowledge about KVM knows the tweaks to improve KVM via ACS and Proxmox, or even get them on-par with the other Hypervisors. On the Proxmox side, I think I set all the parameters to make the VM as quick as possible. But even with that, KVM (in my specific test scenario) is much slower than the other Hypervisors.

If you are a homelabber, you might want to squeeze everything out of the limited hardware you have, so XCP-ng looks like a no-brainer. Compared to Hyper-V, which is very quick, XCP-ng is free. And using XCP-ng for quite a while now I can say it is quite easy to use, has a very quick management interface and ticks off nearly all the boxes for stuff I do at home. The only thing missing and why I might use a slower KVM based alternative at the moment is: XCP-ng does not support nested virtualization at the moment. If you are testing out a lot of stuff which does require nested virtualization, XCP-ng is currently off the table.

Surprises

XCP-ng
Not by much, but XCP-ng hosting Windows was always a few seconds quicker in the Build part of the test runs than Windows Bare Metal. I can not fully wrap my head around that. One could argue that the Base OS / Kernel of XCP-ng is more efficient than Windows, but then we are hosting a Windows VM that effectively runs the code we are testing. It would be fun to see how this workload performs with XCP-ng running a Linux VM, but that specific workload I was testing and interested in does not run in Linux as far as I know.

KVM
I absolutely did not expect that. I should have, from what I have seen at work when testing KVM in another ACS environment (where I am pretty confident that it is configured as efficient as possible) and it ran the same workload 20% slower than on an existing Hyper-V Cluster on old Hardware, but I hoped to see all of the Hypervisors close together and being able to fix the performance difference with a few tweaks. But since ACS and Proxmox are relatively close together, I am not confident that with the flip of a switch, KVM will catch up.