Bye VMware ESXi, Hello Proxmox VE!

As you may -or may not- know, I manage my own server farm. This farm is composed of several virtual machines hosted on a physical server (an HPE ProLiant one). I’ve used ESXi for the past several years, but I’ve decided to move away from it for several reasons.

I – The context

The physical server is located at home (I’m self hosted). It’s mounted into a medium-sized rack cabinet in a room, with a UPS. This physical server runs multiple virtual machines, and I’ve listed some of them below:

  • The backbone infrastructure (LAN): a DNS server for resolving my internal machines’ names and a SMTP relay so that I can be notified when an event occurs somewhere in the system, like automatic security updates ;
  • The website infrastructure (DMZ): a SQL server, a file server with the webroot, two web servers for HA and load balancing, and a reverse proxy reachable from the Internet.

II – The reasons

VMware ESXi is an enterprise-grade type-1 hypervisor and it gets the job done, not gonna lie. But there are some downsides that prevent a lot of people from using it in a lab or semi-professional environment like I used to do until now:

  • The licenses’ prices are crazy high (there’s a free license but it comes with limits);
  • There’s no integrated backup solution (*wink* Veeam *wink*);
  • ESXi can run on unsupported, retail hardware, but the requirements leave some older machines aside;
  • HPE ProLiant servers (Gen8 and up) can’t run a vanilla ESXi, an ISO issued from HPE that integrates the required drivers has to be used;
  • HPE only certifies the latest servers for VMware and releases ISOs for these servers accordingly.

Do you see the problem? Even tho your older HPE server behaves just fine and can still handle modern operating systems, like Windows Server 2019 for instance, HPE won’t build ISOs of the latest ESXi for them.

According to HPE and VMware, my server can run ESXi up to 6.0. I’ve been able to use ESXi 6.5 because the HPE-issued ISO contained the required drivers for my server but it hasn’t been certified for this specific model. ESXi 6.7 has been released a few months ago but I can’t use it without doing some wizardry I’ve seen on the internets because the latest HPE ISOs don’t have the needed drivers. ESXi 6.5 won’t receive security updates after November 15, 2021. So my only choice would have been to purchase a new server and install a newer ESXi by then.

But here comes Proxmox VE:

  • It’s based on Debian with a customized, fine-tuned Linux kernel from Ubuntu;
  • It’s Open Source;
  • It supports a large variety of hardware;
  • It supports virtual machines (with KVM) and containers (with LXC);
  • It supports ZFS;
  • The requirements are quite low;
  • It has an integrated backup solution;
  • It has an integrated firewall;
  • You can create clusters of hosts running Proxmox VE;
  • It’s free, and you can subscribe to an enterprise support service so the Proxmox team can come to the rescue if something goes horribly wrong and the community alone can’t help.

III – Moving away

Since Debian 10 “Buster” was released this year, Proxmox VE has been updated as well to use this new version as its base system. Moreover my VMs were running Debian 9, so it was the perfect time to start using Debian 10 as well.

I didn’t migrate the VMs from ESXi to Proxmox VE (although you can import VMs from VMware’s virtualization suites, not only ESXi). Instead, I exported the most important configuration files and the website’s data (SQL + webroot) and recreated the entire virtualized infrastructure from scratch after erasing the disks and installing Proxmox VE.

Proxmox VE’s Web GUI can be somewhat confusing or overwhelming at times, but it’s fast, lightweight and you’ll get used to it. Sure, you’ll maybe have to use the CLI for advanced tasks more often than before than with ESXi, but the integration of xterm.js and other console viewers like noVNC makes it incredibly easy to do so, with the comfort of you web browser and without the need of a SSH access. And when using the CLI, you get the power of the Debian 10 beneath Proxmox VE.

IV – Final note

In my use case, Proxmox VE has the following advantages:

  • Integrated backup solution means no more 3rd party software to pay for;
  • Completely Open Source and no additional fees for additional features;
  • With ZFS I can put my hard disk drives in a RAIDZ-2 configuration, meaning that the pool can handle up to two disk failures before being unrecoverable (drives where on a RAID10 configuration when ESXi was installed, with a limit of one faulty disk at a time);
  • Once Proxmox VE is fully loaded and the VMs are on, everything becomes really fast: ZFS uses RAM as a cache, and it stores the data you use most frequently inside for faster access times. And when you want to write something to the disk, it gets stored in RAM, and ZFS writes it physically in the background at the most appropriate time. And since the VMs and Proxmox VE itself are stored on a ZFS pool, everything benefits from this technology.