Around 6 months ago, I decided to tear down the Harvester HCI cluster I had built and embark on a journey to use OKD , the open-source upstream of RedHat’s OpenShift product. Through this post, I would like to document my experiences with Harvester, OKD, and ultimately why and how I made the change.
Hello, Harvester!
Harvester is a Hyper Converged Infrastructure (HCI) solution from OpenSUSE’s Rancher project teams. It is designed to be installed onto bare metal systems and provide a Kubernetes-friendly platform. Each node runs an embedded operating system, SUSE’s COS , which itself runs k3s. Each nodes then joins one-by-one to each other to form a cohesive cluster. Volume management is provided by Rancher’s Longhorn , and workloads are designed to be deployed onto KVM guests via KubeVirt . With the exception of KubeVirt, this is an extremely tightly coupled solution all out of SUSE.
System setup
I had a fairly rough start with Harvester, like many others , I could not get it to install via ILO and iDRAC. The disk image is massive, and the low bandwidth between the out of band management device and the host OS would cause the installation to proceed incredibly slowly. And patience did not reward you since the installer had a fixed timeout and would cancel the process, even with the system partially installed.
The fix for this took a bit of setup, but was relatively straight-forward in the end. If you actually want to do this, I suggest Harvester’s great docs
, but to preserve how I did this: I setup isc-dhcp-server, a tftpd server, nginx, and got to netbooting!
For my /etc/dhcp/dhcpd.conf:
group {
if exists user-class and option user-class = "iPXE" {
if option pxe-system-type = 00:07 {
filename "http://10.10.0.8/harvester/ipxe-create-efi";
} else {
filename "http://10.10.0.8/harvester/ipxe-create";
}
} elsif substring (option vendor-class-identifier, 0, 10) = "HTTPClient" {
option vendor-class-identifier "HTTPClient";
filename "http://10.10.0.8/harvester/ipxe.efi";
} else {
if option pxe-system-type = 00:07 {
filename "ipxe.efi";
} else {
filename "undionly.kpxe";
}
next-server 10.10.0.8;
}
}
For my tftp server, I downloaded the swiss-army tool of netbooting, iPXE. Specifically, the ipxe.efi and undionly.kpxe which handle EFI and BIOS systems, respectively. You can fetch these files from iPXE’s website here
.
For nginx, I just served a directory containing the following two configuration files:
ipxe-create-efi
#!ipxe
kernel harvester-v1.1.0-vmlinuz-amd64 initrd=harvester-v1.1.0-initrd-amd64 ip=dhcp net.ifnames=1 rd.cos.disable rd.noverifyssl console=tty1 root=live:http://10.10.0.8/harvester/harvester-v1.1.0-rootfs-amd64.squashfs harvester.install.automatic=true harvester.install.config_url=http://10.10.0.8/harvester/create-cluster.yaml rd.net.dhcp.retry=10
initrd harvester-v1.1.0-initrd-amd64
boot
Key things to pay attention to here:
harvester.install.config_url=xxxmust be a different configuration file between the first, bootstrap node and the later, follower nodes. More on this later. Note: I never actually created a non-EFI version of this since I didn’t need it, but it’s probably largely the same.
create-cluster.yaml:
scheme_version: 1
token: SomeRandomlyGeneratedLongValueThatLetsOtherNodesJoin
os:
ssh_authorized_keys:
- ssh-rsa ...
password: "$6$PASSWORD"
ntp_servers:
- 0.suse.pool.ntp.org
- 1.suse.pool.ntp.org
install:
mode: create
management_interface:
interfaces:
- name: eno1
default_route: true
method: dhcp
iso_url: http://10.10.0.8/harvester/harvester-v1.1.0-amd64.iso
vip: 10.10.0.110
vip_mode: static
device: /dev/sda
Key players here:
management_interfacecannot be later used as a guest network.vipis floated across the cluster nodes to provide HA access to the cluster
The JOIN mode items are exactly the same, but join-cluster uses the “Cluster join token” value generated by your first node as the token field, and runs in mode: create. The ipxe file’s only modification is to use cluster-join.yaml instead.
In my case, this full setup worked on my two HP DL380 Gen9 systems as well as my three R630 systems. Not only did it work, but bootstrapping these nodes over the network took under two minutes (less boot times)! Felt like such a waste the three times I tried installing over ILO/iDRAC!
Additional Hardware Details
Each node is using 1 GBe connections for their management, and use a teamed 10GBe connection for the guest network. Additionally, the Dell nodes were outfitted with SSDs to act as the primary application storage pool.
Cluster Setup
Harvester strongly discourages running additional workloads on its integrated Kubernetes cluster, and instead prefers you to launch guests to handle the workload. This was a bit of a blow to me, realizing that even after all this setup, I’m stuck managing yet more VMs. However, at least Harvester is very API friendly! It is incredibly easy to create and configure instances with a few CLI commands or via the UI. KubeVirt was worlds easier to automate compared to the likes of Proxmox and VMware!
Infra Cluster
I had the idea to build a small infrastructure RKE cluster to house Rancher and Keycloak. This three-node cluster was built rather manually on top of Ubuntu guests, but all this effort provided me the ability to offload a lot of future work to Rancher.
With Rancher installed, I configured it to recognize the parent Harvester cluster, which unlocked a lot of excellent features. For one, the dedicated Harvester UI was no longer necessary since Rancher is now capable of managing the cluster for you.
For two, and even more significantly, Rancher can now provision clusters for you using Harvester. These clusters come up fully automatically, and even extend volumes from Harvester’s Longhorn cluster into your guest RKE cluster! On top of that, you can scale your cluster to new nodes with a single button click.
Prod Cluster
Well, more like “prod” cluster. Using the capabilities outlined above, I created a “production” cluster to contain all of my various hobby projects and experiments. Setup of this cluster was effortless, and could be managed through the Rancher instance provided by the Infra cluster.
My workloads are out of scope of this post, but everything ran here!
The gains
This cluster was my first time diving deep into Kubernetes, so I expected to have some growing pains as I learn how things are done here. With Rancher by my side to provide insight into what I’m doing, I felt unstoppable. It helped me understand how resources map together, the behaviors of various resources, and how to create and optimize containers for the Kubernetes world!
Ending with Harvester
Although Harvester+Rancher provided me a lot of opportunities to learn and grow, it did present some unavoidable challenges that could not easily be remediated. In this section, I will go over in detail challenges I, and likely many others have and will face running this configuration.
I will start this off by saying that RKE, k3s, and Rancher itself are all incredible tools. Without these, I likely would never had the opportunity to learn everything I have done. Similarly, KubeVirt is a solid tool that I continue to use today. It provides a great interface to the reliable KVM ecosystem.
For the components that I am critical of, I’m understanding that they are potentially young projects, and that my environment is less than ideal. They generally are a great foundation for a project, and I’m sure will only continue to get better. With all that said, let’s get digging!
Host management
I had hoped to move away from managing individual VMs with the shift to a converged solution. But given that Harvester doesn’t want/let you run workloads on its embedded cluster, you don’t really have a choice. I feel like I wouldn’t mind this too much if Harvester’s embedded Rancher instance could launch clusters like a full Rancher instance. But unfortunately, even if you access the regular Racher UI, this feature is not enabled.
An additional gripe is on the managed instances launched by Rancher. Accessing these instances is difficult since Rancher doesn’t make it easy to pass ssh-keys to their config. And accessing these instances is vital for making sure they have updates installed. The Rancher team did create the system-upgrade-controller , but I found it relatively difficult to setup/validate.
Longhorn
Longhorn is a file-based volume manager built for Kubernetes. For day-to-day work, it’s quick and relatively reliable. However, it falls flat in disaster recovery scenarios. I found myself lucky to not lose volumes during occasional power failures. Troubleshooting steps were largely unhelpful, and even when I could find the information on disk, it was impossible to convince Longhorn to rebuild the volumes. At the time, I was able to find stories from many people in similar situations who were similarly unable to recover their volumes without incredibly strange rsync hacks.
At this time, I was also running a Ceph cluster that faced the exact same kinds of power failures and unexpected outages, and it handled all of it no problem! Each time the rack came back online, all of my data was present and no signs of corruption were found. Ceph takes a paramount priority on data integrity, and it absolutely shows. And my excellent experience with Ceph made be long for it.
Enter Rook, a Kubernetes volume manager built off of Ceph, where everything is configurable via Rook’s CRDs.
…At least that’s what I would say if Harvester gave you the option not to use Longhorn. The developers have said that they’ve taken a backlog entry to support Rook, but that it’s not a current priority. Ultimately, this is the primary reason I moved away from Harvester.
On my way, OKD!
I had been familiar with OpenShift, and its unfortunate initial reputation where people drop the “f”. However, a former coworker who attended the RedHat conference had nothing but raving reviews about it. This thought clearly stuck with me, and was my primary influence for trying out OKD.
In my next post, I will go over my OKD installation process, my first impressions, and what I consider essential knowledge for folks going down the same path.