[k8s] Step-By-Step Guide: Hosting Longhorn on K3s (ARM)

As you know by default the kubernetes provide a “local-path” storage. However, this local storage has many limitation:

Node Affinity: When using local-path storage, the volume created is tied to the specific node where the pod runs. If that node goes down for maintenance or any other reason, the pods won’t be able to start on other nodes because they won’t find their volumes.
Not Network Storage: Local-path storage is not network-based. The volume remains local to the K3s node where the pod executes. It doesn’t allow data sharing across nodes.
Not Suitability for Production: While local-path storage is suitable for small, single-node development clusters, it’s not recommended for production-grade multi-node clusters.

What is Longhorn?

Longhorn, an innovative open-source project by Rancher Labs, offers a reliable, lightweight, and user-friendly distributed block storage system for Kubernetes.

High Availability: Longhorn replicates storage volumes across multiple nodes in the Kubernetes cluster, ensuring that data remains available even if a node fails.
Cost-Effective: Traditional external storage arrays can be expensive and non-portable. Longhorn offers a cost-effective, cloud-native solution that can run anywhere.
Disaster Recovery: Longhorn allows you to easily create a disaster recovery volume in another Kubernetes cluster and fail over to it in the event of an emergency. This ensures that your applications can quickly recover with a defined Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

Longhorn installation

Longhorn provides a straightforward method for installing the iSCSI driver and NFSv4 directly on all nodes. Follow the steps below to set up the necessary components.

Installing open-iscsi

The open-iscsi package is a prerequisite for Longhorn to create distributed volumes that can be shared across nodes. Ensure that this driver is installed on all worker nodes within your cluster.

Execute the following command on your cluster to install the driver

# please check the latest release of the longhorn here https://github.com/longhorn/longhorn and update the version accordingly. Current version is v1.6.0
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/prerequisite/longhorn-iscsi-installation.yaml

After deploying the iSCSI driver, confirm the status of the installer pods using the following command:

kubectl get pod | grep longhorn-iscsi-installation

# The result
longhorn-iscsi-installation-pdbgq   1/1     Running   0          21m
longhorn-iscsi-installation-qplbb   1/1     Running   0          39m

Additionally, review the installation logs to ensure successful deployment:

kubectl logs longhorn-iscsi-installation-pdbgq -c iscsi-installation

# The result
...
IProcessing triggers for libc-bin (2.35-0ubuntu3.6) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for initramfs-tools (0.140ubuntu13.1) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-18-generic
iscsi install successfully

Once the iscsi installed successfully, Then you can safely uninstall the above with following command.

kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/prerequisite/longhorn-iscsi-installation.yaml

Installing NFSv4 client

To enable Longhorn’s backup functionality and ensure proper operation, the NFSv4 client must be installed on the worker nodes within your cluster.

Follow these steps to set up the necessary components:

# please check the latest release of the longhorn here https://github.com/longhorn/longhorn and update the version accordingly. Current version is v1.6.0
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/prerequisite/longhorn-nfs-installation.yaml

After deploying the NFSv4 client, confirm the status of the installer pods using the following command:

kubectl get pod | grep longhorn-nfs-installation

# The results
NAME                                  READY   STATUS    RESTARTS   AGE
longhorn-nfs-installation-mt5p7   1/1     Running   0          143m
longhorn-nfs-installation-n6nnq   1/1     Running   0          143m

And also can check the log with the following command to see the installation result:

kubectl logs longhorn-nfs-installation-mt5p7 -c nfs-installation

# The results
...
rpc-svcgssd.service is a disabled or a static unit, not starting it.
rpc_pipefs.target is a disabled or a static unit, not starting it.
var-lib-nfs-rpc_pipefs.mount is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
nfs install successfully

Once the NFSv4 installed successfully, Then you can safely uninstall the above with following command.

kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/prerequisite/longhorn-nfs-installation.yaml

Installing Longhorn

Preparing the configuration value.yaml file

csi:
  kubeletRootDir: "/var/lib/kubelet"
defaultSettings:
  diskType: "flesystem"

Install Longhorn in the longhorn-system namespace with configuration above.

# 1. added longhorn chart
helm repo add longhorn https://charts.longhorn.io
helm repo update

# 2. Installing
helm install longhorn longhorn/longhorn -f value.yaml --namespace longhorn-system --create-namespace

Once installed successfully and all the pods are up and running. You should be able to access Longhorn UI through the longhorn-frontend service (needs port-forward or expose through nginx or cloudflare tunnel).

Uninstalling Longhorn

If any reason we would like to uninstall Longhorn helm then the below commands will help.

# Update `deleting-confirmation-flag` to allows uninstall longhorn
kubectl -n longhorn-system patch -p '{"value": "true"}' --type=merge lhs deleting-confirmation-flag
# Uninstall longhorn
helm uninstall longhorn -n longhorn-system
# Delete namespace
kubectl delete namespace longhorn-system

Using the Longhorn storage with Mariadb

To explore Longhorn storage capabilities, we’ll set up a MariaDB Galera multi-primary database cluster for synchronous replication and high availability. Follow these steps:

# Setup chart repo
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Install mariadb-ha
helm install mariadb-ha bitnami/mariadb-galera \
   --set global.storageClass=longhorn \
   --set rootUser.password=Pass@word1 \
   --set galera.mariabackup.password=Password1 \
   --set db.name=drunk_db \
   --namespace db --create-namespace

# Uninstall mariadb-ha
helm uninstall mariadb-ha -n db

After deployed successful, you should find three MariaDB pods running in the db namespace:

kubectl get pod -n db

# The results
NAME                          READY   STATUS    RESTARTS   AGE
mariadb-ha-mariadb-galera-0   1/1     Running   0          5m6s
mariadb-ha-mariadb-galera-1   1/1     Running   0          3m44s
mariadb-ha-mariadb-galera-2   1/1     Running   0          2m41s

You’ll also find three persistent volumes in the Longhorn UI portal. !Longhorn Volumes

Longhorn System Backup

Longhorn supports a variety of backup targets, including Azure Storage, AWS S3, Google Storage, NFS, and SMB/CIFS. This post will initially cover the configuration for Azure Storage, with subsequent posts addressing the other backup targets.

System Backup with Azure Storage

This section assumes that you already have Azure Storage and a Kubernetes cluster that can connect to Azure.

Create a longhorn-azure-blob-backup.yaml secret with your Azure Storage Account credentials:

# Update the secret below with your Azure Storage Account credentials
apiVersion: v1
kind: Secret
metadata:
  name: longhorn-azure-blob-backup
  namespace: longhorn-system
stringData:
  AZBLOB_ACCOUNT_KEY: "YOUR_STORAGE_KEY"
  AZBLOB_ACCOUNT_NAME: "YOUR_STORAGE_NAME"

Apply the secret to the cluster using the following command.

kubectl apply -f longhorn-azure-blob-backup.yaml

Next, update the backup information in the defaultSettings section of the value.yaml file:

csi:
  kubeletRootDir: "/var/lib/kubelet"
defaultSettings:
  diskType: "flesystem"
  backupTargetCredentialSecret: "longhorn-azure-blob-backup" #The name of the secret created above.
  backupTarget: "azblob://[email protected]/" # Trailer slash is important

Upgrade the Longhorn Helm chart with the updated values using the following command:

helm upgrade longhorn longhorn/longhorn -f value.yaml --namespace longhorn-system

# You should see a message indicating that the upgrade was successful
Release "longhorn" has been upgraded. Happy Helming!
NAME: longhorn
LAST DEPLOYED: Wed Feb 28 08:57:56 2024
NAMESPACE: longhorn-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Longhorn is now installed on the cluster!

Please wait a few minutes for other Longhorn components such as CSI deployments, Engine Images, and Instance Managers to be initialized.

Visit our documentation at https://longhorn.io/docs/

Once the upgrade is complete, the backup information under Setting → General should reflect the changes made.
After creating a sys-backup under Setting → System Backup, you should see the following
You should be able to see the data being backed up into the Azure Storage Container.

Thank You

Thank you for taking the time to read this guide! I hope it has been helpful, feel free to explore further, and happy coding! 🌟✨

Steven | GitHub