What is Liqo?

Liqo is an open-source project that enables dynamic and seamless Kubernetes multi-cluster topologies, supporting heterogeneous on-premise, cloud and edge infrastructures.

What does it provide?

Peering

Automatic peer-to-peer establishment of resource and service consumption relationships between independent and heterogeneous clusters. No need to worry about complex VPN configurations and certification authorities: everything is transparently self-negotiated for you.

Offloading

Seamless workloads offloading to remote clusters, without requiring any modification to Kubernetes or the applications themselves. Multi-cluster is made native and transparent: collapse an entire remote cluster to a virtual node compliant with the standard Kubernetes approaches and tools.

Network Fabric

A transparent network fabric, enabling multi-cluster pod-to-pod and pod-to-service connectivity, regardless of the underlying configurations and CNI plugins. Natively access the services exported by remote clusters, and spread interconnected application components across multiple infrastructures, with all cross-cluster traffic flowing through secured network tunnels.

Storage Fabric

A native storage fabric, supporting the remote execution of stateful workloads according to the data gravity approach. Seamlessly extend standard (e.g., database) high availability deployment techniques to the multi-cluster scenarios, for increased guarantees. All without the complexity of managing multiple independent cluster and application replicas.

What to explore next?

Features

New to Liqo? Would you like to know more? Here you can find an in-depth overview of what a peering is, how the virtual node abstraction enables workload offloading, as well as discover about the network and storage fabric subsystems, ensuring the seamless functioning of unmodified multi-cluster applications.

Peering · Offloading · Network Fabric · Storage Fabric

Installation

Ready to give Liqo a try? Learn about installation and connectivity requirements, discover how to download and install liqoctl, the CLI tool to streamline the installation and management of Liqo, and explore the customization options, based on the target environment characteristics.

Requirements · Liqo CLI tool · Install · Uninstall

Examples

Would you like to quickly join the fray and experiment with Liqo? Set up your playground and check out the getting started examples, which will guide you through a scenario-driven tour of the most notable features of Liqo. Discover how to offload (a subset of) your workloads, access services provided by remote clusters, expose multi-cluster applications, and more.

Quick Start · Offloading with Policies · Offloading a Service · Stateful Applications · Global Ingress

Usage

Do you want to make a step further and discover all the Liqo configuration options? These guides get you covered! Find out how to establish and configure a peering between two clusters, as well as how to enable and customize namespace offloading. Explore the details about which and how native resources are reflected to remote clusters, and learn more about the support for stateful applications.

Peer two Clusters · Namespace Offloading · Resource Reflection · Stateful Applications

Peering

In Liqo, we define peering a unidirectional resource and service consumption relationship between two Kubernetes clusters, with one cluster (i.e., the consumer) granted the capability to offload tasks to a remote cluster (i.e., the provider), but not vice versa. In this case, we say that the consumer establishes an outgoing peering towards the provider, which in turn is subjected to an incoming peering from the consumer.

This configuration allows for maximum flexibility in asymmetric setups, while transparently supporting bidirectional peerings through their combination. Additionally, the same cluster can play the role of provider and consumer in multiple peerings.

Overview

Overall, the establishment of a peering relationship between two clusters involves four main tasks:

Authentication: each cluster, once properly authenticated through pre-shared tokens, obtains a valid identity to interact with the other cluster (i.e., its Kubernetes API server). This identity, granted only limited permissions concerning Liqo-related resources, is then leveraged to negotiate the necessary parameters, as well as during the offloading process.
Parameters negotiation: the two clusters exchange the set of parameters required to complete the peering establishment, including the amount of resources shared with the consumer cluster, the information concerning the setup of the network VPN tunnel, and more. The process is completely automatic and requires no user intervention.
Virtual node setup: the consumer cluster creates a new virtual node abstracting the resources shared by the provider cluster. This transparently enables the task offloading process detailed in the offloading section, and it is completely compliant with standard Kubernetes practice (i.e., it requires no API modifications for application deployment and exposition).
Network fabric setup: the two clusters configure their network fabric and establish a secure cross-cluster VPN tunnel, according to the parameters negotiated in the previous phase (endpoints, security keys, address remappings, …). Essentially, this enables pods hosted by the local cluster to seamlessly communicate with the pods offloaded to a remote cluster, regardless of the underlying CNI plugin and configuration. Additional details are presented in the network fabric section.

Approaches

Liqo supports two non-mutually exclusive peering approaches (i.e., the same cluster can leverage a different approach for different remote clusters), respectively referred to as out-of-band control plane and in-band control plane. The following sections briefly overview the differences among them, outlining the respective trade-offs. Additional in-depth details about the networking requirements are presented in the installation requirements section, while the usage section describes the operational commands to establish both types of peering.

Out-of-band control plane

The standard peering approach is referred to as out-of-band control plane, since the Liqo control plane traffic (i.e., including both the initial authentication process and the communication with the remote Kubernetes API server) flows outside the VPN tunnel interconnecting the two clusters (still, TLS is used to ensure secure communications). Indeed, this tunnel is dynamically started in a later stage of the peering process, and it is leveraged only for cross-cluster pods traffic.

The single cross-cluster traffic flow required by this approach is schematized at a high level in the figure below (agnostic from how services are exposed, which is presented in the dedicated installation requirements section).

Out-of-band control plane peering representation

Overall, the out-of-band control plane approach:

Supports clusters under the control of different administrative domains, as each party interacts only with its own cluster: the provider retrieves an authentication token that is subsequently shared with and leveraged by the consumer to start the peering process.
Is characterized by high dynamism, as upon parameters modifications (e.g., concerning VPN setup) the negotiation process ensures synchronization between clusters and the peering automatically re-converges to a stable status.
Requires each cluster to expose three different endpoints (i.e., the Liqo authentication service, the Liqo VPN endpoint and the Kubernetes API server), making them accessible from the pods running in the remote cluster.

In-band control plane

The alternative peering approach is referred to as in-band control plane, since the Liqo control plane traffic flows inside the VPN tunnel interconnecting the two clusters. In this case, the tunnel is statically established at the beginning of the peering process (i.e., part of the negotiation process is carried out directly by the Liqo CLI tool), and it is leveraged from that moment on for all inter-cluster traffic. The three different cross-cluster traffic flows required by this approach are schematized at a high level in figure below (agnostic from how services are exposed, which is presented in the dedicated installation requirements section).

In-band control plane peering representation

Overall, the in-band control plane approach:

Requires the administrator starting the peering process to have access to both clusters (although with limited permissions), as the network parameters negotiation is performed through the Liqo CLI tool (which interacts at the same time with both clusters). The remainder of the peering process, instead, is completed as usual, although the entire communication flows inside the VPN tunnel.
Statically configures the cross-cluster VPN tunnel at peering establishment time, hence requiring manual intervention in case of configuration changes causing connectivity loss.
Relaxes the connectivity requirements, as only the Liqo VPN endpoint needs to be reachable from the pods running in the remote cluster. Specifically, the Kubernetes API service is not required to be exposed outside the cluster.

Offloading

Workload offloading is enabled by the virtual node abstraction. A virtual node is spawned at the end of the peering process in the local (i.e., consumer) cluster, and represents (and aggregates) the subset of resources shared by the remote cluster. This solution enables the transparent extension of the local cluster, with the new node (and its capabilities) seamlessly taken into account by the vanilla Kubernetes scheduler when selecting the best place for the workloads execution. At the same time, this approach is fully compliant with the standard Kubernetes APIs, hence allowing to interact with and inspect offloaded pods just as if they were executed locally.

Virtual kubelet

The virtual node abstraction is implemented by an extended version of the Virtual Kubelet project. A virtual kubelet replaces a traditional kubelet when the controlled entity is not a physical node. In the context of Liqo, it interacts with both the local and the remote clusters (i.e., the respective Kubernetes API servers) to:

Create the virtual node resource and reconcile its status with respect to the negotiated configuration.
Offload the local pods scheduled onto the corresponding (virtual) node to the remote cluster, while keeping their status aligned.
Propagate and synchronize the accessory artifacts (e.g., Services, ConfigMaps, Secrets, …) required for proper execution of the offloaded workloads, a feature we call resource reflection.

For each remote cluster, a different instance of the Liqo virtual kubelet is started in the local cluster, ensuring isolation and segregating the different authentication tokens.

Virtual node

A virtual node summarizes and abstracts the amount of resources (e.g., CPU, memory, …) shared by a given remote cluster. Specifically, the virtual kubelet automatically propagates the negotiated configuration into the capacity and allocatable entries of the node status.

Node conditions reflect the current status of the node, with periodic and configurable healthiness checks performed by the virtual kubelet to assess the reachability of the remote API server. This allows to mark the node as not ready in case of repeated failures, triggering the standard Kubernetes eviction strategies based on the configured pod tolerations (e.g., to enforce service continuity).

Finally, each virtual node includes a set of characterizing labels (e.g., geographical region, underlying provider, …) suggested by the remote cluster. This enables the enforcement of fine-grained scheduling policies (e.g., through affinity constraints), in addition to playing a key role in the namespace extension process presented below.

Namespace extension

To enable seamless workload offloading, Liqo extends Kubernetes namespaces across the cluster boundaries. Specifically, once a given namespace is selected for offloading (see the namespace offloading usage section for the operational procedure), Liqo proceeds with the automatic creation of twin namespaces in the subset of selected remote clusters.

Remote namespaces host the actual pods offloaded to the corresponding cluster, as well as the additional resources propagated by the resource reflection process. This behavior is presented in the figure below, which shows a given namespace existing in the local cluster and extended to a remote cluster. A group of pods is contained in the local namespace, while a subset (i.e., those faded-out) is scheduled onto the virtual node and offloaded to the remote namespace. Additionally, the resource reflection process propagated different resources existing in the local namespace (e.g., Services, ConfigMaps, Secrets, …) in the remote one (represented faded-out), to ensure the correct execution of offloaded pods.

Representation of the namespace extension concept

The Liqo namespace extension process features a high degree of customization, mainly enabling to:

Select a specific subset of the available remote clusters, by means of standard selectors matching the label assigned to the virtual nodes.
Constraint whether pods should be scheduled onto physical nodes only, virtual nodes only, or both. The extension of a namespace, forcing at the same time all pods to be scheduled locally, enables the consumption of local services from the remote cluster, as shown in the service offloading example.
Configure whether the remote namespace name should match the local one (although possibly incurring in conflicts), or be automatically generated, such as to be unique.

Pod offloading

Once a pod is scheduled onto a virtual node, the corresponding Liqo virtual kubelet (indirectly) creates a twin pod object in the remote cluster for actual execution. Liqo supports the offloading of both stateless and stateful pods, the latter either relying on the provided storage fabric or leveraging externally managed solutions (e.g., persistent volumes provided by the cloud provider infrastructure).

Remote pod resiliency (hence, service continuity), even in case of temporary connectivity loss between the two control planes, is ensured through a custom resource (i.e., ShadowPod) wrapping the pod definition, and triggering a Liqo enforcement logic running in the remote cluster. This guarantees that the desired pod is always present, without requiring the intervention of the originating cluster.

The virtual kubelet takes care of the automatic propagation of remote status changes to the corresponding local pod (remapping the appropriate information), allowing for complete observability from the local cluster. Advanced operations, such as metrics and logs retrieval, as well as interactive command execution inside remote containers, are transparently supported, to comply with standard troubleshooting operations.

Additional details concerning how pods are propagated to remote clusters are provided in the resource reflection usage section.

Resource reflection

The resource reflection process is responsible for the propagation and synchronization of selected control plane information into remote clusters, to enable the seamless execution of offloaded pods. Liqo supports the reflection of the resources dealing with service exposition (i.e., Ingresses, Services and EndpointSlices), persistent storage (i.e., PersistentVolumeClaims and PersistentVolumes), as well as those storing configuration data (i.e., ConfigMaps and Secrets).

All resources of the above types that live in a namespace selected for offloading are automatically propagated into the corresponding twin namespaces created in the selected remote clusters. Specifically, the local copy of each resource is the source of trust leveraged to realign the content of the shadow copy reflected remotely. Appropriate remapping of certain information (e.g., endpoint IPs) is transparently performed by the virtual kubelet, accounting for conflicts and different configurations in different clusters.

You can refer to the resource reflection usage section for a detailed characterization of how the different resources are reflected into remote clusters.

Network Fabric

The network fabric is the Liqo subsystem transparently extending the Kubernetes network model across multiple independent clusters, such that offloaded pods can communicate with each other as if they were all executed locally.

In detail, the network fabric ensures that all pods in a given cluster can communicate with all pods on all remote peered clusters, either with or without NAT translation. The support for arbitrary clusters, with different parameters and components (e.g., CNI plugins), makes it impossible to guarantee non-overlapping pod IP address ranges (i.e., PodCIDR). Hence, possibly requiring address translation mechanisms, provided that NAT-less communication is preferred whenever address ranges are disjointed.

The figure below represents at a high level the network fabric established between two clusters, with its main components detailed in the following.

Network fabric representation

Network manager

The network manager (not shown in figure) represents the control plane of the Liqo network fabric. It is executed as a pod, and it is responsible for the negotiation of the connection parameters with each remote cluster during the peering process.

It features an IP Address Management (IPAM) plugin, which deals with possible network conflicts through the definition of high-level NAT rules (enforced by the data plane components). Additionally, it exposes an interface consumed by the reflection logic to handle IP addresses remapping. Specifically, this is leveraged to handle the translation of pod IPs (i.e., during the synchronization process from the remote to the local cluster), as well as during EndpointSlices reflection (i.e., propagated from the local to the remote cluster).

Cross-cluster VPN tunnels

The interconnection between peered clusters is implemented through secure VPN tunnels, made with WireGuard, which are dynamically established at the end of the peering process, based on the negotiated parameters.

Tunnels are set up by the Liqo gateway, a component of the network fabric that is executed as a privileged pod on one of the cluster nodes. Additionally, it appropriately populates the routing table, and configures, by leveraging iptables, the NAT rules requested to comply with address conflicts.

Although this component is executed in the host network, it relies on a separate network namespace and policy routing to ensure isolation and prevent conflicts with the existing Kubernetes CNI plugin. Moreover, active/standby high-availability is supported, to ensure minimum downtime in case the main replica is restarted.

In-cluster overlay network

The overlay network is leveraged to forward all traffic originating from local pods/nodes, and directed to a remote cluster, to the gateway, where it will enter the VPN tunnel. The same process occurs on the other side, with the traffic that exits from the VPN tunnel entering the overlay network to reach the node hosting the destination pod.

Liqo leverages a VXLAN-based setup, which is configured by a network fabric component executed on all physical nodes of the cluster (i.e., as a DaemonSet). Additionally, it is also responsible for the population of the appropriate routing entries to ensure correct traffic forwarding.

Storage Fabric

The Liqo storage fabric subsystem enables the seamless offloading of stateful workloads to remote clusters. The solution is based on two main pillars:

Storage binding deferral until its first consumer is scheduled onto a given cluster (either local or remote). This ensures that new storage pools are created in the exact location where their associated pods have just been scheduled onto for execution.
Data gravity, entering in action in the subsequent scheduling processes, and involving a set of automatic policies to attract pods in the appropriate cluster. This guarantees that pods requesting existing pools of storage (e.g., following a restart) are scheduled onto the cluster physically hosting the corresponding data.

These approaches extend standard Kubernetes practice to multi-cluster scenarios, simplifying at the same time the configuration of high availability and disaster recovery scenarios. To this end, one relevant use-case is represented by database instances that need to be replicated and synchronized across different clusters, which is shown in the stateful applications example.

Under the hood, the Liqo storage fabric leverages a virtual storage class, which embeds the logic to create the appropriate storage pools in the different clusters. Whenever a new PersistentVolumeClaim (PVC) associated with the virtual storage class is created, and its consumer is bound to a (possibly virtual) node, the Liqo logic goes into action, based on the target node:

If it is a local node, PVC operations are remapped to a second one, associated with the corresponding real storage class, to transparently provision the requested volume.
In case of virtual nodes, the reflection logic is responsible for creating the remote shadow PVC, remapped to the negotiated storage class, and synchronizing the PersistentVolume information, to allow pod binding.

In both cases, locality constraints are automatically embedded within the resulting PersistentVolumes (PVs), to make sure each pod is scheduled only onto the cluster where the associated storage pools are available.

Additional details about the configuration of the Liqo storage fabric, as well as concerning the possibility to move storage pools among clusters through the Liqo CLI tool, are presented in the stateful applications usage section.

Note

In addition to the provided storage class, Liqo supports the execution of pods leveraging cross-cluster storage managed by external solutions (e.g., persistent volumes provided by the cloud provider infrastructure).

Requirements

Before starting the tutorials below, you should ensure the following software is installed on your system:

Docker, the container runtime.
Kubectl, the command-line tool for Kubernetes.
Helm, the package manager for Kubernetes.
curl, to interact with the tutorial applications through HTTP/HTTPS.
KinD, the Kubernetes in Docker runtime.
liqoctl command-line tool to interact with Liqo.

The following tutorials were tested on Linux, macOS, and Windows (WSL2 and Docker Desktop).

Warning

To prevent issues with tutorials leveraging more than two clusters, on some systems you may need to increase the maximum number of inotify watches:

sudo sysctl fs.inotify.max_user_watches=52428899
sudo sysctl fs.inotify.max_user_instances=2048

Quick Start

This tutorial aims at presenting how to install Liqo and practicing with its most notable capabilities. You will learn how to create a virtual cluster by peering two Kubernetes clusters and how to deploy a simple application on it.

Provision the playground

First, check that you are compliant with the requirements.

Then, let’s open a terminal on your machine and launch the following script, which creates a pair of clusters with KinD. Each cluster is made by two nodes (one for the control plane and one as a simple worker):

git clone https://github.com/liqotech/liqo.git
cd liqo
git checkout v0.5.2
cd examples/quick-start
./setup.sh

Explore the playground

You can inspect the deployed clusters by typing:

kind get clusters

You should see a couple of entries:

milan
rome

This means that two KinD clusters are deployed and running on your host.

Then, you can simply inspect the status of the clusters. To do so, you can export the KUBECONFIG variable to specify the identity file for kubectl and liqoctl, and then contact the cluster.

By default, the kubeconfigs of the two clusters are stored in the current directory (./liqo_kubeconf_rome, ./liqo_kubeconf_milan). You can export the appropriate environment variables leveraged for the rest of the tutorial (i.e., KUBECONFIG and KUBECONFIG_MILAN), and referring to their location, through the following:

export KUBECONFIG="$PWD/liqo_kubeconf_rome"
export KUBECONFIG_MILAN="$PWD/liqo_kubeconf_milan"

Note

We suggest exporting the kubeconfig of the first cluster as default (i.e., KUBECONFIG), since it will be the entry point of the virtual cluster and you will mainly interact with it.

On the first cluster, you can get the available pods by merely typing:

kubectl get pods -A

Similarly, on the second cluster, you can observe the pods in execution:

kubectl get pods -A --kubeconfig "$KUBECONFIG_MILAN"

If the above commands return each an output similar to the following, your clusters are up and ready.

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-558bd4d5db-9vdr9                     1/1     Running   0          3m58s
kube-system          coredns-558bd4d5db-tzdxg                     1/1     Running   0          3m58s
kube-system          etcd-rome-control-plane                      1/1     Running   0          4m10s
kube-system          kindnet-fcspl                                1/1     Running   0          3m58s
kube-system          kindnet-q6qkm                                1/1     Running   0          3m42s
kube-system          kube-apiserver-rome-control-plane            1/1     Running   0          4m10s
kube-system          kube-controller-manager-rome-control-plane   1/1     Running   0          4m11s
kube-system          kube-proxy-2c9bl                             1/1     Running   0          3m42s
kube-system          kube-proxy-7nngv                             1/1     Running   0          3m58s
kube-system          kube-scheduler-rome-control-plane            1/1     Running   0          4m11s
local-path-storage   local-path-provisioner-85494db59d-skd55      1/1     Running   0          3m58s

Install Liqo

You will now install Liqo on both clusters, using the following characterizing names:

rome: the local cluster, where you will deploy and control the applications.
milan: the remote cluster, where part of your workloads will be offloaded to.

You can install Liqo on the Rome cluster by launching:

liqoctl install kind --cluster-name rome

This command will generate the suitable configuration for your KinD cluster and then install Liqo.

Similarly, you can install Liqo on the Milan cluster by launching:

liqoctl install kind --cluster-name milan --kubeconfig "$KUBECONFIG_MILAN"

On both clusters, you should see the following output:

 INFO  Kubernetes clients successfully initialized
 INFO  Installer initialized
 INFO  Cluster configuration correctly retrieved
 INFO  Installation parameters correctly generated
 INFO  All Set! You can now proceed establishing a peering (liqoctl peer --help for more information)

And the Liqo pods should be up and running:

kubectl get pods -n liqo

NAME                                       READY   STATUS    RESTARTS   AGE
liqo-auth-74c795d84c-x2p6h                 1/1     Running   0          2m8s
liqo-controller-manager-6c688c777f-4lv9d   1/1     Running   0          2m8s
liqo-crd-replicator-6c64df5457-bq4tv       1/1     Running   0          2m8s
liqo-discovery-546fd594dc-kg6db            1/1     Running   0          2m8s
liqo-gateway-78cf7bb86b-pkdpt              1/1     Running   0          2m8s
liqo-metric-agent-5667b979c7-snmdg         1/1     Running   0          2m8s
liqo-network-manager-5b5cdcfcf7-scvd9      1/1     Running   0          2m8s
liqo-proxy-6674dd7bbd-kr2ls                1/1     Running   0          2m8s
liqo-route-7wsrx                           1/1     Running   0          2m8s
liqo-route-sz75m                           1/1     Running   0          2m8s
liqo-webhook-7bb6dbf88d-2hvwx              1/1     Running   0          2m8s

Peer two clusters

Once Liqo is installed in your clusters, you can establish new peerings. In this example, since the two API Servers are mutually reachable, you will use the out-of-band peering approach.

First, get the peer command from the Milan cluster:

liqoctl generate peer-command --kubeconfig "$KUBECONFIG_MILAN"

Second, copy and paste the command in the Rome cluster:

liqoctl peer out-of-band milan --auth-url [redacted] --cluster-id [redacted] --auth-token [redacted]

Now, the Liqo control plane in the Rome cluster will contact the provided authentication endpoint providing the token to the Milan cluster to get its Kubernetes identity.

You can check the peering status by running:

kubectl get foreignclusters

The output should look like the following, indicating that the cross-cluster network tunnel has been established, and an outgoing peering is currently active (i.e., the Rome cluster can offload workloads to the Milan one, but not vice versa):

NAME    OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION   AGE
milan   Established        None               Established   Established      12s

At the same time, you should see a virtual node (liqo-milan) in addition to your physical nodes:

kubectl get nodes

NAME                 STATUS   ROLES                  AGE     VERSION
liqo-milan           Ready    agent                  14s     v1.23.6
rome-control-plane   Ready    control-plane,master   7m56s   v1.23.6
rome-worker          Ready    <none>                 7m25s   v1.23.6

Leverage remote resources

Now, you can deploy a standard Kubernetes application in a multi-cluster environment as you would do in a single cluster scenario (i.e. no modification is required).

Start a hello world application

If you want to deploy an application that is scheduled onto Liqo virtual nodes, you should first create a namespace where your pod will be started. Then tell Liqo to make this namespace eligible for the pod offloading.

kubectl create namespace liqo-demo
liqoctl offload namespace liqo-demo

The liqoctl offload namespace command enables Liqo to offload the namespace to the remote cluster. Since no further configuration is provided, Liqo will add a suffix to the namespace name to make it unique on the remote cluster (see the dedicated usage page for additional information concerning namespace offloading configurations).

Note

The virtual nodes have a taint that prevents the pods from being scheduled on them. The Liqo webhook will add the toleration for this taint to the pods created in the liqo-enabled namespaces.

Then, you can deploy a demo application in the liqo-demo namespace of the local cluster:

kubectl apply -f ./manifests/hello-world.yaml -n liqo-demo

The hello-world.yaml file represents a simple nginx service. It contains two pods running an nginx image and a Service exposing the pods to the cluster. One pods is running in the local cluster, while the other is forced to be scheduled on the remote cluster.

Info

Differently from the traditional examples, the above deployment introduces an affinity constraint. This forces Kubernetes to schedule the first pod (i.e. nginx-local) on a physical node and the second (i.e. nginx-remote) on a virtual node. Virtual nodes are like traditional Kubernetes nodes, but they represent remote clusters and have the liqo.io/type: virtual-node label.

When the affinity constraint is not specified, the Kubernetes scheduler selects the best hosting node based on the available resources. Hence, each pod can be scheduled either in the local cluster or in the remote cluster.

Now you can check the status of the pods. The output should be similar to the one below, confirming that one nginx pod is running locally; while the other is hosted by the virtual node (i.e., liqo-milan).

kubectl get pod -n liqo-demo -o wide

And the output should look like this:

NAME           READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
nginx-local    1/1     Running   0          94s   10.200.1.10   rome-worker   <none>           <none>
nginx-remote   1/1     Running   0          94s   10.202.1.9    liqo-milan    <none>           <none>

Check the pod connectivity

Once both pods are correctly running, it is possible to check one of the abstractions introduced by Liqo. Indeed, Liqo enables each pod to be transparently contacted by every other pod and physical node (according to the Kubernetes model), regardless of whether it is hosted by the local or by the remote cluster.

First, let’s retrieve the IP address of the nginx pods:

LOCAL_POD_IP=$(kubectl get pod nginx-local -n liqo-demo --template={{.status.podIP}})
REMOTE_POD_IP=$(kubectl get pod nginx-remote -n liqo-demo --template={{.status.podIP}})
echo "Local Pod IP: ${LOCAL_POD_IP} - Remote Pod IP: ${REMOTE_POD_IP}"

You can fire up a pod and run curl from inside the cluster:

kubectl run --image=curlimages/curl curl -n default -it --rm --restart=Never -- curl ${LOCAL_POD_IP}
kubectl run --image=curlimages/curl curl -n default -it --rm --restart=Never -- curl ${REMOTE_POD_IP}

Both commands should lead to a successful outcome (i.e., return a demo web page), regardless of whether each pod is executed locally or remotely.

Expose the pods through a Service

The above hello-world.yaml manifest additionally creates a Service which is designed to serve traffic to the previously deployed pods. This is a traditional Kubernetes Service and can work with Liqo with no modifications.

Indeed, inspecting the Service, it is possible to observe that both nginx pods are correctly specified as endpoints. Nonetheless, it is worth noticing that the first endpoint (i.e. 10.200.1.10:80 in this example) refers to a pod running in the local cluster, while the second one (i.e. 10.202.1.9:80) points to a pod hosted by the remote cluster.

kubectl describe service liqo-demo -n liqo-demo

Name:              liqo-demo
Namespace:         liqo-demo
Labels:            <none>
Annotations:       <none>
Selector:          app=liqo-demo
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.93.84.150
IPs:               10.93.84.150
Port:              web  80/TCP
TargetPort:        web/TCP
Endpoints:         10.200.1.10:80,10.202.1.9:80
Session Affinity:  None
Events:            <none>

Check the Service connectivity

It is now possible to contact the Service: as usual, Kubernetes will forward the HTTP request to one of the available back-end pods. Additionally, all traditional mechanisms still work seamlessly (e.g. DNS discovery), even though one of the pods is actually running in a remote cluster.

You can fire up a pod and run curl from inside the cluster:

kubectl run --image=curlimages/curl curl -n default -it --rm --restart=Never -- \
    curl --silent liqo-demo.liqo-demo.svc.cluster.local | grep 'Server'

Note

Executing the previous command multiple times, you will observe that part of the requests are answered by the pod running in the local cluster, and in part by that in the remote cluster (i.e., the Server value changes).

Play with a microservice application

It is very common in a cloud-based environment to deploy microservices applications composed of many pods interacting among each other. This pattern is transparently supported by Liqo and the virtual cluster abstraction.

You can play with a microservices application provided by Google, which includes multiple cooperating Services leveraging different networking protocols:

kubectl apply -k ./manifests/demo-application -n liqo-demo

By default, Kubernetes schedules each pod either in the local or in the remote cluster, optimizing each deployment based on the available resources. However, you can play with affinity constraints to force Kubernetes to schedule of each component in a specific location, and see that everything continues to work smoothly. Specifically, the manifest above forces the frontend component to be executed in the local cluster, as this is required to enable port-forwarding, which is leveraged below.

Each demo component is exposed as a Service and accessed by other components. However, given that nobody knows, a priori, where each Service will be deployed (either locally or in the remote cluster), Liqo replicates all Kubernetes Services across both clusters, although the corresponding pod may be running only in one location. Hence, each microservice deployed across clusters can reach the others seamlessly: independently of the cluster a pod is deployed in, each pod can contact other Services and leverage the traditional Kubernetes discovery mechanisms (e.g., DNS discovery and environment variables).

Additionally, several other objects (e.g. ConfigMaps and Secrets) inside a namespace are replicated in the remote cluster within the twin namespace, thus, ensuring that complex applications can work seamlessly across clusters.

Observe the application deployment

Once the demo application manifest is applied, you can observe the creation of the different pods:

watch kubectl get pods -n liqo-demo -o wide

At steady-state, you should see an output similar to the following. Different pods may be hosted by either the local nodes (rome-worker in the example below) or remote cluster (liqo-milan in the example below), depending on the scheduling decisions.

NAME                                     READY   STATUS    RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
adservice-66f6b5c6fd-w95th               1/1     Running   0          2m23s   10.202.1.19   liqo-milan    <none>           <none>
cartservice-76dc758684-wd5px             1/1     Running   0          2m23s   10.202.1.15   liqo-milan    <none>           <none>
checkoutservice-85b74f746f-lm7gh         1/1     Running   0          2m24s   10.202.1.11   liqo-milan    <none>           <none>
currencyservice-64775746dd-k85z6         1/1     Running   0          2m23s   10.202.1.16   liqo-milan    <none>           <none>
emailservice-58f8b4f854-9mx7g            1/1     Running   0          2m24s   10.202.1.10   liqo-milan    <none>           <none>
frontend-7b648dcb8f-gfbh2                1/1     Running   0          2m23s   10.200.1.15   rome-worker   <none>           <none>
nginx-local                              1/1     Running   0          6m5s    10.200.1.10   rome-worker   <none>           <none>
nginx-remote                             1/1     Running   0          6m5s    10.202.1.9    liqo-milan    <none>           <none>
paymentservice-5dd7bb5855-ssbhf          1/1     Running   0          2m23s   10.202.1.14   liqo-milan    <none>           <none>
productcatalogservice-587c8dbf7d-nmw77   1/1     Running   0          2m23s   10.202.1.13   liqo-milan    <none>           <none>
recommendationservice-6cd468f4d4-h8k2t   1/1     Running   0          2m24s   10.202.1.12   liqo-milan    <none>           <none>
redis-cart-78746d49dc-rkqrn              1/1     Running   0          2m23s   10.202.1.18   liqo-milan    <none>           <none>
shippingservice-59c7b7458d-sqb9x         1/1     Running   0          2m23s   10.202.1.17   liqo-milan    <none>           <none>

Access the demo application

Once the deployment is up and running, you can start using the demo application and verify that everything works correctly, even if its components are distributed across multiple Kubernetes clusters.

By default, the frontend web-page is exposed through a LoadBalancer Service, which can be inspected using:

kubectl get service -n liqo-demo frontend-external

Leverage kubectl port-forward to forward the requests from your local machine (i.e., http://localhost:8080) to the frontend Service:

kubectl port-forward -n liqo-demo service/frontend-external 8080:80

Open the http://localhost:8080 page in your browser and enjoy the demo application.

Tear down the playground

Unoffload namespaces

Before starting the uninstallation process, make sure that all namespaces are unoffloaded:

liqoctl unoffload namespace liqo-demo

Every pod that was offloaded to a remote cluster is going to be rescheduled onto the local cluster.

Revoke peerings

Similarly, make sure that all the peerings are revoked:

liqoctl unpeer out-of-band milan

At the end of the process, the virtual node is removed from the local cluster.

Uninstall Liqo

Now you can uninstall Liqo from your clusters with liqoctl:

liqoctl uninstall
liqoctl uninstall --kubeconfig="$KUBECONFIG_MILAN"

Purge

By default the Liqo CRDs will remain in the cluster, but they can be removed with the --purge flag:

liqoctl uninstall --purge
liqoctl uninstall --purge --kubeconfig="$KUBECONFIG_MILAN"

Destroy clusters

To teardown the KinD clusters, you can issue:

kind delete cluster --name rome
kind delete cluster --name milan

Offloading with Policies

This tutorial aims to guide you through a tour to learn how to use the core Liqo features. You will learn how to tune namespace offloading, and specify the target clusters through the cluster selector concept.

More specifically, you will configure a scenario composed of a single entry point cluster leveraged for the deployment of the applications (i.e., the Venice cluster, located in north Italy) and two worker clusters characterized by different geographical regions (i.e., the Florence and Naples clusters, respectively located in center and south Italy). Then, you will offload a given namespace (and the applications contained therein) to a subset of the worker clusters (i.e., only to the Naples cluster), while allowing pods to be also scheduled on the local cluster (i.e., the Venice one).

Provision the playground

First, check that you are compliant with the requirements.

Then, let’s open a terminal on your machine and launch the following script, which creates the three above-mentioned clusters with KinD and installs Liqo on all of them. Each cluster is made by a single combined control-plane + worker node.

git clone https://github.com/liqotech/liqo.git
cd liqo
git checkout v0.5.2
cd examples/offloading-with-policies
./setup.sh

Export the kubeconfigs environment variables to use them in the rest of the tutorial:

export KUBECONFIG="$PWD/liqo_kubeconf_venice"
export KUBECONFIG_FLORENCE="$PWD/liqo_kubeconf_florence"
export KUBECONFIG_NAPLES="$PWD/liqo_kubeconf_naples"

Note

We suggest exporting the kubeconfig of the first cluster as default (i.e., KUBECONFIG), since it will be the entry point of the virtual cluster and you will mainly interact with it.

At this point, you should have three clusters with Liqo installed on them. The setup script named them venice, florence and naples, and respectively configured the following cluster labels:

venice: topology.liqo.io/region=north
florence: topology.liqo.io/region=center
naples: topology.liqo.io/region=south

You can check that the clusters are correctly labeled through:

liqoctl status
liqoctl --kubeconfig $KUBECONFIG_FLORENCE status
liqoctl --kubeconfig $KUBECONFIG_NAPLES status

These labels will be propagated to the virtual nodes corresponding to each cluster. In this way, you can easily identify the clusters through their characterizing labels, and define the appropriate scheduling policies.

Peer the clusters

Once Liqo is installed in your clusters, you can establish new peerings. In this example, since the two API Servers are mutually reachable, you will use the out-of-band peering approach.

To implement the desired scenario, let’s first retrieve the peer command from the Florence and Naples clusters:

PEER_FLORENCE=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_FLORENCE)
PEER_NAPLES=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_NAPLES)

Then, establish the peerings from the Venice cluster:

echo "$PEER_FLORENCE" | bash
echo "$PEER_NAPLES" | bash

When the above commands return successfully, you can check the peering status by running:

kubectl get foreignclusters

The output should look like the following, indicating that an outgoing peering is currently active towards both the Florence and the Naples clusters, as well as the cross-cluster network tunnels have been established:

NAME       OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION   AGE
florence   Established        None               Established   Established      111s
naples     Established        None               Established   Established      98s

Additionally, you should have two new virtual nodes in the Venice cluster, characterized by the install-time provided labels:

kubectl get node --selector=liqo.io/type=virtual-node --show-labels

NAME            STATUS   ROLES   AGE     VERSION   LABELS
liqo-florence   Ready    agent   2m30s   v1.23.6   liqo.io/remote-cluster-id=4858caa2-7848-4aab-a6b4-6984389bac56,liqo.io/type=virtual-node,topology.liqo.io/region=center
liqo-naples     Ready    agent   2m29s   v1.23.6   liqo.io/remote-cluster-id=53d11339-fc85-4741-a912-a3eda781a69e,liqo.io/type=virtual-node,topology.liqo.io/region=south

Note

Some of the default labels were omitted for the sake of clarity.

Tune namespace offloading

Now, let’s suppose you want to deploy an application that needs to be scheduled in the north and in the south region, but not in the center one. This constraint needs to be respected at the infrastructural level: the dev team does not need to be aware of required affinities and/or node selectors, nor it should be able to bypass them.

First, you should create a new namespace in the Venice cluster, which will host the application:

kubectl create namespace liqo-demo

Then, enable Liqo offloading for that namespace:

liqoctl offload namespace liqo-demo \
  --namespace-mapping-strategy EnforceSameName \
  --pod-offloading-strategy LocalAndRemote \
  --selector 'topology.liqo.io/region=south'

The above command configures the following aspects (see the dedicated usage page for additional information concerning namespace offloading configurations):

the liqo-demo namespace is replicated with the same name in the other clusters.
the liqo-demo namespace, and the contained resources, are offloaded only to the clusters with the topology.liqo.io/region=south label.
the pods living in the liqo-demo namespace are free to be scheduled onto both physical and virtual nodes.

The NamespaceOffloading resource created by liqoctl in the liqo-demo namespace exposes the status of the offloading process, including a global OffloadingPhase, which is expected to be Ready, and a list of RemoteNamespaceConditions, one for each remote cluster.

In this case:

the Florence cluster has not been selected to offload the namespace liqo-demo, since it does not match the cluster selector;
the Naples cluster has been selected to offload the namespace liqo-demo, and the namespace has been correctly created.

kubectl get namespaceoffloadings offloading -n liqo-demo -o yaml

...
status:
  offloadingPhase: Ready
  remoteNamespaceName: liqo-demo
  remoteNamespacesConditions:
    florence-539f8b:
    - lastTransitionTime: "2022-05-05T15:08:33Z"
      message: You have not selected this cluster through ClusterSelector fields
      reason: ClusterNotSelected
      status: "False"
      type: OffloadingRequired
    naples-7881be:
    - lastTransitionTime: "2022-05-05T15:08:43Z"
      message: The remote cluster has been selected through the ClusterSelector field
      reason: ClusterSelected
      status: "True"
      type: OffloadingRequired
    - lastTransitionTime: "2022-05-05T15:08:43Z"
      message: Namespace correctly offloaded on this cluster
      reason: RemoteNamespaceCreated
      status: "True"
      type: Ready

Indeed, if you query for the namespaces in the Naples cluster, you should see the following output, confirming that the remote namespace has been correctly created by Liqo:

kubectl get namespaces liqo-demo --kubeconfig="$KUBECONFIG_NAPLES"

NAME        STATUS   AGE
liqo-demo   Active   70s

Instead, the same command executed in the Florence cluster should return an error, as the namespace has not been replicated:

kubectl get namespaces liqo-demo --kubeconfig="$KUBECONFIG_FLORENCE"

Error from server (NotFound): namespaces "liqo-demo" not found

Deploy applications

All constraints specified during namespace offloading are automatically enforced by Liqo, and merged with other pod-level specifications.

To verify this, you can now create two deployments in the liqo-demo namespace, characterized by additional NodeAffinity constraints. More precisely, one (app-south) is forced to be scheduled onto the virtual node representing the Naples cluster, while the other (app-center) is forced onto the Florence virtual cluster (which is incompatible with the namespace-level constraints).

kubectl apply -f ./manifests/deploy.yaml -n liqo-demo

Checking the pod status, it is possible to verify that one has been scheduled onto the Naples cluster, and it is correctly running, while the other remained Pending due to conflicting requirements (i.e., no node is available to satisfy all its constraints).

kubectl get pod -n liqo-demo -o wide

NAME                          READY   STATUS    RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
app-center-6f6bd7547b-dh4m4   0/1     Pending   0          2m30s   <none>        <none>        <none>           <none>
app-south-85b9b99c4d-stwhw    1/1     Running   0          2m30s   10.204.0.12   liqo-naples   <none>           <none>

Note

You can remove the conflicting node affinity from the app-center deployment, and check that the generated pod gets scheduled onto either the Venice (i.e., locally) or the Naples cluster, as constrained by the namespace offloading configuration.

Tear down the playground

Unoffload namespaces

Before starting the uninstallation process, make sure that all namespaces are unoffloaded:

liqoctl unoffload namespace liqo-demo

Every pod that was offloaded to a remote cluster is going to be rescheduled onto the local cluster.

Revoke peerings

Similarly, make sure that all the peerings are revoked:

liqoctl unpeer out-of-band florence
liqoctl unpeer out-of-band naples

At the end of the process, the virtual nodes are removed from the local cluster.

Uninstall Liqo

Now you can uninstall Liqo from your clusters with liqoctl:

liqoctl uninstall
liqoctl uninstall --kubeconfig="$KUBECONFIG_FLORENCE"
liqoctl uninstall --kubeconfig="$KUBECONFIG_NAPLES"

Purge

By default the Liqo CRDs will remain in the cluster, but they can be removed with the --purge flag:

liqoctl uninstall --purge
liqoctl uninstall --kubeconfig="$KUBECONFIG_FLORENCE" --purge
liqoctl uninstall --kubeconfig="$KUBECONFIG_NAPLES" --purge

Destroy clusters

To teardown the KinD clusters, you can issue:

kind delete cluster --name venice
kind delete cluster --name florence
kind delete cluster --name naples

Offloading a Service

In this tutorial you will learn how to create a multi-cluster Service and how to consume it from each connected cluster.

Specifically, you will deploy an application in a first cluster (London) and then offload the corresponding Service and transparently consume it from a second cluster (New York).

Provision the playground

First, check that you are compliant with the requirements.

Then, let’s open a terminal on your machine and launch the following script, which creates the two above-mentioned clusters with KinD and installs Liqo on them. Each cluster is made by a single combined control-plane + worker node.

git clone https://github.com/liqotech/liqo.git
cd liqo
git checkout v0.5.2
cd examples/service-offloading
./setup.sh

Export the kubeconfigs environment variables to use them in the rest of the tutorial:

export KUBECONFIG="$PWD/liqo_kubeconf_london"
export KUBECONFIG_NEWYORK="$PWD/liqo_kubeconf_newyork"

Note

We suggest exporting the kubeconfig of the first cluster as default (i.e., KUBECONFIG), since it will be the entry point of the virtual cluster and you will mainly interact with it.

At this point, you should have two clusters with Liqo installed on them. The setup script named them london and newyork.

Peer the clusters

Once Liqo is installed in your clusters, you can establish new peerings. In this example, since the two API Servers are mutually reachable, you will use the out-of-band peering approach.

To implement the desired scenario, let’s first retrieve the peer command from the New York cluster:

PEER_NEW_YORK=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_NEWYORK)

Then, establish the peering from the London cluster:

echo "$PEER_NEW_YORK" | bash

When the above command returns successfully, you can check the peering status by running:

kubectl get foreignclusters

The output should look like the following, indicating that an outgoing peering is currently active towards the New York cluster, as well as that the cross-cluster network tunnel has been established:

NAME      OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION   AGE
newyork   Established        None               Established   Established      61s

Offload a service

Now, let’s deploy a simple application composed of a Deployment and a Service in the London cluster.

First, you should create a hosting namespace in the London cluster:

kubectl create namespace liqo-demo

Then, deploy the application in the London cluster:

kubectl apply -f manifests/app.yaml -n liqo-demo

At this moment, you have an HTTP application serving JSON data through a Service, and running in the London cluster (i.e., locally). If you look at the New York cluster, you will not see the application yet.

To make it visible, you need to enable the Liqo offloading of the Services in the desired namespace to the New York cluster:

liqoctl offload namespace liqo-demo \
    --namespace-mapping-strategy EnforceSameName \
    --pod-offloading-strategy Local

This command enables the offloading of the Services in the London cluster to the New York cluster and sets:

the namespace mapping strategy to EnforceSameName, which means that the namespace in the remote cluster is created with the same name as of the local one. This is particularly useful when you want to consume the Services in the remote cluster using the Kubernetes DNS service discovery (i.e. with svc-name.namespace-name.svc.cluster.local).
the pod offloading strategy to Local, which means that the pods running in this namespace will be kept local and not scheduled on virtual nodes (i.e., no pod is offloaded to remote clusters).

Refer to the dedicated usage page for additional information concerning namespace offloading configurations.

Some seconds later, you should see that the Service has been replicated by the resource reflection process, and is now available in the New York cluster:

kubectl get services --namespace liqo-demo --kubeconfig "$KUBECONFIG_NEWYORK"

NAME              TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE   SELECTOR
flights-service   ClusterIP   10.95.177.4   <none>        7999/TCP   14s   run=flights-service

The Service is characterized by a different ClusterIP address in the two clusters, since each cluster handles it independently. Additionally, you can also check that there is no application pod running in the New York cluster:

kubectl get pods --namespace liqo-demo --kubeconfig "$KUBECONFIG_NEWYORK"

No resources found in liqo-demo namespace.

Consume the service

Let’s now consume the Service from both clusters from a different pod (e.g., a temporary shell).

Starting from the London cluster:

kubectl run consumer --rm -i --tty --image dwdraju/alpine-curl-jq -- /bin/sh

When the shell is ready, you can access the Service with curl:

curl -s -H 'accept: application/json' http://flights-service.liqo-demo:7999/schedule | jq .

A similar result is obtained executing the same command in a shell running in the New York cluster, although the backend pod is effectively running in the London cluster:

kubectl run consumer --rm -i --tty --image dwdraju/alpine-curl-jq \
    --kubeconfig $KUBECONFIG_NEWYORK -- /bin/sh

curl -s -H 'accept: application/json' http://flights-service.liqo-demo:7999/schedule | jq .

This quick example demonstrated how Liqo can upgrade ClusterIP Services to multi-cluster Services, allowing your local pods to transparently serve traffic originating from remote clusters with no additional configuration neither in the local cluster and/or applications nor in the remote ones.

Tear down the playground

Unoffload namespaces

Before starting the uninstallation process, make sure that all namespaces are unoffloaded:

liqoctl unoffload namespace liqo-demo

Every pod that was offloaded to a remote cluster is going to be rescheduled onto the local cluster.

Revoke peerings

Similarly, make sure that all the peerings are revoked:

liqoctl unpeer out-of-band newyork

At the end of the process, the virtual node is removed from the local cluster.

Uninstall Liqo

Now you can uninstall Liqo from your clusters with liqoctl:

liqoctl uninstall
liqoctl uninstall --kubeconfig="$KUBECONFIG_NEWYORK"

Purge

By default the Liqo CRDs will remain in the cluster, but they can be removed with the --purge flag:

liqoctl uninstall --purge
liqoctl uninstall --kubeconfig="$KUBECONFIG_NEWYORK" --purge

Destroy clusters

To teardown the KinD clusters, you can issue:

kind delete cluster --name london
kind delete cluster --name newyork

Stateful Applications

This tutorial demonstrates how to use the core Liqo features to deploy stateful applications. In particular, you will deploy a multi-master mariadb-galera database across a multi-cluster environment (composed of two clusters, respectively identified as Turin and Lyon), hence replicating the data in multiple regions.

Provision the playground

First, check that you are compliant with the requirements.

Then, let’s open a terminal on your machine and launch the following script, which creates the two above-mentioned clusters with KinD and installs Liqo on them. Each cluster is made by a single combined control-plane + worker node.

git clone https://github.com/liqotech/liqo.git
cd liqo
git checkout v0.5.2
cd examples/stateful-applications
./setup.sh

Export the kubeconfigs environment variables to use them in the rest of the tutorial:

export KUBECONFIG="$PWD/liqo_kubeconf_turin"
export KUBECONFIG_LYON="$PWD/liqo_kubeconf_lyon"

Note

The install script creates two clusters with no overlapping pod CIDRs. This is required by the mariadb-galera application to work correctly. Given it needs to know the real IP of the connected masters, it will not work correctly when natting is enabled.

Note

We suggest exporting the kubeconfig of the first cluster as default (i.e., KUBECONFIG), since it will be the entry point of the virtual cluster and you will mainly interact with it.

Peer the clusters

Once Liqo is installed in your clusters, you can establish new peerings. In this example, since the two API Servers are mutually reachable, you will use the out-of-band peering approach.

To implement the desired scenario, let’s first retrieve the peer command from the Lyon cluster:

PEER_LYON=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_LYON)

Then, establish the peering from the Turin cluster:

echo "$PEER_LYON" | bash

When the above command returns successfully, you can check the peering status by running:

kubectl get foreignclusters

The output should look like the following, indicating that an outgoing peering is currently active towards the Lyon cluster,, as well as that the cross-cluster network tunnel has been established:

NAME   OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION   AGE
lyon   Established        None               Established   Established      1m28s

Deploy a stateful application

In this step, you will deploy a mariadb-galera database using the Bitnami helm chart.

First, you need to add the helm repository:

helm repo add bitnami https://charts.bitnami.com/bitnami

Then, create the namespace and offload it to remote clusters:

kubectl create namespace liqo-demo
liqoctl offload namespace liqo-demo --namespace-mapping-strategy EnforceSameName

This command will create a twin liqo-demo namespace in the Lyon cluster. Refer to the dedicated usage page for additional information concerning namespace offloading configurations.

Now, deploy the helm chart:

helm install db bitnami/mariadb-galera -n liqo-demo -f manifests/values.yaml

The release is configured to:

have two replicas;
spread the replicas across the cluster (i.e., a hard pod anti-affinity is set);
use the liqo virtual storage class.

Check that these constraints are met by typing:

kubectl get pods -n liqo-demo -o wide

After a while (the startup process might require a few minutes), you should see two replicas of a StatefulSet spread over two different nodes (one local and one remote).

NAME                  READY   STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
db-mariadb-galera-0   1/1     Running   0          3m13s   10.210.0.15   liqo-lyon             <none>           <none>
db-mariadb-galera-1   1/1     Running   0          2m6s    10.200.0.17   turin-control-plane   <none>           <none>

Consume the database

When the database is up and running, check that it is operating as expected executing a simple SQL client in your cluster:

kubectl run db-mariadb-galera-client --rm --tty -i \
    --restart='Never' --namespace default \
    --image docker.io/bitnami/mariadb-galera:10.6.7-debian-10-r56 \
    --command \
      -- mysql -h db-mariadb-galera.liqo-demo -uuser -ppassword my_database

And then create an example table and insert some data:

CREATE TABLE People (
    PersonID int,
    LastName varchar(255),
    FirstName varchar(255),
    Address varchar(255),
    City varchar(255)
);

INSERT INTO People
    (PersonID, LastName, FirstName, Address, City)
    VALUES
    (1, 'Smith', 'John', '123 Main St', 'Anytown');

You are now able to query the database and grab the data:

SELECT * FROM People;

+----------+----------+-----------+-------------+---------+
| PersonID | LastName | FirstName | Address     | City    |
+----------+----------+-----------+-------------+---------+
|        1 | Smith    | John      | 123 Main St | Anytown |
+----------+----------+-----------+-------------+---------+
1 row in set (0.000 sec)

Database failures toleration

With this setup the applications running on a cluster can tolerate failures of the local database replica.

This can be checked deleting one of the replicas:

kubectl delete pod db-mariadb-galera-0 -n liqo-demo

And querying again for your data:

kubectl run db-mariadb-galera-client --rm --tty -i \
    --restart='Never' --namespace default \
    --image docker.io/bitnami/mariadb-galera:10.6.7-debian-10-r56 \
    --command \
      -- mysql -h db-mariadb-galera.liqo-demo -uuser -ppassword my_database \
      --execute "SELECT * FROM People;"

Pro-tip

Try deleting the other replica and query again.

NOTE: at least one of the two replicas should be always running, be careful deleting all of them.

Note

You can run exactly the same commands to query the data from the other cluster, and you will get the same results.

Tear down the playground

Unoffload namespaces

Before starting the uninstallation process, make sure that all namespaces are unoffloaded:

liqoctl unoffload namespace liqo-demo

Every pod that was offloaded to a remote cluster is going to be rescheduled onto the local cluster.

Revoke peerings

Similarly, make sure that all the peerings are revoked:

liqoctl unpeer out-of-band lyon

At the end of the process, the virtual node is removed from the local cluster.

Uninstall Liqo

Now you can remove Liqo from your clusters with liqoctl:

liqoctl uninstall
liqoctl uninstall --kubeconfig="$KUBECONFIG_LYON"

Purge

By default the Liqo CRDs will remain in the cluster, but they can be removed with the --purge flag:

liqoctl uninstall --purge
liqoctl uninstall --kubeconfig="$KUBECONFIG_LYON" --purge

Destroy clusters

To teardown the KinD clusters, you can issue:

kind delete cluster --name turin
kind delete cluster --name lyon

Global Ingress

In this tutorial, you will learn how to leverage Liqo and K8GB to deploy and expose a multi-cluster application through a global ingress. More in detail, this enables improved load balancing and distribution of the external traffic towards the application replicated across multiple clusters.

The figure below outlines the high-level scenario, with a client consuming an application from either cluster 1 (e.g., located in EU) or cluster 2 (e.g., located in the US), based on the endpoint returned by the DNS server.

Global Ingress Overview

Provision the playground

First, check that you are compliant with the requirements. Additionally, this example requires k3d to be installed in your system. Specifically, this tool is leveraged instead of KinD to match the K8GB Sample Demo.

To provision the playground, clone the Liqo repository and run the setup script:

git clone https://github.com/liqotech/liqo.git
cd liqo
git checkout v0.5.2
cd examples/global-ingress
./setup.sh

The setup script creates three k3s clusters and deploys the appropriate infrastructural application on top of them, as detailed in the following:

edgedns: this cluster will be used to deploy the DNS service. In a production environment, this should be an external DNS service (e.g. AWS Route53). It includes the Bind Server (manifests in manifests/edge folder).
gslb-eu and gslb-us: these clusters will be used to deploy the application. They include:
- ExternalDNS: it is responsible for configuring the DNS entries.
- Ingress Nginx: it is responsible for handling the local ingress traffic.
- K8GB: it configures the multi-cluster ingress.
- Liqo: it enables the application to spread across multiple clusters, and takes care of reflecting the required resources.

Export the kubeconfigs environment variables to use them in the rest of the tutorial:

export KUBECONFIG_DNS=$(k3d kubeconfig write edgedns)
export KUBECONFIG=$(k3d kubeconfig write gslb-eu)
export KUBECONFIG_US=$(k3d kubeconfig write gslb-us)

Note

We suggest exporting the kubeconfig of the gslb-eu as default (i.e., KUBECONFIG), since it will be the entry point of the virtual cluster and you will mainly interact with it.

Peer the clusters

Once Liqo is installed in your clusters, you can establish new peerings. In this example, since the two API Servers are mutually reachable, you will use the out-of-band peering approach.

Specifically, to implement the desired scenario, you should enable a peering from the gslb-eu cluster to the gslb-us cluster. This will allow Liqo to offload workloads and reflect services from the first cluster to the second cluster.

To proceed, first generate a new peer command from the gslb-us cluster:

PEER_US=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_US)

And then, run the generated command from the gslb-eu cluster:

echo "$PEER_US" | bash

When the above command returns successfully, you can check the peering status by running:

kubectl get foreignclusters

The output should look like the following, indicating that an outgoing peering is currently active towards the gslb-us cluster, as well as that the cross-cluster network tunnel has been established:

NAME      OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION   AGE
gslb-us   Established        None               Established   Established      57s

Additionally, you should see a new virtual node (liqo-gslb-us) in the gslb-eu cluster, and representing the whole gslb-us cluster. Every pod scheduled onto this node will be automatically offloaded to the remote cluster by Liqo.

kubectl get node --selector=liqo.io/type=virtual-node

The output should be similar to:

NAME           STATUS   ROLES   AGE   VERSION
liqo-gslb-us   Ready    agent   17s   v1.22.6+k3s1

Deploy an application

Now that the Liqo peering is established, and the virtual node is ready, it is possible to proceed deploying the podinfo demo application. This application serves a web-page showing different information, including the name of the pod; hence, easily identifying which replica is generating the HTTP response.

First, create a hosting namespace in the gslb-eu cluster, and offload it to the remote cluster through Liqo.

kubectl create namespace podinfo
liqoctl offload namespace podinfo --namespace-mapping-strategy EnforceSameName

At this point, it is possible to deploy the podinfo helm chart in the podinfo namespace:

helm upgrade --install podinfo --namespace podinfo \
    podinfo/podinfo -f manifests/values/podinfo.yaml

This chart creates a Deployment with a custom affinity to ensure that the two frontend replicas are scheduled on different nodes and clusters:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/control-plane
          operator: DoesNotExist
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - podinfo
      topologyKey: "kubernetes.io/hostname"

Additionally, it creates an Ingress resource configured with the k8gb.io/strategy: roundRobin annotation. This annotation will instruct the K8GB Global Ingress Controller to distribute the traffic across the different clusters.

Check application spreading

Let’s now check that Liqo replicated the ingress resource in both clusters and that each Nginx Ingress Controller was able to assign them the correct IPs (different for each cluster).

Note

You can see the output for the second cluster appending the --kubeconfig $KUBECONFIG_US flag to each command.

kubectl get ingress -n podinfo

The output in the gslb-eu cluster should be similar to:

NAME      CLASS   HOSTS                    ADDRESS                 PORTS   AGE
podinfo   nginx   liqo.cloud.example.com   172.19.0.3,172.19.0.4   80      6m9s

While the output in the gslb-us cluster should be similar to:

NAME      CLASS   HOSTS                    ADDRESS                 PORTS   AGE
podinfo   nginx   liqo.cloud.example.com   172.19.0.5,172.19.0.6   80      6m16s

With reference to the output above, the liqo.cloud.example.com hostname is served in the demo environment on:

172.19.0.3, 172.19.0.4: addresses exposed by cluster gslb-eu
172.19.0.5, 172.19.0.6: addresses exposed by cluster gslb-us

Each local K8GB installation creates a Gslb resource with the Ingress information and the given strategy (RoundRobin in this case), and ExternalDNS populates the DNS records accordingly.

On the gslb-eu cluster, the command:

kubectl get gslbs.k8gb.absa.oss -n podinfo podinfo -o yaml

should return an output along the lines of:

apiVersion: k8gb.absa.oss/v1beta1
kind: Gslb
metadata:
  annotations:
    k8gb.io/strategy: roundRobin
  name: podinfo
  namespace: podinfo
spec:
  ingress:
    ingressClassName: nginx
    rules:
    - host: liqo.cloud.example.com
      http:
        paths:
        - backend:
            service:
              name: podinfo
              port:
                number: 9898
          path: /
          pathType: ImplementationSpecific
  strategy:
    dnsTtlSeconds: 30
    splitBrainThresholdSeconds: 300
    type: roundRobin
status:
  geoTag: eu
  healthyRecords:
    liqo.cloud.example.com:
    - 172.19.0.3
    - 172.19.0.4
    - 172.19.0.5
    - 172.19.0.6
  serviceHealth:
    liqo.cloud.example.com: Healthy

Similarly, when issuing the command from the gslb-us cluster:

kubectl get gslbs.k8gb.absa.oss -n podinfo podinfo -o yaml --kubeconfig $KUBECONFIG_US

apiVersion: k8gb.absa.oss/v1beta1
kind: Gslb
metadata:
  annotations:
    k8gb.io/strategy: roundRobin
  name: podinfo
  namespace: podinfo
spec:
  ingress:
    ingressClassName: nginx
    rules:
    - host: liqo.cloud.example.com
      http:
        paths:
        - backend:
            service:
              name: podinfo
              port:
                number: 9898
          path: /
          pathType: ImplementationSpecific
  strategy:
    dnsTtlSeconds: 30
    splitBrainThresholdSeconds: 300
    type: roundRobin
status:
  geoTag: us
  healthyRecords:
    liqo.cloud.example.com:
    - 172.19.0.5
    - 172.19.0.6
    - 172.19.0.3
    - 172.19.0.4
  serviceHealth:
    liqo.cloud.example.com: Healthy

In both clusters, the Gslb resources are pretty identical; they only differ for the geoTag field. The resource status also reports:

the serviceHealth status, that should be Healthy for both clusters
the list of IPs exposing the HTTP service: they are the IPs of the nodes of both clusters since the Nginx Ingress Controller is deployed in HostNetwork DaemonSet mode.

Check service reachability

Since podinfo is an HTTP service, you can contact it using the curl command. Use the -v option to understand which of the nodes is being targeted.

You need to use the DNS server in order to resolve the hostname to the IP address of the service. To this end, create a pod in one of the clusters (it does not matter which one) overriding its DNS configuration.

HOSTNAME="liqo.cloud.example.com"
K8GB_COREDNS_IP=$(kubectl get svc k8gb-coredns -n k8gb -o custom-columns='IP:spec.clusterIP' --no-headers)

kubectl run -it --rm curl --restart=Never --image=curlimages/curl:7.82.0 --command \
    --overrides "{\"spec\":{\"dnsConfig\":{\"nameservers\":[\"${K8GB_COREDNS_IP}\"]},\"dnsPolicy\":\"None\"}}" \
    -- curl $HOSTNAME -v

Note

Launching this pod several times, you will see different IPs and different frontend pods answering in a round-robin fashion (as set in the Gslb policy).

*   Trying 172.19.0.3:80...
* Connected to liqo.cloud.example.com (172.19.0.3) port 80 (#0)
...
{
  "hostname": "podinfo-67f46d9b5f-xrbmg",
  "version": "6.1.4",
  "revision": "",
...

*   Trying 172.19.0.6:80...
* Connected to liqo.cloud.example.com (172.19.0.6) port 80 (#0)
...
{
  "hostname": "podinfo-67f46d9b5f-xrbmg",
  "version": "6.1.4",
  "revision": "",
...

*   Trying 172.19.0.3:80...
* Connected to liqo.cloud.example.com (172.19.0.3) port 80 (#0)
...
{
  "hostname": "podinfo-67f46d9b5f-cmnp5",
  "version": "6.1.4",
  "revision": "",
...

This brief tutorial showed how you could leverage Liqo and K8GB to deploy and expose a multi-cluster application. In addition to the RoundRobin policy, which provides load distribution among clusters, K8GB allows favoring closer endpoints (through the GeoIP strategy), or adopt a Failover policy. Additional details are provided in its official documentation.

Tear down the playground

Unoffload namespaces

Before starting the uninstallation process, make sure that all namespaces are unoffloaded:

liqoctl unoffload namespace podinfo

Every pod that was offloaded to a remote cluster is going to be rescheduled onto the local cluster.

Revoke peerings

Similarly, make sure that all the peerings are revoked:

liqoctl unpeer out-of-band gslb-us

At the end of the process, the virtual node is removed from the local cluster.

Uninstall Liqo

Now you can remove Liqo from your clusters with liqoctl:

liqoctl uninstall
liqoctl uninstall --kubeconfig="$KUBECONFIG_US"

Purge

By default the Liqo CRDs will remain in the cluster, but they can be removed with the --purge flag:

liqoctl uninstall --purge
liqoctl uninstall --kubeconfig="$KUBECONFIG_US" --purge

Destroy clusters

To teardown the k3d clusters, you can issue:

k3d cluster delete gslb-eu gslb-us edgedns

Peer two Clusters

This section describes the procedure to establish a peering with a remote cluster, using one of the two alternative approaches featured by Liqo. You can refer to the dedicated features section for a high-level presentation of their characteristics, and the associated trade-offs.

Overview

The peering process leverages liqoctl to interact with the clusters, abstracting the creation and update of the appropriate custom resources. To this end, the most important one is the ForeignCluster resource, which represents a remote cluster, including its identity, the associated authentication endpoint, and the desired peering state (i.e., whether it should be established, and in which directions). Additionally, its status reports a summary of the current peering status, detailing whether the different phases (e.g., authentication, network establishment, resource negotiation, …) correctly succeeded.

The following sections present the respective procedures to peer a local cluster A (i.e., the consumer), with a remote cluster B (i.e., the provider). At the end of the process, a new virtual node is created in the consumer, abstracting the resources shared by the provider, and enabling seamless pod offloading to the remote cluster. Additional details are also provided to enable the reverse peering direction, hence achieving a bidirectional peering, allowing both clusters to offload a part of their workloads to the other.

All examples leverage two different contexts to refer to consumer and provider clusters, respectively named consumer and provider.

Note

liqoctl displays a kubectl compatible behavior concerning Kubernetes API access, hence supporting the KUBECONFIG environment variable, as well as the standard flags, including --kubeconfig and --context. Ensure you selected the correct target cluster before issuing liqoctl commands (as you would do with kubectl).

Out-of-band control plane

Briefly, the procedure to establish an out-of-band control plane peering consists of a first step performed on the provider, to retrieve the set of information required (i.e., authentication endpoint and token, cluster ID, …), followed by the creation of the necessary resources to start the actual peering. The remainder of the process, including identity retrieval, resource negotiation and network tunnel establishment is performed automatically by Liqo, through a mutual exchange of information and negotiation between the two clusters involved.

Information retrieval

To proceed, ensure that you are operating in the provider cluster, and then issue the liqoctl generate peer-command command:

liqoctl --context=provider generate peer-command

This retrieves the information concerning the provider cluster (i.e., authentication endpoint and token, cluster ID, …) and generates a command that can be executed on a different cluster (i.e., the consumer) to establish an out-of-band outgoing peering towards the provider cluster.

An example of the resulting command is the following:

liqoctl peer out-of-band <cluster-name> --auth-url <auth-url> \
    --cluster-id <cluster-id> --auth-token <auth-token>

Peering establishment

Once obtained the peering command, it is possible to execute it in the consumer cluster, to kick off the peering process.

Warning

Pay attention to operate in the correct cluster, possibly adding the appropriate flags to the generated command (e.g., --context=consumer).

liqoctl --context=consumer peer out-of-band <cluster-name> --auth-url <auth-url> \
    --cluster-id <cluster-id> --auth-token <auth-token>

The above command configures the appropriate authentication token, and then creates a new ForeignCluster resource in the consumer cluster. Finally, it waits for the different peering phases to complete (this might require a few seconds, depending on the download time of the Liqo virtual kubelet image).

The ForeignCluster resource can be inspected through kubectl:

kubectl --context=consumer get foreignclusters

If the peering process completed successfully, you should observe an output similar to the following, indicating that the cross-cluster network tunnel has been established, and an outgoing peering is currently active (i.e., the consumer cluster can offload workloads to the provider one, but not vice versa):

NAME       OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION
provider   Established        None               Established   Established

At the same time, a new virtual node should have been created in the consumer cluster. Specifically:

kubectl --context=consumer get nodes -l liqo.io/type=virtual-node

Should return an output similar to the following:

NAME            STATUS   ROLES   AGE    VERSION
liqo-provider   Ready    agent   179m   v1.23.4

Note

The name of the ForeignCluster resource, as well as that of the virtual node, reflects the cluster name specified with the liqoctl peer out-of-band command.

Bidirectional peering

Once the peering from the consumer to the provider has been established, the reverse direction (i.e., leading to a bidirectional peering) can be enabled through a simpler command, since the ForeignCluster resource is already present:

liqoctl --context=provider peer consumer

Tear down

An out-of-band peering can be disabled leveraging the symmetric liqoctl unpeer command, causing the local virtual node (abstracting the remote cluster) to be destroyed, and all offloaded workloads to be rescheduled:

liqoctl --context=consumer unpeer out-of-band

Note

The reverse peering direction, if any, is preserved, and the remote cluster can continue offloading workloads to its virtual node representing the local cluster. Hence, the same command shall be executed on both clusters to completely tear down a bidirectional peering.

In-band control plane

Briefly, the procedure to establish an in-band control plane peering consists of a first step performed by liqoctl, which interacts alternatively with both clusters to establish the cross-cluster VPN tunnel, exchange the authentication tokens and configure the Liqo control plane traffic to flow inside the VPN. The remainder of the process, including identity retrieval and resource negotiation, is performed automatically by Liqo, through a mutual exchange of information and negotiation between the two clusters involved.

Peering establishment

The in-band control plane peering process can be started leveraging a single liqoctl command:

liqoctl peer in-band --context=consumer --remote-context=provider

The above command outputs a set of information concerning the different operations performed on the two clusters. Notably, it exchanges the appropriate authentication tokens, establishes the cross-cluster VPN tunnel, and then creates a new ForeignCluster resource in both clusters. Finally, it waits for the different peering phases to complete (this might require a few seconds, depending on the download time of the Liqo virtual kubelet image).

The ForeignCluster resource can be inspected through kubectl (e.g., on the consumer):

kubectl --context=consumer get foreignclusters

If the peering process completed successfully, you should observe an output similar to the following, indicating that the cross-cluster network tunnel has been established, and an outgoing peering is currently active (i.e., the consumer cluster can offload workloads to the provider one, but not vice versa):

NAME       OUTGOING PEERING   INCOMING PEERING   NETWORKING    AUTHENTICATION
provider   Established        None               Established   Established

At the same time, a new virtual node should have been created in the consumer cluster. Specifically:

kubectl --context=consumer get nodes -l liqo.io/type=virtual-node

Should return an output similar to the following:

NAME             STATUS   ROLES   AGE    VERSION
liqo-provider    Ready    agent   179m   v1.23.4

Note

The name of the ForeignCluster resource, as well as that of the virtual node, reflects the cluster name specified by the remote cluster administrators at install time.

Bidirectional peering

A bidirectional in-band peering can be established adding the --bidirectional flag to the liqoctl peer command invocation:

liqoctl peer in-band --context=consumer --remote-context=provider --bidirectional

Note

The liqoctl peer in-band command is idempotent, and can be re-executed without side effects to enable a bidirectional peering.

Alternatively, the reverse peering can be also activated executing the following on the provider cluster:

liqoctl --context=provider peer consumer

Tear down

An in-band peering can be disabled leveraging the symmetric liqoctl unpeer command, causing both virtual nodes (if present) to be destroyed, all offloaded workloads to be rescheduled, and finally tearing down the cross-cluster VPN tunnel:

liqoctl unpeer in-band --context=consumer --remote-context=provider

In case only one peering direction shall be teared down, while preserving the opposite, it is possible to leverage the appropriate liqoctl unpeer command to disable the outgoing peering (e.g., on the provider cluster):

liqoctl --context=provider unpeer consumer

Namespace Offloading

This section presents the operational procedure to offload a namespace to (possibly a subset of) the remote clusters peered with the local cluster. Hence, enabling pod offloading, as well as triggering the resource reflection process: additional details about namespace extension in Liqo are provided in the dedicated namespace extension features section.

Overview

The offloading of a namespace can be easily controlled through the dedicated liqoctl commands, which abstract the creation and update of the appropriate custom resources. In this context, the most important one is the NamespaceOffloading resource, which enables the offloading of the corresponding namespace, configuring at the same time the subset of target remote clusters, additional constraints concerning pod offloading and the naming strategy. Moreover, different namespaces can be characterized by different configurations, hence achieving a high degree of flexibility. Finally, the NamespaceOffloading status reports for each remote cluster a summary about its status (i.e., whether the remote cluster has been selected for offloading, and the twin namespace has been correctly created).

Offloading a namespace

A given namespace foo can be offloaded, leveraging the default configuration, through:

liqoctl offload namespace foo

Alternatively, the underlying NamespaceOffloading resource can be generated and output (either in yaml or json format) leveraging the dedicated --output flag:

liqoctl offload namespace foo --output yaml

Then, the resulting manifest can be applied with kubectl, or through automation tools (e.g., by means of GitOps approaches).

Note

Possible race conditions might occur in case a NamespaceOffloading resource is created at the same time (e.g., as a batch) as pods (or higher level abstractions such as Deployments), preventing them from being considered for offloading until the NamespaceOffloading resource is not processed.

This situation can be prevented manually labeling in advance the hosting namespace with the liqo.io/scheduling-enabled=true label, hence enabling the Liqo mutating webhook and causing pod creations to be rejected until pod offloading is possible. Still, this causes no problems, as the Kubernetes abstractions (e.g., Deployments) ensure that the desired pods get eventually created correctly.

Regardless of the approach adopted, namespace offloading can be further configured in terms of the three main parameters presented below, each one exposed through a dedicated CLI flag.

Namespace mapping strategy

The namespace mapping strategy defines the naming strategy used to create the remote namespaces, and can be configured through the --namespace-mapping-strategy flag. The accepted values are:

DefaultName (default): to prevent conflicts on the target cluster, remote namespace names are generated as the concatenation of the local namespace name, the cluster name of the local cluster and a unique identifier (e.g., foo could be mapped to foo-lively-voice-dd8531).
EnforceSameName: remote namespaces are named after the local cluster’s namespace. This approach ensures naming transparency, which is required by certain applications, as well as guarantees that cross-namespace DNS queries referring to reflected services work out of the box (i.e., without adapting the target namespace name). Yet, it can lead to conflicts in case a namespace with the same name already exists inside the selected remote clusters, ultimately causing the remote namespace creation request to be rejected.

Note

Once configured for a given namespace, the namespace mapping strategy is immutable, and any modification is prevented by a dedicated Liqo webhook. In case a different strategy is desired, it is necessary to first unoffload the namespace, and then re-offload it with the new parameters.

Pod offloading strategy

The pod offloading strategy defines high-level constraints about pod scheduling, and can be configured through the --pod-offloading-strategy flag. The accepted values are:

LocalAndRemote (default): pods deployed in the local namespace can be scheduled both onto local nodes and onto virtual nodes, hence possibly offloaded to remote clusters.
Local: pods deployed in the local namespace are enforced to be scheduled onto local nodes only, hence never offloaded to remote clusters. The extension of a namespace, forcing at the same time all pods to be scheduled locally, enables the consumption of local services from the remote cluster, as shown in the service offloading example.
Remote: pods deployed in the local namespace are enforced to be scheduled onto remote nodes only, hence always offloaded to remote clusters.

Note

The pod offloading strategy applies to pods only, while the other objects that live in namespaces selected for offloading, and managed by the resource refletion process, are always replicated to (possibly a subset of) the remote clusters, as specified through the cluster selector (more details below).

Warning

Due to current limitations of Liqo, the pods violating the pod offloading strategy are not automatically evicted following an update of this policy to a more restrictive value (e.g., LocalAndRemote to Remote) after the initial creation.

Cluster selector

The cluster selector provides the possibility to restrict the set of remote clusters (in case more than one peering is active) selected as targets for offloading the given namespace. The twin namespace is not created in clusters that do not match the cluster selector, as well as the resource reflection mechanism is not activated for those namespaces. Yet, different cluster selectors can be specified for different namespaces, depending on the desired configuration.

The cluster selector follows the standard label selector syntax, and refers to the Kubernetes labels characterizing the virtual nodes. Specifically, these include both the set of labels suggested by the remote cluster during the peering process and automatically propagated by Liqo, as well as possible additional ones added by the local cluster administrators.

The cluster selector can be expressed through the --selector flag, which can be optionally repeated multiple times to specify alternative requirements (i.e., in logical OR). For instance:

--selector 'region in (europe,us-west), !staging' would match all clusters located in the europe or us-west region, AND not including the staging label.
--selector 'region in (europe,us-west)' --selector '!staging' would match all clusters located in the europe or us-west region, OR not including the staging label.

In case no cluster selector is specified, all remote clusters are selected as targets for namespace offloading. In other words, an empty cluster selector matches all virtual clusters.

Unoffloading a namespace

The offloading of a namespace can be disabled through the dedicated liqoctl command, causing in turn the deletion of all resources reflected to remote clusters (including the namespaces themselves), and triggering the rescheduling of all offloaded pods locally:

liqoctl unoffload namespace foo

Warning

Disabling the offloading of a namespace is a destructive operation, since all resources created in remote namespaces (either automatically or manually) get removed, including possible persistent storage volumes. Before proceeding, double-check that the correct namespace has been selected, and ensure no important data is still present.

Resource Reflection

This section characterizes the resource reflection process (including also pod offloading), detailing how the different resources are propagated to remote clusters and which fields are mutated.

Briefly, the set of supported resources includes (by category):

Workload: Pods
Exposition: Services, EndpointSlices, Ingresses
Storage: PersistentVolumeClaims, PresistentVolumes
Configuration: ConfigMaps, Secrets

Pods offloading

Liqo leverages a custom resource, named ShadowPod, combined with an appropriate enforcement logic to ensure remote pod resiliency even in case of temporary connectivity loss between the local and remote clusters.

Pod specifications are propagated to the remote cluster verbatim, except for the following fields that are mutated:

Removal of scheduling constraints (e.g., Affinity, NodeSelector, SchedulerName, Preemption, …), as referring to the local cluster.
Mutation of service account related information, to allow offloaded pods to transparently interact with the local (i.e., origin) API server, instead of the remote one.
Enforcement of the properties concerning the usage of host namespaces (e.g., network, IPC, PID) to false (i.e., disabled), as potentially invasive and troublesome.

Differently, pod status is propagated from the remote cluster to the local one, performing the following modifications:

The PodIP is remapped according to the network fabric configuration, such as to be reachable from the other pods running in the same cluster.
The NodeIP is replaced with the one of the corresponding virtual kubelet pod.
The number of container restarts is augmented to account for the possible deletions of the remote pod (whose presence is enforced by the controlling ShadowPod resource).

Note

A pod living in a namespace not enabled for offloading, but manually forced to be scheduled in a virtual node, remains in Pending status, and it is signaled with the OffloadingBackOff reason. For instance, this can happen for system DaemonSets (e.g., CNI plugins), which tolerate all taints (hence, including the one associated with virtual nodes) and thus get scheduled on all nodes.

To prevent this behavior, it is necessary to explicitly modify the involved DaemonSets, adding a suitable affinity constraint excluding virtual nodes:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
          - key: liqo.io/type
            operator: NotIn
            values:
            - virtual-node

Service exposition

The reflection of Service and EndpointSlice resources is a key element to allow the seamless intercommunication between microservices spread across multiple clusters, enabling the usage of standard DNS discovery mechanisms. In addition, the propagation of Ingresses enables the definition of multiple points of entrance for the external traffic, especially when combined with additional tools such as K8GB (see the global ingress example for additional details).

Services

Services are reflected verbatim into remote clusters, except for what concerns the ClusterIP, LoadBalancerIP and NodePort fields (when applicable), which are left empty (hence defaulted by the remote cluster), as likely conflicting. Still, the usage of standard DNS discovery mechanisms (i.e., based on service name/namespace) abstracts away the ClusterIP differences, with each pod retrieving the correct IP address.

Note

In case node port correspondence across clusters is required, its propagation can be enforced adding the liqo.io/force-remote-node-port=true annotation to the involved service.

EndpointSlices

In the local cluster, Services are transparently handled by the vanilla Kubernetes control plane, since it has full visibility of all pods (even those offloaded), hence leading to the creation of the corresponding EndpointSlice entries. Differently, the control plane of each remote cluster perceives only the pods running in that cluster, and the standard EndpointSlice creation logic alone is not sufficient (as it would not include the pods hosted by other clusters).

This gap is filled by the Liqo EndpointSlice reflection logic, which takes care of propagating all EndpointSlice entries (i.e. endpoints) not already present in the destination cluster. During the propagation process, endpoint addresses are appropriately remapped according to the network fabric configuration, ensuring that the resulting IPs are reachable from the destination cluster.

Thanks to this approach, multiple replicas of the same microservice spread across different clusters, and backed by the same service, are handled transparently. Each pod, no matter where it is located, contributes with a distinct EndpointSlice entry, either by the standard control plane or through resource reflection, hence becoming eligible during the Service load-balancing process.

Ingresses

The propagation of Ingress resources enables the configuration of multiple points of entrance for external traffic. Ingress resources are propagated verbatim into remote clusters, except for the IngressClassName field, which is left empty. Hence, selecting the default ingress class in the remote cluster, as the local one (i.e., the one in the origin cluster) might not be present.

Persistent storage

The reflection of PersistentVolumeClaims (PVCs) and PersistentVolumes (PVs) is a key to enable the cross-cluster Liqo storage fabric. Specifically, the process is triggered when a PVC requiring the Liqo storage class is bound for the first time, and the requesting pod is scheduled in a virtual node (i.e., remote cluster). Upon this event, the PVC is propagated verbatim to the remote cluster, replacing the requested StorageClass with the one negotiated during the peering process.

Once created, the resulting PV is reflected backwards (i.e., from the remote to the local cluster), and the proper affinity selectors are added to bind it to the virtual node. Hence, subsequent pods mounting that PV will be scheduled on that virtual node, and eventually offloaded to the same remote cluster.

Configuration data

ConfigMaps and Secrets typically hold configuration data consumed by pods, and both types of resources are propagated by Liqo verbatim into remote clusters. In this respect, Liqo features also the propagation of Secrets holding ServiceAccount tokens, to enable offloaded pods to contact the Kubernetes API server of the origin cluster, as well as to support those applications leveraging ServiceAccounts for internal authentication purposes.

Warning

Currently, Liqo supports only the propagation of ServiceAccount tokens contained in the respective Secret object (i.e., first party tokens), and not of those to be retrieved from the TokenRequest API (i.e., third party tokens). Due to this limitation, service account reflection is currently disabled by default in Kubernetes v1.24+, as ServiceAccounts do not longer automatically generate the corresponding Secret.

Stateful Applications

As introduced in the storage fabric features section, Liqo supports multi-cluster stateful applications by extending the classical approaches adopted in standard Kubernetes clusters.

Liqo virtual storage class

The Liqo virtual storage class is a Storage Class that embeds the logic to create the appropriate PersistentVolumes, depending on the target cluster the mounting pod is scheduled onto. All operations performed on virtual objects (i.e., PersistentVolumeClaims (PVCs) and PersistentVolumes (PVs) associated with the liqo storage class) are then automatically propagated by Liqo to the corresponding real ones (i.e., associated with the storage class available in the target cluster).

Additionally, once a real PV gets created, the corresponding virtual one is enriched with a set of policies to attract mounting pods in the appropriate cluster, following the data gravity approach.

The figure below shows an application pod consuming a virtual PVC, which in turn led to the creation of the associated virtual PV. This process is completely transparent from the management point of view, with the only difference being the name of the storage class.

Virtual Storage Class

Warning

The deletion of the virtual PVC will cause the deletion of the real PVC/PV, and the stored data will be permanently lost.

The Liqo control plane handles the binding of virtual PVC resources (i.e., associated with the liqo storage class) differently depending on the cluster where the mounting pod gets eventually scheduled onto, as detailed in the following.

Local cluster binding

In case a virtual PVC is bound to a pod initially scheduled onto the local cluster (i.e., a physical node), the Liqo control plane takes care of creating a twin PVC (in turn originating the corresponding twin PV) in the liqo-storage namespace, while mutating the storage class to that configured at Liqo installation time (with a fallback to the default one). A virtual PV is eventually created by Liqo to mirror the real one, effectively allowing pods to mount it and enforcing the data gravity constraints.

The resulting configuration is depicted in the figure below.

Virtual Storage Class Local

Current Limitations

Currently, the virtual storage class does not support the configuration of Kubernetes mount options and parameters.

Remote cluster binding

In case a virtual PVC is bound to a pod initially scheduled onto a remote cluster (i.e., a virtual node), the Liqo control plane takes care of creating a twin PVC (in turn originating the corresponding twin PV) in the offloaded namespace, while mutating the storage class to that negotiated at peering time (i.e., configured at Liqo installation time in the remote cluster, with a fallback to the default one). A virtual PV is eventually created by Liqo to mirror the real one, effectively allowing pods to mount it and enforcing the data gravity constraints.

The resulting configuration is depicted in the figure below.

Virtual Storage Class Remote

Warning

The tearing down of the peering and/or the deletion of the offloaded namespace will cause the deletion of the real PVC, and the stored data will be permanently lost.

Move PVCs across clusters

Once a PVC is created in a given cluster, subsequent pods mounting that volume will be forced to be scheduled onto the same cluster to achieve storage locality, following the data gravity approach.

Still, if necessary, you can manually move the storage backing a virtual PVC (i.e., associated with the liqo storage class) from a cluster to another, leveraging the appropriate liqoctl command. Then, subsequent pods will get scheduled in the cluster the storage has been moved to.

Warning

This procedure requires the PVC/PV not to be bound to any pods during the entire process. In other words, live migration is currently not supported.

A given PVC can be moved to a target node (either physical, i.e., local, or virtual, i.e., remote) through the following command:

liqoctl move volume $PVC_NAME --namespace $NAMESPACE_NAME --node $TARGET_NODE_NAME

Where:

$PVC_NAME is the name of the PVC to be moved.
$NAMESPACE_NAME is the name of the namespace where the PVC lives in.
$TARGET_NODE_NAME is the name of the node where the PVC will be moved to.

Under the hood, the migration process leverages the Liqo cross-cluster network fabric and the Restic project to back up the original data in a temporary repository, and then restore it in a brand-new PVC forced to be created in the target cluster.

Warning

Liqo and liqoctl are not backup tools. Make sure to properly back up important data before starting the migration process.

Externally managed storage

In addition to the virtual storage class, Liqo supports the offloading of pods that bind to cross-cluster storage managed by external solutions (e.g., managed by the cloud provider, or manually provisioned). Specifically, the volumes stanza of the pod specification is propagated verbatim to the offloaded pods, hence allowing to specify volumes available only remotely.

Note

In case a piece of externally managed storage is available only in one remote cluster, it is likely necessary to manually force pods to get scheduled exactly in that cluster. To prevent scheduling issues (e.g., the pod is marked as Pending since the local cluster has no visibility on the remote PVC), it is suggested to configure the target NodeName in the pod specifications to match that of the corresponding virtual nodes, hence bypassing the standard Kubernetes scheduling logic.

Warning

Due to current Liqo limitations, the remote namespace, including any PVC therein contained, will be deleted in case the local namespace is unoffloaded/deleted, or the peering is torn down.

Contributing to Liqo

First off, thank you for taking the time to contribute to Liqo!

This page lists a set of contributing guidelines, including suggestions about the local development of Liqo components and the execution of the automatic tests.

Repository structure

The Liqo repository structure follows the Standard Go Project Layout.

Release notes generation

Liqo leverages the automatic release notes generation capabilities featured by GitHub. Specifically, PRs characterized by the following labels get included in the respective category:

kind/breaking: 💥 Breaking Change
kind/feature: 🚀 New Features
kind/bug: 🐛 Bug Fixes
kind/cleanup: 🧹 Code Refactoring
kind/docs: 📝 Documentation

Local development

While developing a new feature, it is typically useful to test the changes in a local environment, as well as debug the code to identify possible problems. To this end, you can leverage the setup.sh script provided for the quick start example to spawn two development clusters using KinD, and then install Liqo on both of them (you can refer to the dedicated section for additional information concerning the installation of development versions through liqoctl):

./examples/quick-start/setup.sh
liqoctl install kind --kubeconfig=./liqo_kubeconf_rome --version ...
liqoctl install kind --kubeconfig=./liqo_kubeconf_milan --version ...

Once the environment is properly setup, it is possible to proceed according to one of the following approaches:

Building and pushing the Docker image of the component under development to a registry, and appropriately editing the corresponding Deployment/DaemonSet to use the custom version. This allows to observe the modified component in realistic conditions, and it is mandatory for the networking substratum, since it needs to interact with the underlying host configuration.
Scaling to 0 the number of replicas of the component under development, copying its current configuration (i.e., command-line flags), and executing it locally (while targeting the appropriate cluster). This allows for faster development cycles, as well as for the usage of standard debugging techniques to troubleshoot possible issues.

Automatic tests

Liqo features two major test suites:

End-to-end (E2E) tests, which assess the correct functioning of the main Liqo features.
Unit Tests, which focus on each specific component, in multiple operating conditions.

Both test suites are automatically executed through the GitHub Actions pipelines, following the corresponding slash command trigger. A successful outcome is required to make PRs eligible for being considered for review and merged.

The following sections provide additional details concerning how to run the above tests in a local environment, for troubleshooting.

End-to-end tests

We suggest executing the E2E tests on a system with at least 8 GB of free RAM. Additionally, please review the requirements presented in the Liqo examples section, which also apply in this case (including the suggestions concerning increasing the maximum number of inotify watches).

Once all requirements are met, it is necessary to export the set of environment variables shown below, to configure the tests. In most scenarios, the only variable that needs to be modified is LIQO_VERSION, which should point to the SHA of the commit referring to the Liqo development version to be tested (the appropriate Docker images shall have been built in advance through the appropriate GitHub Actions pipeline).

export CLUSTER_NUMBER=4
export K8S_VERSION=v1.21.1
export CNI=kindnet
export TMPDIR=$(mktemp -d)
export BINDIR=${TMPDIR}/bin
export TEMPLATE_DIR=${PWD}/test/e2e/pipeline/infra/kind
export NAMESPACE=liqo
export KUBECONFIGDIR=${TMPDIR}/kubeconfigs
export LIQO_VERSION=<YOUR_COMMIT_ID>
export INFRA=kind
export LIQOCTL=${BINDIR}/liqoctl
export POD_CIDR_OVERLAPPING=false
export TEMPLATE_FILE=cluster-templates.yaml.tmpl

Finally, it is possible to launch the tests:

make e2e

Unit tests

Most unit tests can be run directly using the ginkgo CLI, which in turn supports the standard testing API (go test, IDE features, …). The only requirement is the controller-runtime envtest environment, which can be installed through:

export K8S_VERSION=1.19.2
curl --fail -sSLo envtest-bins.tar.gz "https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-${K8S_VERSION}-$(go env GOOS)-$(go env GOARCH).tar.gz"
mkdir /usr/local/kubebuilder/
tar -C /usr/local/kubebuilder/ --strip-components=1 -zvxf envtest-bins.tar.gz

Some networking tests, however, require an isolated environment. To this end, you can leverage the dedicated liqo-test Docker image (the Dockerfile is available in build/liqo-test):

# Build the liqo-test Docker image
make test-container

# Run all unit tests, and retrieve coverage
make unit

# Run the tests for a specific package.
# Note, the package path must start with ./ to avoid the "package ... is not in GOROOT error".
docker run --rm --entrypoint="" --mount type=bind,src=$(pwd),dst=/go/src/liqo \
   --privileged=true --workdir /go/src/liqo liqo-test go test <package>

Debugging unit tests

When executing the unit tests from the liqo-test container, it is possible to use Delve to perform remote debugging:

Start the liqo-test container with an idle entry point, exposing a port of choice (e.g. 2345):

docker run --name=liqo-test -d -p 2345:2345 --mount type=bind,src=$(pwd),dst=/go/src/liqo \
   --privileged=true --workdir /go/src/liqo --entrypoint="" liqo-test tail -f /dev/null

Open a shell inside the liqo-test container, and install Delve:

docker exec -it liqo-test bash
go install github.com/go-delve/delve/cmd/dlv@latest

Run a specific test inside the container:

dlv test --headless --listen=:2345 --api-version=2  \
   --accept-multiclient ./path/to/test/directory

From the host, connect to localhost:2345 with your remote debugging client of choice (e.g. GoLand), and enjoy!

What is Liqo?

What does it provide?

What to explore next?

Peering

Overview

Approaches

Out-of-band control plane

In-band control plane

Offloading

Virtual kubelet

Virtual node

Namespace extension

Pod offloading

Resource reflection

Network Fabric

Network manager

Cross-cluster VPN tunnels

In-cluster overlay network

Storage Fabric

Requirements

Overview

Connectivity

Out-of-band control plane peering

Additional considerations

In-band control plane peering

Network firewalls

Liqo CLI tool

Introduction

Install liqoctl with Homebrew

Install liqoctl manually

Install Kubectl plugin with Krew

Install liqoctl from source

Enable shell autocompletion

Install

Install with liqoctl

Customization options

Global

Control plane

Networking

Install with Helm

Install development versions

Liqo and Calico

Uninstall

Purge CRDs

Requirements

Quick Start

Provision the playground

Explore the playground

Install Liqo

Peer two clusters

Leverage remote resources

Start a hello world application

Check the pod connectivity

Expose the pods through a Service

Check the Service connectivity

Play with a microservice application

Observe the application deployment

Access the demo application

Tear down the playground

Unoffload namespaces

Revoke peerings

Uninstall Liqo

Destroy clusters

Offloading with Policies

Provision the playground

Peer the clusters

Tune namespace offloading

Deploy applications

Tear down the playground

Unoffload namespaces

Revoke peerings

Uninstall Liqo

Destroy clusters

Offloading a Service

Provision the playground

Peer the clusters

Offload a service

Consume the service

Tear down the playground

Unoffload namespaces