NOTE: This article is part of a serie dedicated on Scaleway Kosmos. If you haven’t read from the beggining you can should start here:
- Part 1: Discovering Kosmos
- Part 2: Playing with stateless apps
- Part 3: Playing with stateful apps and volumes
- Part 4: Let’s talk about networking and CNI
- Part 5: Conclusions (this article)
NB: Sources can be grabbed from https://github.com/webofmars/labs-kosmos/
📚 Contexte
When i started this lab i wanted to deep dive on the product Scaleway Kosmos that allow to create kubernetes clusters on a multi-cloud fashion, subject very famous at the moment.
For testing it i tested 9 nodes on 3 different cloud-providers:
- 3 x CX31 in Hetzner (helsinki DC): 2 vCPUs / 8Gb Ram
- 3 x D2–8 in OVH (GRA5 DC): 4 vCPUs / 8Gb Ram
- 3 x 2gb in DigitalOcean (AMS3 DC): 2vCPUs / 2Gb Ram
🤔 Choices …
The choices i made for the lab revealed to be crucial and i made 2 errors:
- The cheap nodes especially on OVH where too limited on internet bandwidth. Next time i won’t be affraid to spend more for better benchmarks. The same goes for the 2 CPUs node on DO.
- Testing network performances on cloud is hard. There too much parameters that can interfer in the results. But here i guess this was a no choice because it’s what is Kosmos for.
🔧 Setup of the cluster
Setup of the control plane is very easy and can be done in a few minutes. Setup of the nodes is less easy until you find the correct parameters to pass region labels.
👨🎤 Deploying stateless apps
Deploying stateless is easy like a breeze and quite satisfactory if you don’t have a specific requirements for network latency or quantity of network traffic. Thsi can be an issue both in terms of performance and billing since most CP bills you on traffic out only.
But this put aside, that is really doing a great job !
💂♀️Deploying stateful apps
Deploying stateful apps that work with volumes is a bit more complex. You must either find a universal storage class (Rancher Longhorn / Portworx / NFS …) or deploy each of the cloud-provider CSI with node selection and tie the workload to a given cloud-provider. I used Local-Path-Provisionner as an exemple but for a realistic application my choice would go to Longhorn or Rook.
It’s a bit more setup to do but once ready that’s look quite promising.
🥷 Going crazy about CNI
When testing Stateless and Staeful apps it became obvious that the CNI used (kilo, a wireguard enabled CNI) was central in the performance point of view of this multi-cloud cluster. I decide to bench it using KnB tool by Alexis Ducastel. As mentioned above this showed us that this is very difficult to conduct a bench campain in cloud context because cloud is not a standard, it’s a kind of implementation and a lot of parameters can interfer in the bench.
That being said the performance seems to have reasonable performances but showed a massive drop when going above 1200 Mbps, being unable to determine if this is related to bench protocol or to the CNI itself. Would love to bench it on premises too to understand.
💰Finances
The cost of Kosmos Control Plane, now that it is officialy GA, is 99€/month not including the worker nodes you will deploy in differents cloud-providers. This is actualy quite cheap for an HA k8s control-plane with auto-scaling and a CNI with encryption.
So this is clearly a positive point that can make you consider “why do we need a Kosmos Control Plane ?”
⚡ To infinity, and beyond!
If you are going to be serious about multi-cloud kubernetes clusters you might lack a few things more:
- A way to deploy synchronized versions of your apps on all the regions
- A monitoring tool taking in account the locality
- Locality aware storage solution
- Locality routing solution with possible failover
🌀 Conclusions
Kosmos is really an appealing solution if you have a not complex app stack that do need multi-cloud presence. Easy to setup, cheap and easy to manage. So don’t hesitate to go for it or to give it a try.
If you need more control or fine grained setups i would go for a setup of independant cluster and use any kind of federation process to join them. But it’s by far a step higher in complexity comparaide to Kosmos.
➕ More on this @ DevOps D-day
If you are interested in such problematics i will give a review of a concrete implementation of a native multi-cloud solution designed for one customer with specific needs at the devops D Day conference in Marseille the 18th Novemeber 2021. See you there !
