Feature: Multi-Cluster Controller Support

[maximiliantech]

I would like to run the azure-serivce-operator on a Kubernetes cluster A and let it reconcile custom resources (e.g. StorageAccount, MySQLDatabase, AzureLoadBalancer etc.) from other Kubernetes clusters B, C, D, ... . This would enable us (openmcp-project) to realize shared workload use cases to separate and isolate APIs from workload.

The topic of multi-cluster controllers has gained significant traction lately, with frequent discussions in community calls, conference CFPs, and SIG meetings. A use case where this has an impact is https://www.kcp.io/ - a framework for building multi-tenant platforms.

While ASO already excels at managing Azure resources within a single cluster, it might be worth discussing whether emerging multi-cluster controller concepts could enhance its roadmap.
A way to achieve this would be kubernetes-sigs/multicluster-runtime (extends controller-runtime for dynamic cluster fleets).

With this issue, I don't want to propose any specific implementation. I simply want to trigger a discussion here if/when multi-cluster support aligns with ASO's future direction. What are your thoughts on feasibility, user demand, or alternative approaches?

References

• https://www.kcp.io/
• https://github.com/kubernetes-sigs/multicluster-runtime
• https://www.youtube.com/watch?v=7op_r9R0fCo

[matthchr]

This is something I also learned about relatively recently. Can you talk more about resiliency and security w.r.t. managing other clusters from a "hub" cluster?

This is obviously something that's come up before in the Kubernetes community and hub clusters aren't uncommon, but I know I've spoke with users with concerns that:

1. A controller in the hub cluster reconciling resources in all the child clusters is a single point of failure from a fault-tolerance perspective in a way that running that controller once on each cluster would alleviate.
2. From a security perspective, the hub cluster becomes very enticing as a target, because it has high-level permissions to all of the other clusters.

That's not to say that this hub-spoke model doesn't have advantages as well, but it would be good to understand a bit more about the use-case and why running 1x controller on each cluster wouldn't work? This is especially useful for things like rolling out CRD updates, as even if the hub cluster can manage the resources in the spoke clusters, the spoke clusters still need the CRDs - what install those CRDs (in ASO right now usually it's the controller)?

[maximiliantech]

Hi @matthchr, I am sorry that I get back to you so late. KubeCon Amsterdam was keeping me busy the last weeks. Let me introduce you to @ntnn who is contributing to kcp and multicluster-runtime.
@ntnn could you please support here with the questions and concerns raised above ⬆️

@matthchr If you like we can also get the three of us in a short meeting to discuss this further ✌️ Happy to hear from you!

[ntnn]

Hi!

multicluster-runtime is a drop-in replacement for controller-runtime for the most part. There's almost no change in the actual code between some types switching to the multicluster variant and to get the cluster out of the multicluster request context to rerieve the client to interact with the cluster the request originated from.

The default modus operandi of an operator using multicluster-runtime still is to just target the local cluster - if the multicluster manager gets no provider passed it acts exactly like a controller-runtime manager.

Both point 1 and 2 are not wrong, but for both redundancy and security I'd argue that this is the duty of the user and not of the developer. If the user wants the operator to only work locally they can still deploy one operator per cluster.
But supporting multicluster-runtime would allow downstream to deploy in multicluster-native topologies if they wish.

We already have hub-spoke models in the wild where "hub"/"orchestration" clusters are running one operator per "spoke" cluster - utilizing multicluster-runtime just makes these operations easier to implement and scale because the operators can handle more than one cluster at a time and multicluster-runtime allows operators to automatically shard the clusters they service between themselves.

I'd love to jump in a meeting to discuss whatever comes up :)

[matthchr]

What do users do for CRD versioning among hub/spoke clusters? Are CRDs deployed to each spoke cluster individually (if so... how do operators deal with drift between the spoke and the hub w.r.t. CRD versions?) Or are CRDs managed on the hub cluster and installed across all spoke clusters?

What about conversion webhooks? Not having a pod to reconcile things on the spokes sorta makes some sense (modulo the SPOF and security questions raised above), but not having a pod for webhooks on each spoke means that you won't be able to actually use kubectl on the spokes in the same way as you would normally expect, since at least for ASO we make heavy use of CRD conversion webhooks and without them you're going to have a pretty degraded experience if you're just purely on the spoke.