This submit was co-authored by Qi Ke, Company Vice President, Azure Kubernetes Service.
In the present day, we’re thrilled to announce the final availability of Azure CNI Overlay. This can be a large step ahead in addressing networking efficiency and the scaling wants of our clients.
As cloud-native workloads proceed to develop, clients are always pushing the size and efficiency boundaries of our current networking options in Azure Kubernetes Service (AKS). For Occasion, the normal Azure Container Networking Interface (CNI) approaches require planning IP addresses prematurely, which might result in IP handle exhaustion as demand grows. In response to this demand, now we have developed a brand new networking answer referred to as “Azure CNI Overlay”.
On this weblog submit, we are going to focus on why we would have liked to create a brand new answer, the size it achieves, and the way its efficiency compares to the present options in AKS.
Fixing for efficiency and scale
In AKS, clients have a number of community plugin choices to select from when making a cluster. Nonetheless, every of those choices have their very own challenges in relation to large-scale clusters.
The “kubenet” plugin, an current overlay community answer, is constructed on Azure route tables and the bridge plugin. Since kubenet (or host IPAM) leverages route tables for cross node communication it was designed for, not more than 400 nodes or 200 nodes in twin stack clusters.
The Azure CNI VNET gives IPs from the digital community (VNET) handle house. This can be tough to implement because it requires a big, distinctive, and consecutive Classless Inter-Area Routing (CIDR) house and clients could not have the accessible IPs to assign to a cluster.
Deliver your Personal Container Community Interface (BYOCNI) brings a whole lot of flexibility to AKS. Prospects can use encapsulation—like Digital Extensible Native Space Community (VXLAN)—to create an overlay community as properly. Nonetheless, the extra encapsulation will increase latency and instability because the cluster measurement will increase.
To handle these challenges, and to assist clients who wish to run giant clusters with many nodes and pods with no limitations on efficiency, scale, and IP exhaustion, now we have launched a brand new answer: Azure CNI Overlay.
Azure CNI Overlay
Azure CNI Overlay assigns IP addresses from the user-defined overlay handle house as a substitute of utilizing IP addresses from the VNET. It makes use of the routing of those personal handle areas as a local digital community function. Which means cluster nodes don’t have to carry out any further encapsulation to make the overlay container community work. This additionally permits this overlay addressing house to be reused for various AKS clusters even when linked through the identical VNET.
When a node joins the AKS cluster, we assign a /24 IP handle block (256 IPs) from the Pod CIDR to it. Azure CNI assigns IPs to Pods on that node from the block, and beneath the hood, VNET maintains a mapping of the Pod CIDR block to the node. This fashion, when Pod visitors leaves the node, VNET platform is aware of the place to ship the visitors. This enables the Pod overlay community to realize the identical efficiency as native VNET visitors and paves the way in which to assist hundreds of thousands of pods and throughout hundreds of nodes.
Datapath efficiency comparability
This part sneaks into among the datapath efficiency comparisons now we have been working towards Azure CNI Overlay.
Word: We used the Kubernetes benchmarking instruments accessible at kubernetes/perf-tests for this train. Comparability can differ based mostly on a number of components resembling underlining {hardware} and Node proximity inside a datacenter amongst others. Precise outcomes would possibly differ.
Azure CNI Overlay vs. VXLAN-based Overlay
As talked about earlier than, the one choices for giant clusters with many Nodes and lots of Pods are Azure CNI Overlay and BYO CNI. Right here we examine Azure CNI Overlay with VXLAN-based overlay implementation utilizing BYO CNI.
TCP Throughput – Greater is Higher (19% achieve in TCP Throughput)
Azure CNI Overlay confirmed a major efficiency enchancment over VXLAN-based overlay implementation. We discovered that the overhead of encapsulating CNIs was a major think about efficiency degradation, particularly because the cluster grows. In distinction, Azure CNI Overlay’s native Layer 3 implementation of overlay routing eradicated the double-encapsulation useful resource utilization and confirmed constant efficiency throughout varied cluster sizes. In abstract, Azure CNI Overlay is a most viable answer for working manufacturing grade workloads in Kubernetes.
Azure CNI Overlay vs. Host Community
This part will cowl how pod networking performs towards node networking and see how native L3 routing of pod networking helps Azure CNI Overlay implementation.
Azure CNI Overlay and Host Community have comparable throughput and CPU utilization outcomes, and this reinforces that the Azure CNI Overlay implementation for Pod routing throughout nodes utilizing the native VNET function is as environment friendly as native VNET visitors.
TCP Throughput – Greater is Higher (Much like HostNetwork)
Azure CNI Overlay powered by Cilium: eBPF dataplane
Up up to now, we’ve solely taken a have a look at Azure CNI Overlay advantages alone. Nonetheless, via a partnership with Isovalent, the subsequent era of Azure CNI is powered by Cilium. A number of the advantages of this strategy embrace higher useful resource utilization by Cilium’s prolonged Berkeley Packet Filter (eBPF) dataplane, extra environment friendly intra cluster load balancing, Community Coverage enforcement by leveraging eBPF over iptables, and extra. To learn extra about Cilium’s efficiency beneficial properties via eBPF, see Isovalent’s weblog submit on the topic.
In Azure CNI Overlay Powered by Cilium, Azure CNI Overlay units up the IP-address administration (IPAM) and Pod routing, and Cilium provisions the Service routing and Community Coverage programming. In different phrases, Azure CNI Overlay Powered by Cilium permits us to have the identical overlay networking efficiency beneficial properties that we’ve seen to this point on this weblog submit plus extra environment friendly Service routing and Community Coverage implementation.
It is nice to see that Azure CNI Overlay powered by Cilium is ready to present even higher efficiency than Azure CNI Overlay with out Cilium. The upper pod to service throughput achieved with the Cilium eBPF dataplane is a promising enchancment. The added advantages of elevated observability and extra environment friendly community coverage implementation are additionally vital for these seeking to optimize their AKS clusters.
TCP Throughput – Greater is best
To wrap up, Azure CNI Overlay is now usually accessible in Azure Kubernetes Service (AKS) and gives important enhancements over different networking choices in AKS, with efficiency akin to Host Community configurations and assist for linearly scaling the cluster. And pairing Azure CNI Overlay with Cilium brings much more efficiency advantages to your clusters. We’re excited to ask you to attempt Azure CNI Overlay and expertise the advantages in your AKS surroundings.
To get began right now, go to the documentation accessible.
