Join Kafka consumer purposes securely to your Amazon MSK cluster from completely different VPCs and AWS accounts

Now you can use Amazon Managed Streaming for Apache Kafka (Amazon MSK) multi-VPC non-public connectivity (powered by AWS PrivateLink) and cluster coverage assist for MSK clusters to simplify connectivity of your Kafka shoppers to your brokers. Amazon MSK is a totally managed service that makes it simple so that you can construct and run purposes that use Kafka to course of streaming information. Once you create an MSK cluster, the cluster sources can be found to shoppers inside the similar Amazon VPC. This lets you launch the cluster inside particular subnets of the VPC, affiliate it with safety teams, and connect IP addresses out of your VPC’s handle area by means of elastic community interfaces (ENIs). Community site visitors between shoppers and the cluster stays inside the AWS community, with web entry to the cluster not attainable by default.

In case you have workloads segmented throughout a number of VPCs and AWS accounts, there could also be situations during which you’ll want to make your MSK brokers accessible to Kafka shoppers throughout VPCs. With the launch of Amazon MSK multi-VPC non-public connectivity, now you can privately entry your MSK brokers out of your consumer purposes in one other VPC inside the similar AWS account or one other AWS account with out enabling public entry or creating and managing your personal networking infrastructure for personal connectivity. A cluster coverage is an AWS Id and Entry Administration (IAM) resource-based coverage, which is outlined to your MSK cluster to supply cross-account IAM principals permissions to arrange non-public connectivity to the cluster.

This publish introduces Amazon MSK multi-VPC connectivity and how one can privately entry your MSK clusters out of your shoppers in different VPCs. It additionally reveals how one can outline a cluster coverage to your MSK clusters. These new two capabilities simplify configuring cross-VPC community entry and organising permissions wanted for Kafka shoppers to privately connect with MSK brokers in a distinct account.

Earlier than Amazon MSK multi-VPC connectivity

Earlier than Amazon MSK multi-VPC connectivity, the community admin wanted to decide on one of many following safe connectivity patterns. Admins needed to repeat sure steps for every dealer within the cluster.

• Amazon VPC peering is the only networking assemble that permits bidirectional connectivity between two VPCs. On this method, the community admin needed to replace every VPC with the IP addresses of every dealer within the routing tables of all subnets. You possibly can’t use this connectivity sample when there are overlapping IPv4 or IPv6 CIDR blocks within the VPCs.
• AWS Transit Gateway gives a extremely accessible and scalable design for connecting VPCs. On this method, the community admin consistently needed to replace the routing tables hooked up to every transit gateway. Not like VPC peering that may go cross-Area, AWS Transit Gateway is a regional service, however you need to use inter-Area peering between transit gateways to route site visitors throughout areas. AWS Transit Gateway has the utmost bandwidth (burst) per Availability Zone per VPC connection (50 Gbps). This might change into a problem for some workloads.
• AWS PrivateLink is an AWS networking service that gives non-public entry to a selected service as a substitute of all sources inside a VPC and with out traversing the general public web. It additionally eliminates the necessity to expose your entire VPC or subnet, and prevents points like having to cope with overlapping CIDR blocks between the VPC that hosts the MSK cluster ENIs and the Kafka consumer VPC. AWS PrivateLink can scale to a vast variety of VPCs and in contrast to the opposite choices, site visitors right here is unidirectional. Due to these advantages, AWS PrivateLink is a well-liked option to handle non-public connectivity. Nevertheless, this connectivity sample comes with further complexity. It requires creating a number of Community Load Balancers (NLBs) per cluster and creating non-public service endpoints per NLB within the service account. Moreover, admins needed to create non-public endpoints per non-public service endpoint, and an Amazon Route 53 alias document per non-public endpoint in each consumer account.

The next diagram illustrates the structure of customer-managed VPC endpoints between completely different VPCs in numerous AWS accounts with IAM authentication.

After Amazon MSK multi-VPC connectivity and cluster coverage

Now you can allow multi-VPC and cross-account connectivity to your MSK clusters in a number of easy steps and pay for what you utilize. This eliminates the overhead of making and managing AWS PrivateLink infrastructure. When new brokers are added to a cluster, non-public connectivity is maintained with out the necessity to make configuration adjustments, saving you from the overhead and complexity of managing the underlying community infrastructure.

The next diagram illustrates this up to date structure of utilizing Amazon MSK multi-VPC connectivity to attach a consumer from a distinct AWS account.

Resolution overview

Establishing multi-VPC non-public connectivity entails turning on this characteristic for the cluster and configuring the Kafka shoppers to attach privately to the cluster.

The next are the high-level steps to configure the cluster:

1. Allow the multi-VPC non-public connectivity characteristic for a subset of authentication schemes which might be enabled to your MSK cluster.
2. If a Kafka consumer is in an AWS account that’s completely different than the cluster, connect a resource-based coverage to the MSK cluster to authorize IAM principals for creating cross-account connectivity.
3. Share the cluster ARN with the IAM principal related to the Kafka consumer that should create the cross-account entry to MSK cluster.

The next are the high-level steps to configure the shoppers:

1. Create a managed VPC endpoint for the consumer VPC that should join privately to the MSK cluster.
2. Replace the VPC endpoint’s safety group settings to allow outbound connectivity to the MSK cluster.
3. Arrange the consumer to make use of the cluster’s connection string to attach privately to the cluster.

Cluster setup

On this publish, we solely present the steps for enabling Amazon MSK multi-VPC connectivity for a provisioned cluster.

1. To allow Amazon MSK multi-VPC connectivity in your present cluster, select Activate multi-VPC connectivity on the Amazon MSK console.
   
   Notice that multi-VPC connectivity can’t be turned on with a cluster that permits unauthenticated entry. That is to stop unauthenticated entry from completely different VPCs.
2. Choose the authentication strategies that you just permit shoppers in different VPCs to make use of.
   The record of authentication strategies is populated based mostly in your cluster’s safety configuration.
3. Overview the settings and select Activate choice. After the multi-VPC connectivity is enabled in your cluster, Amazon MSK will create the NLB and VPC endpoint service infrastructure required for personal connectivity. Amazon MSK will vend a brand new set of bootstrap dealer strings that can be utilized for personal connectivity. These may be accessed utilizing the View consumer data possibility on the Amazon MSK console. The following step is to supply the IAM principals related together with your shoppers the permissions to attach privately to your cluster. To do that, you’ll want to connect a cluster coverage to the cluster.
4. Select Edit cluster coverage within the Safety part of the cluster particulars web page on the Amazon MSK console.
   The brand new cluster coverage permits for outlining a Primary or Superior cluster coverage. With the Primary possibility, you’ll be able to merely enter AWS account IDs of your consumer’s VPCs. This coverage permits all allowed principals in these AWS accounts to carry out CreateVPCConnection, GetBootstrapBrokers, DescribeCluster, and DescribeClusterV2 actions which might be required for creating the cross-VPC connectivity to your cluster. Nevertheless, in different instances, you might want a extra advanced coverage that permits for extra actions, or principals aside from AWS accounts, reminiscent of IAM roles, position classes, IAM customers, and extra. You possibly can creator a cluster coverage in accordance with IAM JSON coverage steerage and supply that to the cluster in Superior mode.
5. Outline your cluster coverage and select Save adjustments.

Shopper setup

On the consumer aspect, first you’ll want to connect an identification coverage to the IAM principal who desires to create a managed VPC connection. The identification coverage should present permission for making a managed VPC connection. The required permissions are a part of the AWS managed coverage AmazonMSKFullAccess.

1. Within the different AWS account with the IAM principal you configured, use the brand new Managed VPC connection web page on the Amazon MSK console to create Amazon MSK managed VPC connections.
   A managed VPC connection maps to an AWS PrivateLink endpoint beneath the hood, and Amazon MSK makes use of the managed VPC connection to orchestrate non-public connectivity to the cluster. You merely must create the managed VPC connection and pay commonplace AWS PrivateLink costs for the underlying endpoint.
2. Enter the AWS Useful resource Title (ARN) of the cluster that you just need to connect with.
3. Select Confirm to confirm the cluster data and its minimal necessities for cross-connectivity.
4. Choose an authentication technique from the offered values.
5. Select the VPC ID the place your Kafka shoppers are positioned, and select their subnet IDs. You possibly can add extra subnets utilizing the Add subnet possibility.
   The desired consumer subnet will need to have Availability Zone IDs that match the cluster’s Availability Zone IDs. This makes certain the shoppers are positioned in a similar bodily Availability Zone because the cluster brokers. Amazon MSK makes use of the port vary 14001:14100 for all authentication strategies. You have to choose a safety group that permits outbound site visitors to this port. The next screenshot reveals an instance.
6. Overview the settings and select Create connection.
   The method will take a couple of minutes.
7. When it’s full, you’ll be able to acquire the shoppers’ connection string from the small print web page of your connection.
8. The following step is to replace the outbound guidelines for the VPC endpoint safety group to permit communication to the port vary 14001:14100.

Use the Amazon MSK-managed VPC connection

After you create the managed VPC connection, connecting privately to the cluster is straightforward. Merely use the brand new connection string to connect with the cluster. For instance, you might join from an Amazon Elastic Compute Cloud (Amazon EC2) occasion in your consumer VPC. Then run the next command to confirm should you can join and carry out actions towards the matters within the MSK cluster:

export MSK_VPC=<YOUR CLIENT CONNECTION STRING GOES HERE>
bin/kafka-topics.sh --bootstrap-server $MSK_VPC -command-config /residence/ec2-user/kafka/config/client-config.properties –record

IAM authentication

Earlier than the launch of Amazon MSK multi-VPC connectivity, Kafka shoppers in different AWS accounts who opted in IAM authentication, wanted to imagine one other IAM position within the cluster’s account. To facilitate this, admins needed to create a number of IAM roles and write a belief coverage that permits authenticated principals from the consumer’s accounts to imagine corresponding roles by means of the sts:AssumeRole API name. This method was difficult to scale when the variety of VPCs or AWS accounts grew. With the launch of this cluster coverage, cross-account entry management is now simplified as a result of you’ll be able to connect a cluster coverage to your clusters to specify which cross-account shoppers have what permissions on sources inside the cluster.

This functionality permits you to handle all entry to the cluster and matters in a single place. For instance, you’ll be able to management which IAM principals have write entry to sure matters, and which principals can solely learn from them. Customers who’re utilizing IAM consumer authentication may add permissions for required kafka-cluster actions within the cluster useful resource coverage.

Availability and pricing

Now you can use Amazon MSK multi-VPC connectivity in all business Areas the place Amazon MSK is obtainable, together with China and GovCloud (US) Areas.

You pay $0.006 per GB information processed for personal connectivity and $0.0225 per non-public connectivity hour per authentication scheme in US East (Ohio). Consult with our Pricing web page for extra particulars.

Conclusion

With Amazon MSK multi-VPC non-public connectivity, now you can privately entry your MSK brokers out of your consumer purposes in one other VPC inside the similar AWS account or one other AWS account, with minimal configuration. You not should create, handle, and replace a number of networking sources in a number of VPCs, or make Amazon MSK configuration adjustments to attach your Kafka shoppers throughout VPCs and accounts. Amazon MSK creates and manages the sources for you. With Cluster coverage assist, you’ll be able to simply present your cross-account consumer principals permissions to attach privately to your MSK cluster. Additional, if you’re utilizing IAM consumer authentication, it’s also possible to leverage the cluster coverage to centrally management shoppers’ permissions to carry out operations on the cluster. Use the Amazon MSK multi-VPC connectivity and the cluster coverage characteristic immediately to simplify your safe connectivity infrastructure.

For additional studying on Amazon MSK, go to the official product web page and our AWS Documentation.

Concerning the authors

Ali Alemi is a Streaming Specialist Options Architect at AWS. Ali advises AWS clients with architectural greatest practices and helps them design real-time analytics information techniques which might be dependable, safe, environment friendly, and cost-effective. He works backward from clients’ use instances and designs information options to unravel their enterprise issues. Previous to becoming a member of AWS, Ali supported a number of public sector clients and AWS consulting companions of their software modernization journey and migration to the cloud.