Multi-tenancy Apache Kafka clusters in Amazon MSK with IAM entry management and Kafka Quotas – Half 1

With Amazon Managed Streaming for Apache Kafka (Amazon MSK), you’ll be able to construct and run purposes that use Apache Kafka to course of streaming knowledge. To course of streaming knowledge, organizations both use a number of Kafka clusters primarily based on their utility groupings, utilization situations, compliance necessities, and different components, or a devoted Kafka cluster for your entire group. It doesn’t matter what sample is used, Kafka clusters are usually multi-tenant, permitting a number of producer and shopper purposes to eat and produce streaming knowledge concurrently.

With multi-tenant Kafka clusters, nonetheless, one of many challenges is to be sure that knowledge shopper and producer purposes don’t overuse cluster sources. There’s a risk that a number of poorly behaved purposes might overuse cluster sources, affecting the well-behaved purposes consequently. Subsequently, groups who handle multi-tenant Kafka clusters want a mechanism to stop purposes from overconsuming cluster sources with the intention to keep away from points. That is the place Kafka quotas come into play. Kafka quotas management the quantity of sources consumer purposes can use inside a Kafka cluster.

In Half 1 of this two-part collection, we clarify the ideas of the best way to implement Kafka quotas in MSK multi-tenant Kafka clusters whereas utilizing AWS Id and Entry Administration (IAM) entry management for authentication and authorization. In Half 2, we cowl detailed implementation steps together with pattern Kafka consumer purposes.

Temporary introduction to Kafka quotas

Kafka quotas management the quantity of sources consumer purposes can use inside a Kafka cluster. It’s potential for the multi-tenant Kafka cluster to expertise efficiency degradation or an entire outage as a consequence of useful resource constraints if a number of consumer purposes produce or eat giant volumes of knowledge or generate requests at a really excessive fee for a steady time frame, monopolizing Kafka cluster’s sources.

To stop purposes from overwhelming the cluster, Apache Kafka permits configuring quotas that decide how a lot site visitors every consumer utility produces and consumes per Kafka dealer in a cluster. Kafka brokers throttle the consumer purposes’ requests in accordance with their allotted quotas. Kafka quotas may be configured for particular customers, or particular consumer IDs, or each. The consumer ID is a logical identify outlined within the utility code that Kafka brokers use to establish which utility despatched messages. The consumer represents the authenticated consumer principal of a consumer utility in a safe Kafka cluster with authentication enabled.

There are two forms of quotas supported in Kafka:

• Community bandwidth quotas – The byte-rate thresholds outline how a lot knowledge consumer purposes can produce to and eat from every particular person dealer in a Kafka cluster measured in bytes per second.
• Request fee quotas – This limits the share of time every particular person dealer spends processing consumer purposes requests.

Relying on the enterprise necessities, you should utilize both of those quota configurations. Nonetheless, using community bandwidth quotas is widespread as a result of it permits organizations to cap platform sources consumption in response to the quantity of knowledge produced and consumed by purposes per second.

As a result of this put up makes use of an MSK cluster with IAM entry management, we particularly talk about configuring community bandwidth quotas primarily based on the purposes’ consumer IDs and authenticated consumer principals.

Issues for Kafka quotas

Maintain the next in thoughts when working with Kafka quotas:

• Enforcement stage – Quotas are enforced on the dealer stage reasonably than on the cluster stage. Suppose there are six brokers in a Kafka cluster and also you specify a 12 MB/sec produce quota for a consumer ID and consumer. The producer utility utilizing the consumer ID and consumer can produce a max of 12MB/sec on every dealer on the identical time, for a complete of max 72 MB/sec throughout all six brokers. Nonetheless, if management for each partition of a subject resides on one dealer, the identical producer utility can solely produce a max of 12 MB/sec. As a result of the truth that throttling happens per dealer, it’s important to take care of an excellent steadiness of matters’ partitions management throughout all of the brokers.
• Throttling – When an utility reaches its quota, it’s throttled, not failed, which means the dealer doesn’t throw an exception. Purchasers who attain their quota on a dealer will start to have their requests throttled by the dealer to stop exceeding the quota. As an alternative of sending an error when a consumer exceeds a quota, the dealer makes an attempt to sluggish it down. Brokers calculate the quantity of delay essential to deliver shoppers below quotas and delay responses accordingly. On account of this method, quota violations are clear to shoppers, and shoppers don’t must implement any particular backoff or retry insurance policies. Nonetheless, when utilizing an asynchronous producer and sending messages at a fee better than the dealer can settle for as a consequence of quota, the messages will likely be queued within the consumer utility reminiscence first. The consumer will finally run out of buffer house if the speed of sending messages continues to exceed the speed of accepting messages, inflicting the subsequent Producer.ship() name to be blocked. Producer.ship() will finally throw a TimeoutException if the timeout delay isn’t ample to permit the dealer to catch as much as the producer utility.
• Shared quotas – If a couple of consumer utility has the identical consumer ID and consumer, the quota configured for the consumer ID and consumer will likely be shared amongst all these purposes. Suppose you configure a produce quota of 5 MB/sec for the mixture of client-id="marketing-producer-client" and consumer="marketing-app-user". On this case, all producer purposes which have marketing-producer-client as a consumer ID and marketing-app-user as an authenticated consumer principal will share the 5 MB/sec produce quota, impacting one another’s throughput.
• Produce throttling – The produce throttling conduct is uncovered to producer shoppers by way of consumer metrics equivalent to produce-throttle-time-avg and produce-throttle-time-max. If these are non-zero, it signifies that the vacation spot brokers are slowing the producer down and the quotas configuration needs to be reviewed.
• Devour throttling – The eat throttling conduct is uncovered to shopper shoppers by way of consumer metrics equivalent to fetch-throttle-time-avg and fetch-throttle-time-max. If these are non-zero, it signifies that the origin brokers are slowing the patron down and the quotas configuration needs to be reviewed.

Observe that consumer metrics are metrics uncovered by shoppers connecting to Kafka clusters.

• Quota configuration – It’s potential to configure Kafka quotas both statically by means of the Kafka configuration file or dynamically by means of kafka-config.sh or the Kafka Admin API. The dynamic configuration mechanism is way more handy and manageable as a result of it permits quotas for the brand new producer and shopper purposes to be configured at any time with out having to restart brokers. Even whereas utility shoppers are producing or consuming knowledge, dynamic configuration adjustments take impact in actual time.
• Configuration keys – With the kafka-config.sh command-line instrument, you’ll be able to set dynamic eat, produce, and request quotas utilizing the next three configuration keys, respectively: consumer_byte_rate, producer_byte_rate, and request_percentage.

For extra details about Kafka quotas, consult with Kafka documentation.

Implement community bandwidth quotas with IAM entry management

Following our understanding of Kafka quotas, let’s take a look at the best way to implement them in an MSK cluster whereas utilizing IAM entry management for authentication and authorization. IAM entry management in Amazon MSK eliminates the necessity for 2 separate mechanisms for authentication and authorization.

The next determine reveals an MSK cluster that’s configured to make use of IAM entry management within the demo account. Every producer and shopper utility has a quota that determines how a lot knowledge they’ll produce or eat in bytes per second. For instance, ProducerApp-1 has a produce quota of 1024 bytes/sec, and ConsumerApp-1 and ConsumerApp-2 every have a eat quota of 5120 and 1024 bytes/sec, respectively. It’s vital to notice that Kafka quotas are set on the Kafka cluster reasonably than within the consumer purposes.

The previous determine illustrates how Kafka consumer purposes (ProducerApp-1, ConsumerApp-1, and ConsumerApp-2) entry Subject-B within the MSK cluster by assuming write and skim IAM roles. The workflow is as follows:

• P1 – ProducerApp-1 (by way of its ProducerApp-1-Function IAM function) assumes the Subject-B-Write-Function IAM function to ship messages to Subject-B within the MSK cluster.
• P2 – With the Subject-B-Write-Function IAM function assumed, ProducerApp-1 begins sending messages to Subject-B.
• C1 – ConsumerApp-1 (by way of its ConsumerApp-1-Function IAM function) and ConsumerApp-2 (by way of its ConsumerApp-2-Function IAM function) assume the Subject-B-Learn-Function IAM function to learn messages from Subject-B within the MSK cluster.
• C2 – With the Subject-B-Learn-Function IAM function assumed, ConsumerApp-1 and ConsumerApp-2 begin consuming messages from Subject-B.

ConsumerApp-1 and ConsumerApp-2 are two separate shopper purposes. They don’t belong to the identical shopper group.

Configuring consumer IDs and understanding authenticated consumer principal

As defined earlier, Kafka quotas may be configured for particular customers, particular consumer IDs, or each. Let’s discover consumer ID and consumer ideas and configurations required for Kafka quota allocation.

Consumer ID

A consumer ID representing an utility’s logical identify may be configured inside an utility’s code. In Java purposes, for instance, you’ll be able to set the producer’s and shopper’s consumer IDs utilizing ProducerConfig.CLIENT_ID_CONFIG and ConsumerConfig.CLIENT_ID_CONFIG configurations, respectively. The next code snippet illustrates how ProducerApp-1 units the consumer ID to this-is-me-producerapp-1 utilizing ProducerConfig.CLIENT_ID_CONFIG:

Properties props = new Properties();
props.put(ProducerConfig.CLIENT_ID_CONFIG,"this-is-me-producerapp-1");

Person

The consumer refers to an authenticated consumer principal of the consumer utility within the Kafka cluster with authentication enabled. As proven within the answer structure, producer and shopper purposes assume the Subject-B-Write-Function and Subject-B-Learn-Function IAM roles, respectively, to carry out write and skim operations on Subject-B. Subsequently, their authenticated consumer principal will appear to be the next IAM identifier:

arn:aws:sts::<AWS Account Id>:assumed-role/<assumed Function Identify>/<function session identify>

For extra data, consult with IAM identifiers.

The function session identify is a string identifier that uniquely identifies a session when IAM principals, federated identities, or purposes assume an IAM function. In our case, ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 purposes assume an IAM function utilizing the AWS Safety Token Service (AWS STS) SDK, and supply a job session identify within the AWS STS SDK name. For instance, if ProducerApp-1 assumes the Subject-B-Write-Function IAM function and makes use of this-is-producerapp-1-role-session as its function session identify, its authenticated consumer principal will likely be as follows:

arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Write-Function/this-is-producerapp-1-role-session

The next is an instance code snippet from the ProducerApp-1 utility utilizing this-is-producerapp-1-role-session because the function session identify whereas assuming the Subject-B-Write-Function IAM function utilizing the AWS STS SDK:

StsClient stsClient = StsClient.builder().area(area).construct();
AssumeRoleRequest roleRequest = AssumeRoleRequest.builder()
          .roleArn("<Subject-B-Write-Function ARN>")
          .roleSessionName("this-is-producerapp-1-role-session") //role-session-name string literal
          .construct();

Configure community bandwidth (produce and eat) quotas

The next instructions configure the produce and eat quotas dynamically for consumer purposes primarily based on their consumer ID and authenticated consumer principal within the MSK cluster configured with IAM entry management.

The next code configures the produce quota:

kafka-configs.sh --bootstrap-server <MSK cluster bootstrap servers IAM endpoint> 
--command-config config_iam.properties 
--alter --add-config "producer_byte_rate=<variety of bytes per second>" 
--entity-type shoppers --entity-name <ProducerApp consumer Id> 
--entity-type customers --entity-name <ProducerApp consumer principal>

The producer_byes_rate refers back to the variety of messages, in bytes, {that a} producer consumer recognized by consumer ID and consumer is allowed to supply to a single dealer per second. The choice --command-config factors to config_iam.properties, which accommodates the properties required for IAM entry management.

The next code configures the eat quota:

kafka-configs.sh --bootstrap-server <MSK cluster bootstrap servers IAM endpoint> 
--command-config config_iam.properties 
--alter --add-config "consumer_byte_rate=<variety of bytes per second>" 
--entity-type shoppers --entity-name <ConsumerApp consumer Id> 
--entity-type customers --entity-name <ConsumerApp consumer principal>

The consumer_bytes_rate refers back to the variety of messages, in bytes, {that a} shopper consumer recognized by consumer ID and consumer allowed to eat from a single dealer per second.

Let’s take a look at some instance quota configuration instructions for ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 consumer purposes:

• ProducerApp-1 produce quota configuration – Let’s assume ProducerApp-1 has this-is-me-producerapp-1 configured because the consumer ID within the utility code and makes use of this-is-producerapp-1-role-session because the function session identify when assuming the Subject-B-Write-Function IAM function. The next command units the produce quota for ProducerApp-1 to 1024 bytes per second:

kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint> 
--command-config config_iam.properties 
--alter --add-config "producer_byte_rate=1024" 
--entity-type shoppers --entity-name this-is-me-producerapp-1 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Write-Function/this-is-producerapp-1-role-session

• ConsumerApp-1 eat quota configuration – Let’s assume ConsumerApp-1 has this-is-me-consumerapp-1 configured because the consumer ID within the utility code and makes use of this-is-consumerapp-1-role-session because the function session identify when assuming the Subject-B-Learn-Function IAM function. The next command units the eat quota for ConsumerApp-1 to 5120 bytes per second:

kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint> 
--command-config config_iam.properties 
--alter --add-config "consumer_byte_rate=5120" 
--entity-type shoppers --entity-name this-is-me-consumerapp-1 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Learn-Function/this-is-consumerapp-1-role-session

ConsumerApp-2 eat quota configuration – Let’s assume ConsumerApp-2 has this-is-me-consumerapp-2 configured because the consumer ID within the utility code and makes use of this-is-consumerapp-2-role-session because the function session identify when assuming the Subject-B-Learn-Role IAM function. The next command units the eat quota for ConsumerApp-2 to 1024 bytes per second per dealer:

kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint> 
--command-config config_iam.properties 
--alter --add-config "consumer_byte_rate=1024" 
--entity-type shoppers --entity-name this-is-me-consumerapp-2 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Learn-Function/this-is-consumerapp-2-role-session

On account of the previous instructions, the ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 consumer purposes will likely be throttled by the MSK cluster utilizing IAM entry management in the event that they exceed their assigned produce and eat quotas, respectively.

Implement the answer

Half 2 of this collection showcases the step-by-step detailed implementation of Kafka quotas configuration with IAM entry management together with the pattern producer and shopper consumer purposes.

Conclusion

Kafka quotas provide groups the power to set limits for producer and shopper purposes. With Amazon MSK, Kafka quotas serve two vital functions: eliminating guesswork and stopping points brought on by poorly designed producer or shopper purposes by limiting their quota, and allocating operational prices of a central streaming knowledge platform throughout totally different price facilities and tenants (utility and product groups).

On this put up, we discovered the best way to configure community bandwidth quotas inside Amazon MSK whereas utilizing IAM entry management. We additionally coated some pattern instructions and code snippets to make clear how the consumer ID and authenticated principal are used when configuring quotas. Though we solely demonstrated Kafka quotas utilizing IAM entry management, you can too configure them utilizing different Amazon MSK-supported authentication mechanisms.

In Half 2 of this collection, we display the best way to configure community bandwidth quotas with IAM entry management in Amazon MSK and offer you instance producer and shopper purposes so to see them in motion.

Take a look at the next sources to be taught extra:

In regards to the Writer

Vikas Bajaj is a Senior Supervisor, Options Architects, Monetary Providers at Amazon Net Providers. Having labored with monetary companies organizations and digital native prospects, he advises monetary companies prospects in Australia on expertise selections, architectures, and product roadmaps.