Multi-tenancy Apache Kafka clusters in Amazon MSK with IAM entry management and Kafka quotas – Half 2

Kafka quotas are integral to multi-tenant Kafka clusters. They stop Kafka cluster efficiency from being negatively affected by poorly behaved functions overconsuming cluster assets. Moreover, they permit the central streaming knowledge platform to be operated as a multi-tenant platform and utilized by downstream and upstream functions throughout a number of enterprise traces. Kafka helps two kinds of quotas: community bandwidth quotas and request price quotas. Community bandwidth quotas outline byte-rate thresholds similar to how a lot knowledge shopper functions can produce to and eat from every particular person dealer in a Kafka cluster measured in bytes per second. Request price quotas restrict the share of time every particular person dealer spends processing shopper functions requests. Relying in your configuration, Kafka quotas might be set for particular customers, particular shopper IDs, or each.

In Half 1 of this two-part sequence, we mentioned the ideas of the way to implement Kafka quotas in Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters whereas utilizing AWS Id and Entry Administration (IAM) entry management.

On this put up, we stroll you thru the step-by-step implementation of organising Kafka quotas in an MSK cluster whereas utilizing IAM entry management and testing them by way of pattern shopper functions.

Answer overview

The next determine, which we first launched in Half 1, illustrates how Kafka shopper functions (ProducerApp-1, ConsumerApp-1, and ConsumerApp-2) entry Subject-B within the MSK cluster by assuming write and browse IAM roles. Every producer and client shopper software has a quota that determines how a lot knowledge they’ll produce or eat in bytes/second. The ProducerApp-1 quota permits it to supply as much as 1024 bytes/second per dealer. Equally, the ConsumerApp-1 and ConsumerApp-2 quotas enable them to eat 5120 and 1024 bytes/second per dealer, respectively. The next is a short rationalization of the move proven within the structure diagram:

• P1 – ProducerApp-1 (by way of its ProducerApp-1-Function IAM position) assumes the Subject-B-Write-Function IAM position to ship messages to Subject-B
• P2 – With the Subject-B-Write-Function IAM position assumed, ProducerApp-1 begins sending messages to Subject-B
• C1 – ConsumerApp-1 (by way of its ConsumerApp-1-Function IAM position) and ConsumerApp-2 (by way of its ConsumerApp-2-Function IAM position) assume the Subject-B-Learn-Function IAM position to learn messages from Subject-B
• C2 – With the Subject-B-Learn-Function IAM position assumed, ConsumerApp-1 and ConsumerApp-2 begin consuming messages from Subject-B

Observe that this put up makes use of the AWS Command Line Interface (AWS CLI), AWS CloudFormation templates, and the AWS Administration Console for provisioning and modifying AWS assets, and assets provisioned shall be billed to your AWS account.

The high-level steps are as follows:

1. Provision an MSK cluster with IAM entry management and Amazon Elastic Compute Cloud (Amazon EC2) situations for shopper functions.
2. Create Subject-B on the MSK cluster.
3. Create IAM roles for the shopper functions to entry Subject-B.
4. Run the producer and client functions with out setting quotas.
5. Configure the produce and eat quotas for the shopper functions.
6. Rerun the functions after setting the quotas.

Stipulations

It is suggested that you just learn Half 1 of this sequence earlier than persevering with. With a purpose to get began, you want the next:

• An AWS account that shall be known as the demo account on this put up, assuming that its account ID is 1111 1111 1111
• Permissions to create, delete, and modify AWS assets within the demo account

Provision an MSK cluster with IAM entry management and EC2 situations

This step entails provisioning an MSK cluster with IAM entry management in a VPC within the demo account. Moreover, we create 4 EC2 situations to make configuration adjustments to the MSK cluster and host producer and client shopper functions.

Deploy CloudFormation stack

1. Clone the GitHub repository to obtain the CloudFormation template information and pattern shopper functions:

git clone https://github.com/aws-samples/amazon-msk-kafka-quotas.git

2. On the AWS CloudFormation console, select Stacks within the navigation pane.
3. Select Create stack.
4. For Put together template, choose Template is prepared.
5. For Template supply, choose Add a template file.
6. Add the cfn-msk-stack-1.yaml file from amazon-msk-kafka-quotas/cfn-templates listing, then select Subsequent.
7. For Stack title, enter MSKStack.
8. Depart the parameters as default and select Subsequent.
9. Scroll to the underside of the Configure stack choices web page and select Subsequent to proceed.
10. Scroll to the underside of the Evaluation web page, choose the examine field I acknowledge that CloudFormation could create IAM assets, and select Submit.

It would take roughly half-hour for the stack to finish. After the stack has been efficiently created, the next assets shall be created:

• A VPC with three personal subnets and one public subnet
• An MSK cluster with three brokers with IAM entry management enabled
• An EC2 occasion referred to as MSKAdminInstance for modifying MSK cluster settings in addition to creating and modifying AWS assets
• EC2 situations for ProducerApp-1, ConsumerApp-1, and ConsumerApp-2, one for every shopper software
• A separate IAM position for every EC2 occasion that hosts the shopper software, as proven within the structure diagram

11. From the stack’s Outputs tab, be aware the MSKClusterArn worth.

Create a subject on the MSK cluster

To create Subject-B on the MSK cluster, full the next steps:

1. On the Amazon EC2 console, navigate to your record of working EC2 situations.
2. Choose the MSKAdminInstance EC2 occasion and select Join.
3. On the Session Supervisor tab, select Join.
4. Run the next instructions on the brand new tab that opens in your browser:

sudo su - ec2-user

# Add Kafka binaries to the trail
sed -i 's|HOME/bin|HOME/bin:~/kafka/bin|' .bash_profile

# Set your AWS area
aws configure set area <AWS Area>

5. Set the surroundings variable to level to the MSK Cluster brokers IAM endpoint:

MSK_CLUSTER_ARN=<Use the worth of MSKClusterArn that you just famous earlier>
echo "export BOOTSTRAP_BROKERS_IAM=$(aws kafka get-bootstrap-brokers --cluster-arn $MSK_CLUSTER_ARN | jq -r .BootstrapBrokerStringSaslIam)" >> .bash_profile
supply .bash_profile
echo $BOOTSTRAP_BROKERS_IAM

6. Pay attention to the worth of BOOTSTRAP_BROKERS_IAM.
7. Run the next Kafka CLI command to create Subject-B on the MSK cluster:

kafka-topics.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--create --topic Subject-B 
--partitions 3 --replication-factor 3 
--command-config config_iam.properties

As a result of the MSK cluster is provisioned with IAM entry management, the choice --command-config factors to config_iam.properties, which incorporates the properties required for IAM entry management, created by the MSKStack CloudFormation stack.

The next warnings could seem while you run the Kafka CLI instructions, however it’s possible you’ll ignore them:

The configuration 'sasl.jaas.config' was provided however is not a identified config. 
The configuration 'sasl.shopper.callback.handler.class' was provided however is not a identified config.

8. To confirm that Subject-B has been created, record all of the subjects:

kafka-topics.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties --list

Create IAM roles for shopper functions to entry Subject-B

This step entails creating Subject-B-Write-Function and Subject-B-Learn-Function as proven within the structure diagram. Subject-B-Write-Function allows write operations on Subject-B, and might be assumed by the ProducerApp-1 . In the same means, the ConsumerApp-1 and ConsumerApp-2 can assume Subject-B-Learn-Function to carry out learn operations on Subject-B. To carry out learn operations on Subject-B, ConsumerApp-1 and ConsumerApp-2 should additionally belong to the patron teams specified through the MSKStack stack replace within the subsequent step.

Create the roles with the next steps:

1. On the AWS CloudFormation console, select Stacks within the navigation pane.
2. Choose MSKStack and select Replace.
3. For Put together template, choose Change present template.
4. For Template supply, choose Add a template file.
5. Add the cfn-msk-stack-2.yaml file from amazon-msk-kafka-quotas/cfn-templates listing, then select Subsequent.
6. Present the next extra stack parameters:

• • For Subject B ARN, enter the Subject-B ARN.

The ARN should be formatted as arn:aws:kafka:area:account-id:matter/msk-cluster-name/msk-cluster-uuid/Subject-B. Use the cluster title and cluster UUID from the MSK cluster ARN you famous earlier and supply your AWS Area. For extra data, seek advice from the IAM entry management for Amazon MSK.

• • For ConsumerApp-1 Client Group title, enter ConsumerApp-1 client group ARN.

It should be formatted as arn:aws:kafka:area:account-id:group/msk-cluster-name/msk-cluster-uuid/consumer-group-name

• • For ConsumerApp-2 Client Group title, enter ConsumerApp-2 client group ARN.

Use the same format because the earlier ARN.

7. Select Subsequent to proceed.
8. Scroll to the underside of the Configure stack choices web page and select Subsequent to proceed.
9. Scroll to the underside of the Evaluation web page, choose the examine field I acknowledge that CloudFormation could create IAM assets, and select Replace stack.

It would take roughly 3 minutes for the stack to replace. After the stack has been efficiently up to date, the next assets shall be created:

• Subject-B-Write-Function – An IAM position with permission to carry out write operations on Subject-B. Its belief coverage permits the ProducerApp-1-Function IAM position to imagine it.
• Subject-B-Learn-Function – An IAM position with permission to carry out learn operations on Subject-B. Its belief coverage permits the ConsumerApp-1-Function and ConsumerApp-2-Function IAM roles to imagine it. Moreover, ConsumerApp-1 and ConsumerApp-2 should additionally belong to the patron teams you specified when updating the stack to carry out learn operations on Subject-B.

10. From the stack’s Outputs tab, be aware the TopicBReadRoleARN and TopicBWriteRoleARN values.

Run the producer and client functions with out setting quotas

Right here, we run ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 with out setting their quotas. From the earlier steps, you’ll need BOOTSTRAP_BROKERS_IAM worth, Subject-B-Write-Function ARN, and Subject-B-Learn-Function ARN. The supply code of shopper functions and their packaged variations can be found within the GitHub repository.

Run the ConsumerApp-1 software

To run the ConsumerApp-1 software, full the next steps:

1. On the Amazon EC2 console, choose the ConsumerApp-1 EC2 occasion and select Join.
2. On the Session Supervisor tab, select Join.
3. Run the next instructions on the brand new tab that opens in your browser:

sudo su - ec2-user

# Set your AWS area
aws configure set area <aws area>

# Set BOOTSTRAP_BROKERS_IAM variable to MSK cluster's IAM endpoint
BOOTSTRAP_BROKERS_IAM=<Use the worth of BOOTSTRAP_BROKERS_IAM that you just famous earlier> 

echo "export BOOTSTRAP_BROKERS_IAM=$(echo $BOOTSTRAP_BROKERS_IAM)" >> .bash_profile

# Clone GitHub repository containing supply code for shopper functions
git clone https://github.com/aws-samples/amazon-msk-kafka-quotas.git

cd amazon-msk-kafka-quotas/uber-jars/

4. Run the ConsumerApp-1 software to begin consuming messages from Subject-B:

java -jar kafka-consumer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn <Subject-B-Learn-Function-ARN> 
--topic-name <Subject-Title> 
--region <AWS Area> 
--consumer-group <ConsumerApp-1 client group title> 
--role-session-name <position session title for ConsumerApp-1 to make use of throughout STS assume position name> 
--client-id <ConsumerApp-1 shopper.id> 
--print-consumer-quota-metrics Y 
--cw-dimension-name <CloudWatch Metrics Dimension Title> 
--cw-dimension-value <CloudWatch Metrics Dimension Worth> 
--cw-namespace <CloudWatch Metrics Namespace>

You could find the supply code on GitHub to your reference. The command line parameter particulars are as follows:

• –bootstrap-servers – MSK cluster bootstrap brokers IAM endpoint.
• –assume-role-arn – Subject-B-Learn-Function IAM position ARN. Assuming this position, ConsumerApp-1 will learn messages from the subject.
• –area – Area you’re utilizing.
• –topic-name – Subject title from which ConsumerApp-1 will learn messages. The default is Subject-B.
• –consumer-group – Client group title for ConsumerApp-1, as specified through the stack replace.
• –role-session-name – ConsumerApp-1 assumes the Subject-B-Learn-Function utilizing the AWS Safety Token Service (AWS STS) SDK. ConsumerApp-1 will use this position session title when calling the assumeRole operate.
• –client-id – Shopper ID for ConsumerApp-1 .
• –print-consumer-quota-metrics – Flag indicating whether or not shopper metrics ought to be printed on the terminal by ConsumerApp-1.
• –cw-dimension-name – Amazon CloudWatch dimension title that shall be used to publish shopper throttling metrics from ConsumerApp-1.
• –cw-dimension-value – CloudWatch dimension worth that shall be used to publish shopper throttling metrics from ConsumerApp-1.
• –cw-namespace – Namespace the place ConsumerApp-1 will publish CloudWatch metrics in an effort to monitor throttling.

5. Should you’re glad with the remainder of parameters, use the next command and alter --assume-role-arn and --region as per your surroundings:

java -jar kafka-consumer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn arn:aws:iam::111111111111:position/MSKStack-TopicBReadRole-xxxxxxxxxxx 
--topic-name Subject-B 
--region <AWS Area> 
--consumer-group consumerapp-1-cg 
--role-session-name consumerapp-1-role-session 
--client-id consumerapp-1-client-id 
--print-consumer-quota-metrics Y 
--cw-dimension-name ConsumerApp 
--cw-dimension-value ConsumerApp-1 
--cw-namespace ConsumerApps

The fetch-throttle-time-avg and fetch-throttle-time-max shopper metrics ought to show 0.0, indicating no throttling is going on for ConsumerApp-1. Keep in mind that we haven’t set the eat quota for ConsumerApp-1 but. Let it run for some time.

Run the ConsumerApp-2 software

To run the ConsumerApp-2 software, full the next steps:

1. On the Amazon EC2 console, choose the ConsumerApp-2 EC2 occasion and select Join.
2. On the Session Supervisor tab, select Join.
3. Run the next instructions on the brand new tab that opens in your browser:

sudo su - ec2-user

# Set your AWS area
aws configure set area <aws area>

# Set BOOTSTRAP_BROKERS_IAM variable to MSK cluster's IAM endpoint
BOOTSTRAP_BROKERS_IAM=<Use the worth of BOOTSTRAP_BROKERS_IAM that you just famous earlier> 

echo "export BOOTSTRAP_BROKERS_IAM=$(echo $BOOTSTRAP_BROKERS_IAM)" >> .bash_profile

# Clone GitHub repository containing supply code for shopper functions
git clone https://github.com/aws-samples/amazon-msk-kafka-quotas.git

cd amazon-msk-kafka-quotas/uber-jars/

4. Run the ConsumerApp-2 software to begin consuming messages from Subject-B:

java -jar kafka-consumer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn <Subject-B-Learn-Function-ARN> 
--topic-name <Subject-Title> 
--region <AWS Area> 
--consumer-group <ConsumerApp-2 client group title> 
--role-session-name <position session title for ConsumerApp-2 to make use of throughout STS assume position name> 
--client-id <ConsumerApp-2 shopper.id> 
--print-consumer-quota-metrics Y 
--cw-dimension-name <CloudWatch Metrics Dimension Title> 
--cw-dimension-value <CloudWatch Metrics Dimension Worth> 
--cw-namespace <CloudWatch Metrics Namespace>

The code has related command line parameters particulars as ConsumerApp-1 mentioned beforehand, aside from the next:

• –consumer-group – Client group title for ConsumerApp-2, as specified through the stack replace.
• –role-session-name – ConsumerApp-2 assumes the Subject-B-Learn-Function utilizing the AWS STS SDK. ConsumerApp-2 will use this position session title when calling the assumeRole operate.
• –client-id – Shopper ID for ConsumerApp-2 .

5. Should you’re glad with the remainder of parameters, use the next command and alter --assume-role-arn and --region as per your surroundings:

java -jar kafka-consumer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn arn:aws:iam::111111111111:position/MSKStack-TopicBReadRole-xxxxxxxxxxx 
--topic-name Subject-B 
--region <AWS Area> 
--consumer-group consumerapp-2-cg 
--role-session-name consumerapp-2-role-session 
--client-id consumerapp-2-client-id 
--print-consumer-quota-metrics Y 
--cw-dimension-name ConsumerApp 
--cw-dimension-value ConsumerApp-2 
--cw-namespace ConsumerApps

The fetch-throttle-time-avg and fetch-throttle-time-max shopper metrics ought to show 0.0, indicating no throttling is going on for ConsumerApp-2. Keep in mind that we haven’t set the eat quota for ConsumerApp-2 but. Let it run for some time.

Run the ProducerApp-1 software

To run the ProducerApp-1 software, full the next steps:

1. On the Amazon EC2 console, choose the ProducerApp-1 EC2 occasion and select Join.
2. On the Session Supervisor tab, select Join.
3. Run the next instructions on the brand new tab that opens in your browser:

sudo su - ec2-user

# Set your AWS area
aws configure set area <aws area>

# Set BOOTSTRAP_BROKERS_IAM variable to MSK cluster's IAM endpoint
BOOTSTRAP_BROKERS_IAM=<Use the worth of BOOTSTRAP_BROKERS_IAM that you just famous earlier> 

echo "export BOOTSTRAP_BROKERS_IAM=$(echo $BOOTSTRAP_BROKERS_IAM)" >> .bash_profile

# Clone GitHub repository containing supply code for shopper functions
git clone https://github.com/aws-samples/amazon-msk-kafka-quotas.git

cd amazon-msk-kafka-quotas/uber-jars/

4. Run the ProducerApp-1 software to begin sending messages to Subject-B:

java -jar kafka-producer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn <Subject-B-Write-Function-ARN> 
--topic-name <Subject-Title> 
--region <AWS Area> 
--num-messages <Variety of occasions> 
--role-session-name <position session title for ProducerApp-1 to make use of throughout STS assume position name> 
--client-id <ProducerApp-1 shopper.id> 
--producer-type <Producer Sort, choices are sync or async> 
--print-producer-quota-metrics Y 
--cw-dimension-name <CloudWatch Metrics Dimension Title> 
--cw-dimension-value <CloudWatch Metrics Dimension Worth> 
--cw-namespace <CloudWatch Metrics Namespace>

You could find the supply code on GitHub to your reference. The command line parameter particulars are as follows:

• –bootstrap-servers – MSK cluster bootstrap brokers IAM endpoint.
• –assume-role-arn – Subject-B-Write-Function IAM position ARN. Assuming this position, ProducerApp-1 will write messages to the subject.
• –topic-name – ProducerApp-1 will ship messages to this matter. The default is Subject-B.
• –area – AWS Area you’re utilizing.
• –num-messages – Variety of messages the ProducerApp-1 software will ship to the subject.
• –role-session-name – ProducerApp-1 assumes the Subject-B-Write-Function utilizing the AWS STS SDK. ProducerApp-1 will use this position session title when calling the assumeRole operate.
• –client-id – Shopper ID of ProducerApp-1 .
• –producer-type – ProducerApp-1might be run both synchronously or asynchronously. Choices are sync or async.
• –print-producer-quota-metrics – Flag indicating whether or not the shopper metrics ought to be printed on the terminal by ProducerApp-1.
• –cw-dimension-name – CloudWatch dimension title that shall be used to publish shopper throttling metrics from ProducerApp-1.
• –cw-dimension-value – CloudWatch dimension worth that shall be used to publish shopper throttling metrics from ProducerApp-1.
• –cw-namespace – The namespace the place ProducerApp-1 will publish CloudWatch metrics in an effort to monitor throttling.

5. Should you’re glad with the remainder of parameters, use the next command and alter --assume-role-arn and --region as per your surroundings. To run a synchronous Kafka producer, it makes use of the choice --producer-type sync:

java -jar kafka-producer.jar --bootstrap-servers $BOOTSTRAP_BROKERS_IAM 
--assume-role-arn arn:aws:iam::111111111111:position/MSKStack-TopicBWriteRole-xxxxxxxxxxxx 
--topic-name Subject-B 
--region <AWS Area> 
--num-messages 10000000 
--role-session-name producerapp-1-role-session 
--client-id producerapp-1-client-id 
--producer-type sync 
--print-producer-quota-metrics Y 
--cw-dimension-name ProducerApp 
--cw-dimension-value ProducerApp-1 
--cw-namespace ProducerApps

Alternatively, use --producer-type async to run an asynchronous producer. For extra particulars, seek advice from Asynchronous ship.

The produce-throttle-time-avg and produce-throttle-time-max shopper metrics ought to show 0.0, indicating no throttling is going on for ProducerApp-1. Keep in mind that we haven’t set the produce quota for ProducerApp-1 but. Verify that ConsumerApp-1 and ConsumerApp-2 can eat messages and see they aren’t throttled. Cease the patron and producer shopper functions by urgent Ctrl+C of their respective browser tabs.

Set produce and eat quotas for shopper functions

Now that we now have run the producer and client functions with out quotas, we set their quotas and rerun them.

Open the Classes Supervisor terminal for the MSKAdminInstance EC2 occasion as described earlier and run the next instructions to search out the default configuration of one of many brokers within the MSK cluster. MSK clusters are provisioned with the default Kafka quotas configuration.

# Describe Dealer-1 default configurations
kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--entity-type brokers 
--entity-name 1 
--all --describe > broker1_default_configurations.txt

cat broker1_default_configurations.txt | grep quota.client.default
cat broker1_default_configurations.txt | grep quota.producer.default

The next screenshot exhibits the Dealer-1 default values for quota.client.default and quota.producer.default.

ProducerApp-1 quota configuration

Change placeholders in all of the instructions on this part with values that correspond to your account.

In keeping with the structure diagram mentioned earlier, set the ProducerApp-1 produce quota to 1024 bytes/second. For <ProducerApp-1 Shopper Id> and <ProducerApp-1 Function Session>, be sure to use the identical values that you just used whereas working ProducerApp-1 earlier (producerapp-1-client-id and producerapp-1-role-session, respectively):

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--alter --add-config 'producer_byte_rate=1024' 
--entity-type shoppers --entity-name <ProducerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBWriteRole-xxxxxxxxxxx/<ProducerApp-1 Function Session>

Confirm the ProducerApp-1 produce quota utilizing the next command:

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--describe 
--entity-type shoppers --entity-name <ProducerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBWriteRole-xxxxxxxxxxx/<ProducerApp-1 Function Session>

You may take away the ProducerApp-1 produce quota through the use of the next command, however don’t run the command as we’ll take a look at the quotas subsequent.

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--alter --delete-config producer_byte_rate 
--entity-type shoppers --entity-name <ProducerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBWriteRole-xxxxxxxxxxx/<ProducerApp-1 Function Session>

ConsumerApp-1 quota configuration

Change placeholders in all of the instructions on this part with values that correspond to your account.

Let’s set a eat quota of 5120 bytes/second for ConsumerApp-1. For <ConsumerApp-1 Shopper Id> and <ConsumerApp-1 Function Session>, be sure to use the identical values that you just used whereas working ConsumerApp-1 earlier (consumerapp-1-client-id and consumerapp-1-role-session, respectively):

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--alter --add-config 'consumer_byte_rate=5120' 
--entity-type shoppers --entity-name <ConsumerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBReadRole-xxxxxxxxxxx/<ConsumerApp-1 Function Session>

Confirm the ConsumerApp-1 eat quota utilizing the next command:

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--describe 
--entity-type shoppers --entity-name <ConsumerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBReadRole-xxxxxxxxxxx/<ConsumerApp-1 Function Session>

You may take away the ConsumerApp-1 eat quota, through the use of the next command, however don’t run the command as we’ll take a look at the quotas subsequent.

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--alter --delete-config consumer_byte_rate 
--entity-type shoppers --entity-name <ConsumerApp-1 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBReadRole-xxxxxxxxxxx/<ConsumerApp-1 Function Session>

ConsumerApp-2 quota configuration

Change placeholders in all of the instructions on this part with values that correspond to your account.

Let’s set a eat quota of 1024 bytes/second for ConsumerApp-2. For <ConsumerApp-2 Shopper Id> and <ConsumerApp-2 Function Session>, be sure to use the identical values that you just used whereas working ConsumerApp-2 earlier (consumerapp-2-client-id and consumerapp-2-role-session, respectively):

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--alter --add-config 'consumer_byte_rate=1024' 
--entity-type shoppers --entity-name <ConsumerApp-2 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBReadRole-xxxxxxxxxxx/<ConsumerApp-2 Function Session>

Confirm the ConsumerApp-2 eat quota utilizing the next command:

kafka-configs.sh --bootstrap-server $BOOTSTRAP_BROKERS_IAM 
--command-config config_iam.properties 
--describe 
--entity-type shoppers --entity-name <ConsumerApp-2 Shopper Id> 
--entity-type customers --entity-name arn:aws:sts::<AWS Account Id>:assumed-role/MSKStack-TopicBReadRole-xxxxxxxxxxx/<ConsumerApp-2 Function Session>

As with ConsumerApp-1, you possibly can take away the ConsumerApp-2 eat quota utilizing the identical command with ConsumerApp-2 shopper and consumer particulars.

Rerun the producer and client functions after setting quotas

Let’s rerun the functions to confirm the impact of the quotas.

Rerun ProducerApp-1

Rerun ProducerApp-1 in synchronous mode with the identical command that you just used earlier. The next screenshot illustrates that when ProducerApp-1 reaches its quota on any of the brokers, the produce-throttle-time-avg and produce-throttle-time-max shopper metrics worth shall be above 0.0. A price above 0.0 signifies that ProducerApp-1 is throttled. Enable ProducerApp-1 to run for a couple of seconds after which cease it through the use of Ctrl+C.

You may as well take a look at the impact of the produce quota by rerunning ProducerApp-1 once more in asynchronous mode (--producer-type async). Just like a synchronous run, the next screenshot illustrates that when ProducerApp-1 reaches its quota on any of the brokers, the produce-throttle-time-avg and produce-throttle-time-max shopper metrics worth shall be above 0.0. A price above 0.0 signifies that ProducerApp-1 is throttled. Enable asynchronous ProducerApp-1 to run for some time.

You’ll finally see a TimeoutException stating org.apache.kafka.widespread.errors.TimeoutException: Expiring xxxxx file(s) for Subject-B-2:xxxxxxx ms has handed since batch creation

When utilizing an asynchronous producer and sending messages at a price higher than the dealer can settle for because of the quota, the messages shall be queued within the shopper software reminiscence first. The shopper will finally run out of buffer area if the speed of sending messages continues to exceed the speed of accepting messages, inflicting the subsequent Producer.ship() name to be blocked. Producer.ship() will finally throw a TimeoutException if the timeout delay shouldn’t be adequate to permit the dealer to catch as much as the producer software. Cease ProducerApp-1 through the use of Ctrl+C.

Rerun ConsumerApp-1

Rerun ConsumerApp-1 with the identical command that you just used earlier. The next screenshot illustrates that when ConsumerApp-1 reaches its quota, the fetch-throttle-time-avg and fetch-throttle-time-max shopper metrics worth shall be above 0.0. A price above 0.0 signifies that ConsumerApp-1 is throttled.

Enable ConsumerApp-1 to run for a couple of seconds after which cease it through the use of Ctrl+C.

Rerun ConsumerApp-2

Rerun ConsumerApp-2 with the identical command that you just used earlier. Equally, when ConsumerApp-2 reaches its quota, the fetch-throttle-time-avg and fetch-throttle-time-max shopper metrics worth shall be above 0.0. A price above 0.0 signifies that ConsumerApp-2 is throttled. Enable ConsumerApp-2 to run for a couple of seconds after which cease it by urgent Ctrl+C.

Shopper quota metrics in Amazon CloudWatch

In Half 1, we defined that shopper metrics are metrics uncovered by shoppers connecting to Kafka clusters. Let’s study the shopper metrics in CloudWatch.

1. On the CloudWatch console, select All metrics.
2. Underneath Customized Namespaces, select the namespace you offered whereas working the shopper functions.
3. Select the dimension title and choose produce-throttle-time-max, produce-throttle-time-avg, fetch-throttle-time-max, and fetch-throttle-time-avg metrics for all of the functions.

These metrics point out throttling habits for ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 functions examined with the quota configurations within the earlier part. The next screenshots point out the throttling of ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 primarily based on community bandwidth quotas. ProducerApp-1, ConsumerApp-1, and ConsumerApp-2 functions feed their respective shopper metrics to CloudWatch. You could find the supply code on GitHub to your reference.

Safe shopper ID and position session title

We mentioned the way to configure Kafka quotas utilizing an software’s shopper ID and authenticated consumer principal. When a shopper software assumes an IAM position to entry Kafka subjects on a MSK cluster with IAM authentication enabled, its authenticated consumer principal is represented within the following format (for extra data, seek advice from IAM identifiers):

arn:aws:sts::111111111111:assumed-role/Subject-B-Write-Function/producerapp-1-role-session

It incorporates the position session title (on this case, producerapp-1-role-session) used within the shopper software whereas assuming an IAM position by way of the AWS STS SDK. The shopper software supply code is out there to your reference. The shopper ID is a logical title string (for instance, producerapp-1-client-id) that’s configured within the software code by the applying staff. Due to this fact, an software can impersonate one other software if it obtains the shopper ID and position session title of the opposite software, and if it has permission to imagine the identical IAM position.

As proven within the structure diagram, ConsumerApp-1 and ConsumerApp-2 are two separate shopper functions with their respective quota allocations. As a result of each have permission to imagine the identical IAM position (Subject-B-Learn-Function) within the demo account, they’re allowed to eat messages from Subject-B. Thus, MSK cluster brokers distinguish them primarily based on their shopper IDs and customers (which comprise their respective position session title values). If ConsumerApp-2 by some means obtains the ConsumerApp-1 position session title and shopper ID, it might probably impersonate ConsumerApp-1 by specifying the ConsumerApp-1 position session title and shopper ID within the software code.

Let’s assume ConsumerApp-1 makes use of consumerapp-1-client-id and consumerapp-1-role-session as its shopper ID and position session title, respectively. Due to this fact, ConsumerApp-1's authenticated consumer principal will seem as follows when it assumes the Subject-B-Learn-Function IAM position:

arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Learn-Function/consumerapp-1-role-session

Equally, ConsumerApp-2 makes use of consumerapp-2-client-id and consumerapp-2-role-session as its shopper ID and position session title, respectively. Due to this fact, ConsumerApp-2's authenticated consumer principal will seem as follows when it assumes the Subject-B-Learn-Function IAM position:

arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Learn-Function/consumerapp-2-role-session

If ConsumerApp-2 obtains ConsumerApp-1's shopper ID and position session title and specifies them in its software code, MSK cluster brokers will deal with it as ConsumerApp-1 and think about its shopper ID as consumerapp-1-client-id, and the authenticated consumer principal as follows:

arn:aws:sts::<AWS Account Id>:assumed-role/Subject-B-Learn-Function/consumerapp-1-role-session

This enables ConsumerApp-2 to eat knowledge from the MSK cluster at a most price of 5120 bytes per second quite than 1024 bytes per second as per its unique quota allocation. Consequently, ConsumerApp-1's throughput shall be negatively impacted if ConsumerApp-2 runs concurrently.

Enhanced structure

You may introduce AWS Secrets and techniques Supervisor and AWS Key Administration Service (AWS KMS) within the structure to safe functions’ shopper IDs and position session names. To offer stronger governance, the functions’ shopper ID and position session title should be saved as encrypted secrets and techniques within the Secrets and techniques Supervisor. The IAM useful resource insurance policies related to encrypted secrets and techniques and a KMS buyer managed key (CMK) will enable functions to entry and decrypt solely their respective shopper ID and position session title. On this means, functions won’t be able to entry one another’s shopper ID and position session title and impersonate each other. The next picture exhibits the improved structure.

The up to date move has the next levels:

• P1 – ProducerApp-1 retrieves its client-id and role-session-name secrets and techniques from Secrets and techniques Supervisor
• P2 – ProducerApp-1 configures the key client-id as CLIENT_ID_CONFIG within the software code, and assumes Subject-B-Write-Function (by way of its ProducerApp-1-Function IAM position) by passing the key role-session-name to the AWS STS SDK assumeRole operate name
• P3 – With the Subject-B-Write-Function IAM position assumed, ProducerApp-1 begins sending messages to Subject-B
• C1 – ConsumerApp-1 and ConsumerApp-2 retrieve their respective client-id and role-session-name secrets and techniques from Secrets and techniques Supervisor
• C2 – ConsumerApp-1 and ConsumerApp-2 configure their respective secret client-id as CLIENT_ID_CONFIG of their software code, and assume Subject-B-Write-Function (by way of ConsumerApp-1-Function and ConsumerApp-2-Function IAM roles, respectively) by passing their secret role-session-name within the AWS STS SDK assumeRole operate name
• C3 – With the Subject-B-Learn-Function IAM position assumed, ConsumerApp-1 and ConsumerApp-2 begin consuming messages from Subject-B

Discuss with the documentation for AWS Secrets and techniques Supervisor and AWS KMS to get a greater understanding of how they match into the structure.

Clear up assets

Navigate to the CloudFormation console and delete the MSKStack stack. All assets created throughout this put up shall be deleted.

Conclusion

On this put up, we lined detailed steps to configure Amazon MSK quotas and demonstrated their impact by way of pattern shopper functions. As well as, we mentioned how you should use shopper metrics to find out if a shopper software is throttled. We additionally highlighted a possible situation with plaintext shopper IDs and position session names. We advocate implementing Kafka quotas with Amazon MSK utilizing Secrets and techniques Supervisor and AWS KMS as per the revised structure diagram to make sure a zero-trust structure.

When you’ve got suggestions or questions on this put up, together with the revised structure, we’d be comfortable to listen to from you. We hope you loved studying this put up.

In regards to the Writer

Vikas Bajaj is a Senior Supervisor, Options Architects, Monetary Providers at Amazon Internet Providers. With over 20 years of expertise in monetary companies and dealing with digital-native companies, he advises clients on product design, know-how roadmaps, and software architectures.