Stream VPC Circulation Logs to Datadog by way of Amazon Kinesis Information Firehose

It’s widespread to retailer the logs generated by buyer’s purposes and providers in varied instruments. These logs are necessary for compliance, audits, troubleshooting, safety incident responses, assembly safety insurance policies, and lots of different functions. You may carry out log evaluation on these logs to grasp customers’ software conduct and patterns to make knowledgeable selections.

When operating workloads on Amazon Internet Companies (AWS), you should analyze Amazon Digital Non-public Cloud (Amazon VPC) Circulation Logs to trace the IP visitors going to and from the community interfaces for the workloads of their VPC. Analyzing VPC stream logs helps you perceive how your purposes are speaking over the VPC community and acts as a important supply of knowledge to the community in your VPC.

You may simply ship knowledge to supported locations utilizing the Amazon Kinesis Information Firehose integration with VPC stream logs. Kinesis Information Firehose is a totally managed service for delivering near-real-time streaming knowledge to varied locations for storage and performing near-real-time analytics. With its extensible knowledge transformation capabilities, you can too streamline log processing and log supply pipelines right into a single Kinesis Information Firehose supply stream. You may carry out analytics on VPC stream logs delivered out of your VPC utilizing the Kinesis Information Firehose integration with Datadog as a vacation spot.

Datadog is a monitoring and safety platform and AWS Associate Community (APN) Superior Know-how Associate with AWS Competencies in AWS Cloud Operations, DevOps, Migration, Safety, Networking, Containers, and Microsoft Workloads, together with many others.

Datadog allows you to simply discover and analyze logs to realize deeper insights into the state of your purposes and AWS infrastructure. You may analyze all of your AWS service logs whereas storing solely those you want, generate metrics from aggregated logs to uncover, and ship alerts about traits in your AWS providers.

On this put up, you discover ways to combine VPC stream logs with Kinesis Information Firehose and ship it to Datadog.

Answer overview

This answer makes use of native integration of VPC stream logs streaming to Kinesis Information Firehose. We use a Kinesis Information Firehose supply stream to buffer the streamed VPC stream logs to a Datadog vacation spot endpoint in your Datadog account. You need to use these logs with Datadog Log Administration and Datadog Cloud SIEM to investigate the well being, efficiency, and safety of your cloud assets.

The next diagram illustrates the answer structure.

We stroll you thru the next high-level steps:

1. Hyperlink your AWS account together with your Datadog account.
2. Create the Kinesis Information Firehose stream the place VPC service streams the stream logs.
3. Create the VPC stream log subscription to Kinesis Information Firehose.
4. Visualize VPC stream logs within the Datadog dashboard.

The account ID 123456781234 used on this put up is a dummy account. It’s used just for demonstration functions.

Stipulations

You need to have the next conditions:

Hyperlink your AWS account together with your Datadog account for AWS integration

Comply with the directions offered on the Datadog web site for AWS Integration. To configure log archiving and enrich the log knowledge despatched out of your AWS account with helpful context, hyperlink the accounts. If you full the linking setup, proceed to the next step.

Create a Kinesis Information Firehose stream

Now that your Datadog integration with AWS is full, you’ll be able to create a Kinesis Information Firehose supply stream the place VPC Circulation Logs are streamed by following these steps:

1. On the Amazon Kinesis console, select Kinesis Information Firehose within the navigation pane.
2. Select Create supply stream.
3. Select Direct PUT because the supply.
4. Set Vacation spot as Datadog.
   

5. For Supply stream title, enter PUT-DATADOG-DEMO.
6. Maintain Information transformation set to Disabled underneath Rework data.
7. In Vacation spot settings, for HTTP endpoint URL, select the specified log’s HTTP endpoint primarily based in your Area and Datadog account configuration.
   
8. For API key, enter your Datadog API key.

This enables your supply stream to publish VPC Circulation logs to the Datadog endpoint. API keys are distinctive to your group. An API key is required by the Datadog Agent to submit metrics and occasions to Datadog.

9. Set Content material encoding to GZIP to scale back the dimensions of information transferred.
10. Set the Retry length to 60.You may change the Retry length worth if you should. This is dependent upon the request dealing with capability of the Datadog endpoint.
    
    Underneath Buffer hints, Buffer measurement and Buffer interval are set with default values for Datadog integration.
    

11. Underneath Backup settings, as talked about within the conditions, select the S3 bucket that you simply created to retailer failed logs and backup with particular prefix.
12. Underneath S3 buffer hints part, set Buffer measurement to five and Buffer interval to 300.

You may change the S3 buffer measurement and interval primarily based in your necessities.

13. Underneath S3 compression and encryption, choose GZIP for Compression for knowledge data or one other compression methodology of your selection.

Compressing knowledge reduces the required space for storing.

14. Choose Disabled for Encryption of the info data. You may allow encryption of the info data to safe entry to your logs.
    

15. Optionally, in Superior settings, choose Allow server-side encryption for supply data in supply stream.
    You need to use AWS managed keys or a CMK managed by you for the encryption kind.

16. Allow CloudWatch error logging.
17. Select Create or replace IAM position, which is created by Kinesis Information Firehose as a part of this stream.
    

18. Select Subsequent.
19. Overview your settings.
20. Select Create supply stream.

Create a VPC stream logs subscription

Create a VPC stream logs subscription for the Kinesis Information Firehose supply stream you created within the earlier step:

1. On the Amazon VPC console, select Your VPCs.
2. Choose the VPC that you simply to create the stream log for.
3. On the Actions menu, select Create stream log.
   

4. Choose All to ship all stream log data to the Firehose vacation spot.

If you wish to filter the stream logs, you could possibly alternatively choose Settle for or Reject.

5. For Most aggregation interval, choose 10 minutes or the minimal setting of 1 minute when you want the stream log knowledge to be obtainable for near-real-time evaluation in Datadog.
6. For Vacation spot, choose Ship to Kinesis Information Firehose in the identical account if the supply stream is ready up on the identical account the place you create the VPC stream logs.

If you wish to ship the info to a unique account, check with Publish stream logs to Kinesis Information Firehose.

7. Select an possibility for Log report format:
8. In case you go away Log report format because the AWS default format, the stream logs are despatched as model 2 format.
9. Alternatively, you’ll be able to specify the customized fields for stream logs to seize and ship it to Datadog.

For extra info on log format and obtainable fields, check with Circulation log data.

10. Select Create stream log.
    

Now let’s discover the VPC stream logs in Datadog.

Visualize VPC stream logs within the Datadog dashboard

Within the Logs Search possibility within the navigation pane, filter to supply:vpc. The VPC stream logs out of your VPC are within the Datadog Log Explorer and are mechanically parsed so you’ll be able to analyze your logs by supply, vacation spot, motion, or different attributes.

Clear up

After you take a look at this answer, delete all of the assets you created to keep away from incurring future expenses. Discuss with the next hyperlinks for directions for deleting the assets:

Conclusion

On this put up, we walked by an answer of the right way to combine VPC stream logs with a Kinesis Information Firehose supply stream, ship it to a Datadog vacation spot with no code, and visualize it in a Datadog dashboard. With Datadog, you’ll be able to simply discover and analyze logs to realize deeper insights into the state of your purposes and AWS infrastructure.

Do that new, fast, and hassle-free means of sending your VPC stream logs to a Datadog vacation spot utilizing Kinesis Information Firehose.

In regards to the Writer

Chaitanya Shah is a Sr. Technical Account Supervisor(TAM) with AWS, primarily based out of New York. He has over 22 years of expertise working with enterprise clients. He likes to code and actively contributes to the AWS options labs to assist clients clear up complicated issues. He supplies steering to AWS clients on finest practices for his or her AWS Cloud migrations. He’s additionally specialised in AWS knowledge switch and the info and analytics area.