How Ontraport diminished knowledge processing value by 80% with AWS Glue

This publish is written in collaboration with Elijah Ball from Ontraport.

Clients are implementing knowledge and analytics workloads within the AWS Cloud to optimize value. When implementing knowledge processing workloads in AWS, you might have the choice to make use of applied sciences like Amazon EMR or serverless applied sciences like AWS Glue. Each choices decrease the undifferentiated heavy lifting actions like managing servers, performing upgrades, and deploying safety patches and assist you to concentrate on what’s vital: assembly core enterprise aims. The distinction between each approaches can play a important position in enabling your group to be extra productive and revolutionary, whereas additionally saving cash and sources.

Companies like Amazon EMR concentrate on providing you flexibility to assist knowledge processing workloads at scale utilizing frameworks you’re accustomed to. For instance, with Amazon EMR, you possibly can select from a number of open-source knowledge processing frameworks akin to Apache Spark, Apache Hive, and Presto, and fine-tune workloads by customizing issues akin to cluster occasion sorts on Amazon Elastic Compute Cloud (Amazon EC2) or use containerized environments working on Amazon Elastic Kubernetes Service (Amazon EKS). This selection is greatest suited when migrating workloads from huge knowledge environments like Apache Hadoop or Spark, or when utilized by groups which might be aware of open-source frameworks supported on Amazon EMR.

Serverless providers like AWS Glue decrease the necessity to consider servers and concentrate on providing further productiveness and DataOps tooling for accelerating knowledge pipeline growth. AWS Glue is a serverless knowledge integration service that helps analytics customers uncover, put together, transfer, and combine knowledge from a number of sources by way of a low-code or no-code strategy. This selection is greatest suited when organizations are resource-constrained and must construct knowledge processing workloads at scale with restricted experience, permitting them to expedite growth and diminished Complete Price of Possession (TCO).

On this publish, we present how our AWS buyer Ontraport evaluated using AWS Glue and Amazon EMR to scale back TCO, and the way they diminished their storage value by 92% and their processing value by 80% with just one full-time developer.

Ontraport’s workload and answer

Ontraport is a CRM and automation service that powers companies’ advertising, gross sales and operations multi function place—empowering companies to develop quicker and ship extra worth to their clients.

Log processing and evaluation is important to Ontraport. It permits them to supply higher providers and perception to clients akin to e mail marketing campaign optimization. For instance, e mail logs alone report 3–4 occasions for each one of many 15–20 million messages Ontraport sends on behalf of their purchasers every day. Evaluation of e mail transactions with suppliers akin to Google and Microsoft enable Ontraport’s supply crew to optimize open charges for the campaigns of purchasers with huge contact lists.

A number of the huge log contributors are internet server and CDN occasions, e mail transaction information, and customized occasion logs inside Ontraport’s proprietary functions. The next is a pattern breakdown of their each day log contributions:

Ontraport’s answer makes use of Amazon Kinesis and Amazon Kinesis Knowledge Firehose to ingest log knowledge and write latest information into an Amazon OpenSearch Service database, from the place analysts and directors can analyze the final 3 months of knowledge. Customized utility logs report interactions with the Ontraport CRM so consumer accounts might be audited or recovered by the client assist crew. Initially, all logs had been retained again to 2018. Retention is multi-leveled by age:

• Lower than 1 week – OpenSearch scorching storage
• Between 1 week and three months – OpenSearch chilly storage
• Greater than 3 months – Extract, remodel, and cargo (ETL) processed in Amazon Easy Storage Service (Amazon S3), obtainable by Amazon Athena

The next diagram reveals the structure of their log processing and analytics knowledge pipeline.

Evaluating the optimum answer

With the intention to optimize storage and evaluation of their historic information in Amazon S3, Ontraport carried out an ETL course of to remodel and compress TSV and JSON recordsdata into Parquet recordsdata with partitioning by the hour. The compression and transformation helped Ontraport cut back their S3 storage prices by 92%.

In part 1, Ontraport carried out an ETL workload with Amazon EMR. Given the dimensions of their knowledge (tons of of billions of rows) and just one developer, Ontraport’s first try on the Apache Spark utility required a 16-node EMR cluster with r5.12xlarge core and job nodes. The configuration allowed the developer to course of 1 12 months of knowledge and decrease out-of-memory points with a tough model of the Spark ETL utility.

To assist optimize the workload, Ontraport reached out to AWS for optimization suggestions. There have been a substantial variety of choices to optimize the workload inside Amazon EMR, akin to right-sizing Amazon Elastic Compute Cloud (Amazon EC2) occasion kind primarily based on workload profile, modifying Spark YARN reminiscence configuration, and rewriting parts of the Spark code. Contemplating the useful resource constraints (just one full-time developer), the AWS crew beneficial exploring related logic with AWS Glue Studio.

A number of the preliminary advantages with utilizing AWS Glue for this workload embrace the next:

• AWS Glue has the idea of crawlers that gives a no-code strategy to catalog knowledge sources and establish schema from a number of knowledge sources, on this case, Amazon S3.
• AWS Glue offers built-in knowledge processing capabilities with summary strategies on high of Spark that cut back the overhead required to develop environment friendly knowledge processing code. For instance, AWS Glue helps a DynamicFrame class akin to a Spark DataFrame that gives further flexibility when working with semi-structured datasets and might be rapidly remodeled right into a Spark DataFrame. DynamicFrames might be generated immediately from crawled tables or immediately from recordsdata in Amazon S3. See the next instance code:

  dyf = glueContext.create_dynamic_frame.from_options(
  
  connection_type="s3",
  connection_options = {'paths': [s3://<bucket/paths>]},
  format="json")

• It minimizes the necessity for Ontraport to right-size occasion sorts and auto scaling configurations.
• Utilizing AWS Glue Studio interactive periods permits Ontraport to rapidly iterate when code adjustments the place wanted when detecting historic log schema evolution.

Ontraport needed to course of 100 terabytes of log knowledge. The price of processing every terabyte with the preliminary configuration was roughly $500. That value got here all the way down to roughly $100 per terabyte after utilizing AWS Glue. By utilizing AWS Glue and AWS Glue Studio, Ontraport’s value of processing the roles was diminished by 80%.

Diving deep into the AWS Glue workload

Ontraport’s first AWS Glue utility was a PySpark workload that ingested knowledge from TSV and JSON recordsdata in Amazon S3, carried out fundamental transformations on timestamp fields, and transformed the info kinds of a pair fields. Lastly, it writes output knowledge right into a curated S3 bucket as compressed Parquet recordsdata of roughly 1 GB in dimension and partitioned in 1-hour intervals to optimize for queries with Athena.

With an AWS Glue job configured with 10 employees of the sort G.2x configuration, Ontraport was capable of course of roughly 500 million information in lower than 60 minutes. When processing 10 billion information, they had been capable of improve the job configuration to a most of 100 employees with auto scaling enabled to finish the job inside 1 hour.

What’s subsequent?

Ontraport has been capable of course of logs as early as 2018. The crew is updating the processing code to permit for situations of schema evolution (akin to new fields) and parameterized some elements to completely automate the batch processing. They’re additionally seeking to fine-tune the variety of provisioned AWS Glue employees to acquire optimum price-performance.

Conclusion

On this publish, we confirmed you ways Ontraport used AWS Glue to assist cut back growth overhead and simplify growth efforts for his or her ETL workloads with just one full-time developer. Though providers like Amazon EMR supply nice flexibility and optimization, the benefit of use and simplification in AWS Glue typically supply a quicker path for cost-optimization and innovation for small and medium companies. For extra details about AWS Glue, take a look at Getting Began with AWS Glue.

Concerning the Authors

Elijah Ball has been a Sys Admin at Ontraport for 12 years. He’s presently working to maneuver Ontraport’s manufacturing workloads to AWS and develop knowledge evaluation methods for Ontraport.

Pablo Redondo is a Principal Options Architect at Amazon Internet Companies. He’s an information fanatic with over 16 years of FinTech and healthcare trade expertise and is a member of the AWS Analytics Technical Subject Group (TFC). Pablo has been main the AWS Achieve Insights Program to assist AWS clients obtain higher insights and tangible enterprise worth from their knowledge analytics initiatives.

Vikram Honmurgi is a Buyer Options Supervisor at Amazon Internet Companies. With over 15 years of software program supply expertise, Vikram is obsessed with helping clients and accelerating their cloud journey, delivering frictionless migrations, and making certain our clients seize the complete potential and sustainable enterprise benefits of migrating to the AWS Cloud.