Job Notifications in SQL Stream Builder

Posted in Technical |
February 09, 2023 5 min learn

Particular co-author credit: Adam Andras Toth, Software Engineer Intern

With enterprises’ wants for information analytics and processing getting extra complicated by the day, Cloudera goals to maintain up with these wants, providing consistently evolving, cutting-edge options to all of your information associated issues. Cloudera Stream Processing goals to take real-time information analytics to the subsequent degree. We’re excited to spotlight job monitoring with notifications, a brand new characteristic for SQL Stream Builder (SSB).

The sudden failing of a fancy information pipeline can result in devastating penalties—particularly if it goes unnoticed. An actual-time monetary fraud detector, or a fancy structure gathering and aggregating information to create insights and permit their clients to make data-driven choices—these are methods which have little to no room for error or prolonged downtime. For this reason we construct job notifications performance into SSB, to ship most reliability in your complicated real-time information pipelines.

Job notifications may also help just be sure you can detect failed jobs with out checking on the UI, which might save loads of time for the person. This characteristic may be very helpful, particularly when the person has quite a few jobs operating and conserving observe of their state can be arduous with out notifications.

First, we want to introduce the structure of job notifications. Allow us to use a determine to display how job notifications match into SSB, then we are going to focus on every kind individually.

Overview

In SSB you possibly can handle a number of tasks. Initiatives signify the software program growth life cycle (SDLC) in SQL Stream Builder (SSB): you possibly can create an atmosphere for growing SQL jobs, verifying the outcomes after which pushing them to manufacturing. The sources for job creation are shared throughout the jobs of a undertaking, and there are sources accessible that can be utilized between the tasks. The essential idea of a undertaking can be expanded for collaboration by sharing the tasks with group members in streaming SQL console, or utilizing supply management to synchronize the undertaking with a Git repository. 

Job notifications additionally belong to tasks. Meaning in a single undertaking you possibly can outline a number of notifications, and people notifications can solely belong to the roles of that undertaking. Within the determine beneath, you possibly can see the structure of a undertaking from the attitude of job notifications. As of now there are two forms of notifications: electronic mail and webhook. The notifications can be organized into teams. The advantage of that is that if you wish to assign the identical set of notifications to a number of jobs you don’t have to do that one after the other in each job, you possibly can simply create a notification group and assign that to the roles. One notification could be included in a number of teams and a bunch may even include one other group.

Within the determine beneath, the identical job notifications are marked with the identical colour. As you possibly can see within the undertaking we’ve three jobs. Within the first one we solely have notifications, so if that job fails these 4 notifications will hearth. In the second we’ve a webhook notification and a notification group that has one other webhook and an electronic mail notification, so if this job fails these three notifications will go off. The third job has a webhook notification, a bunch that comprises an electronic mail notification, and one other notification group that has two notifications, so if this job fails these 4 notifications will hearth.

Notifications

As I discussed earlier than, there are two forms of notifications and you may assign them to teams. I’ll first introduce placeholders, which you need to use to create notifications.

Placeholders

The e-mail message or webhook request that’s despatched upon the set off for a notification could be utterly personalized. Greater than that, SSB additionally permits the utilization of placeholders, which can be utilized to supply all essential data within the notification. With the power to customise messages and to make use of placeholders, customers will even doubtlessly be capable to robotically parse the incoming notifications and create automated responses for them, thus guaranteeing that crucial pipelines could be restarted with out requiring human intervention. 

The placeholders at the moment accessible for utilization are:

• jobName
• jobStatus
• jobStatusDescription
• ssbJobId
• flinkJobId
• clusterId
• lastException

You should use a placeholder within the following format: “Houston we’ve an issue, your job with identify ${jobName} has failed.”

E-mail notifications

E-mail notifications are (as you possibly can guess from its identify) sending emails to the given electronic mail deal with upon job failure. To make this work some CM properties have to be configured:

• Mail server host for job notifications: The host of the SMTP server for job failure notifications
• Mail server username for job notifications: The username to entry the SMTP server for job failure notifications
• Mail server password for job notifications: The password to entry the SMTP server for job failure notifications
• SMTP authentication for job notifications: Allow SMTP authentication for job notifications (default worth: True)
• StartTLS for job notifications: Use the StartTLS command to determine a safe connection to the SMTP server for job notifications (default worth: True)
• Job notifications sender mail deal with: Sender mail deal with for job notifications
• Mail server port for job notifications: The port of the SMTP server for job failure notifications (default worth: 587)

When you have this stuff arrange correctly and also you add a notification to your job, it is best to get an electronic mail if the job fails.

Webhook notifications

With webhook notifications you may make webhook requests upon a job failure. For those who use the placeholders accurately, then you need to use the outlined webhook endpoints of exterior functions to deal with the failures in a extra environment friendly means. (For instance, you possibly can arrange a webhook notification with Slack to ship you a message instantly if a job fails.)

Within the case of webhook notifications you possibly can set one property in CM:

• Job notifications webhook sender parallelism: Variety of threads utilized by the job notification activity to name user-specified webhooks when notifying a couple of failed or lacking job (default worth: 10)

DISCLAIMER: The payload template of a webhook notification have to be a legitimate JSON! Additionally ensure to place placeholders inside quotes!

E.g.:

• “identify”: ${jobName} is invalid
• “identify”:”${jobName}” is legitimate
• “identify”:”no matter i need right here ${jobName}” can also be legitimate

Notification teams

As I discussed above you possibly can assign your notifications into teams. This fashion you don’t want so as to add all of the notifications to the roles one after the other. A cool factor concerning the teams is that they will additionally include different notification teams. 

SSB’s job notifications characteristic is a cool method to maintain observe of your failing jobs and thus reduce the downtime of them. You simply want to verify the “allow job notifications” performance in CM is checked. The job-monitoring activity periodically queries the state of your jobs, and triggers the assigned notifications if a failed job is discovered. The examine interval in CM could be configured with the job notifications monitoring interval property (default worth: 60s).

On this part I’ll present you some video examples for the usages of the job notifications.

Create and use an E-mail notification:

Create and use a Webhook notification:

Create and use a Notification Group

Anyone can check out SSB utilizing the Stream Processing Neighborhood Version (CSP-CE). CE makes growing stream processors simple, as it may be accomplished proper out of your desktop or some other growth node. Analysts, information scientists, and builders can now consider new options, develop SQL-based stream processors regionally utilizing SQL Stream Builder powered by Flink, and develop Kafka Customers/Producers and Kafka Join Connectors, all regionally earlier than shifting to manufacturing in CDP.