Monetary and Know-how Heavyweight Decreasing Time Spent on Knowledge Discovery by One-third with Atlan
When one thinks of Nasdaq, their eponymous alternate in New York Metropolis seemingly involves thoughts. It’s the world’s second-largest, with $18 trillion in market capitalization; residence to the world’s most prestigious know-how corporations.
Michael Weiss, Product Supervisor at Nasdaq, joined Atlan on the 2023 Gartner Knowledge & Analytics Summit to share how Nasdaq is a lot greater than its flagship alternate, and the way an organization that has run on information for over 5 many years is turning into a frontrunner in information know-how, as nicely.
“It may be a shock simply how huge Nasdaq is,” Michael defined. “Nasdaq operates over 30 exchanges throughout quite a lot of asset courses throughout the globe. We’re additionally residence to over 4,100 publicly listed corporations globally. Within the U.S., we’re residence to 73% of all publicly listed tech corporations, and the 4 largest by market cap. Nasdaq is actually the house of know-how.”
A Decade of Cloud Expertise
Contemplating their place within the cautious and infrequently slow-moving monetary sector, it’d come as a shock to many who Nasdaq started their cloud journey with AWS way back to 2012. Initially, they constructed inner merchandise specializing in regulatory reporting, then income administration, supported by the cloud. “From there, we went to information warehousing and analytics. Ever since then, we’ve been systematically transferring towards the cloud,” Michael shared.
With greater than a decade of expertise working in AWS, Nasdaq is starting to maneuver the majority of its important workloads to the cloud, and is defining how the subsequent technology of capital markets will function.
Regardless of this deep expertise, Nasdaq sees ample alternative to enhance. First, buying and selling system information is difficult in measurement, and in complexity, with as many as 140 billion occasions per day processed within the U.S., alone. “If you take a look at buying and selling system information, it’s vital to comprehend it’s optimized for [operational] efficiency, not for analytics,” Michael defined.
Nasdaq’s course of for getting ready and presenting information was one other focus, with their legacy ETL instruments unable to maintain up when each the sorts of information, and the demand for information, scaled considerably. “They had been a bit inflexible,” Michael shared. “They didn’t actually alter to the place we had been making an attempt to get to on the time.”
Between the complexity of their information, legacy instruments that had been sluggish to develop to Nasdaq’s ambitions. A staff that had its arms full sustaining their technical panorama, Nasdaq’s information staff realized a possibility to raised help their enterprise companions. “Should you had been a product proprietor or enterprise consumer who needed to get a brand new set of insights into your information, it may take fairly a while to have somebody construct it out on the prevailing platform,” Michael shared. “We weren’t actually enabling our enterprise customers to get what they wanted.”
In consequence, parallel groups started to emerge, every with a singular method to creating information options. “Should you had been a enterprise unit, you may go to one in all 4 groups to try to get a solution. As a matter of truth, they’d go to all 4 groups to see who would win first,” Michael defined.
It could take a compelling imaginative and prescient and technique, backed by a extra fashionable information stack and staff construction, to raised help Nasdaq’s rising inner demand for information.
A Knowledge Chief’s Knowledge Stack
“It was solely about 18 months in the past that we determined to take a step again and re-evaluate our tech stack,” Michael defined. And in that quick time, Nasdaq has made a number of key investments.
First was an implementation of dbt, accelerating their staff’s skill to construct information fashions. “We actually needed a instrument that was versatile, and will actually allow individuals who had SQL abilities already to reap the benefits of that know-how,” Michael shared. Nasdaq continued to function on AWS, embracing Redshift as their major information retailer, alongside quite a lot of AWS providers equivalent to S3 as their information lake, Glue for information integration, and QuickSight for enterprise intelligence.
This new stack additionally included Monte Carlo, adopted for information observability. “A number of our information monitoring on the time was very reactive. The pipeline may succeed however we wouldn’t know if the info was proper, fallacious, or detached,” Michael shared. “With a instrument like Monte Carlo, we’re in a position to catch a few of that earlier on.”
However the closing piece of the puzzle was Atlan. Atlan was adopted as Nasdaq’s window to their modernizing information stack, and as a vessel for a knowledge governance apply that was shortly maturing.
That is how we’re actually going to take ourselves to the subsequent degree.
Michael Weiss, Product Supervisor
Evolving Groups and Processes
Past taking a leap ahead in technical maturity, Nasdaq’s information staff understood that their staff construction and working mannequin would want to make an analogous leap in the event that they had been to fulfill, then exceed enterprise wants. “You may herald new instruments and to maintain the identical methodology and the identical cultural approaches you don’t actually remedy the core drawback,” Michael shared. “How do you convey a data-first tradition to the whole group?”
Nasdaq lengthy had a centralized construction for reporting; applicable for his or her legacy know-how and duties, however a mannequin that struggled to allow their enterprise counterparts to make use of new instruments and applied sciences.
“We appeared on the talent units inside totally different groups to attempt to determine the easiest way to convey forth change at a extra elementary organizational degree,” Michael defined.
This method led to rigorously focusing Nasdaq’s information group into extra particular roles. First is the Platform Workforce, chargeable for sustaining core know-how and lowering cognitive load on analysts and customers. Subsequent is the Financial Analysis staff, centered on information science and serving as a middle of experience for enterprise models conducting deeper evaluation.
Whereas a less-centralized mannequin was a greater match for the rising variety of information customers, it required a extra mature engagement and onboarding mannequin as customers started to use new capabilities and information.
“We began how we essentially embed information into every one in all our core companies to allow them to self-serve,” Michael shared. “So we got here up with an engagement mannequin the place we’d go to one in all our organizations. We do a survey to begin. We had been making an attempt to get an evaluation of the place they had been with their abilities, after which from there we might really embed with them a set of knowledge initiatives they needed to pursue and get them up and operating.”
This new course of, contemplating the distinctive wants of Nasdaq’s enterprise models, and guiding them via necessities, joint improvement of enterprise fashions, the creation of a metrics layer, and enablement to transition duty, meant that somewhat than appearing as a service middle, Nasdaq’s central information staff may give attention to bigger initiatives, teaching and consulting when vital.
We didn’t need to simply say, ‘Hey, right here’s the platform. Have enjoyable, good luck. Come again inside six months and tell us what you consider it.’ We discovered that this enabling actually has helped drive success getting groups on board.
Michael Weiss, Product Supervisor
Selecting Atlan for Energetic Metadata Administration
With core information know-how in place, a staff construction that matched Nasdaq’s information service technique, and an onboarding and engagement mannequin to make sure enterprise companions had been well-equipped to make use of new capabilities, Michael’s staff turned to energetic metadata administration as a path for training, collaboration, and discovery.
“This isn’t the primary time we’ve launched new know-how at Nasdaq for information. I’ve been at Nasdaq for 10 years. That is the third or fourth time we did it. Every time you be taught slightly one thing totally different,” Michael shared. “Once we launched this one of many key issues that our govt staff introduced up was ‘What will be totally different this time?’ ‘What different issues are you going to usher in the combination to scale back friction between information producers and customers?’”
Essential to Nasdaq’s govt staff was the flexibility for customers to grasp what information and instruments they’ve at their disposal, and energetic metadata administration grew to become a vital know-how for answering this query. With out a platform like Atlan, information and capabilities would stay undiscoverable, lowering the potential return on funding Michael had hoped for at the start of Nasdaq’s journey.
“To get key stakeholders to purchase into every thing, it was important we had a plan in place. We as soon as discovered ourselves centered on BI and ETL however forgot about the best way to scale back friction between producers and customers of knowledge. It was front-of-mind for all stakeholders that we deliberate to resolve this in any new information method. With out this the remainder of our mannequin can be DOA,” Michael defined.
Amongst Nasdaq’s stringent necessities for energetic metadata administration was a SaaS mannequin – avoiding the necessity to handle their very own infrastructure – in addition to the flexibility to rigorously mannequin entry factors throughout customers and roles, and a confirmed report of safety aligning with Nasdaq’s place as a extremely regulated enterprise entity.
Maybe most vital was cultural alignment, a deciding issue for why Atlan proved to be the very best match for Nasdaq.
It’s not simply the know-how, however the group behind it. We actually needed a associate who would are available in and assist us actually perceive the facility and advantage of this instrument. We discovered that.
Michael Weiss, Product Supervisor
Scoping the Knowledge Service Alternative
An vital a part of Nasdaq’s path to implementing Atlan was surveying information customers to raised perceive how they might enhance the invention and software of knowledge.
Almost 100 finish customers responded to the survey, and what Michael’s staff discovered was intriguing. 75% of respondents reported spending time making an attempt to grasp the context round information. And energy customers, who spend six or extra hours per week on information, spent two of these hours making an attempt to grasp the context round what they have already got entry to.
Take into consideration that! A 3rd of their time each week is spent simply making an attempt to grasp what’s there. Think about if we may convey a product in that helps scale back that effort and actually permits them to get proper to the guts of the issue — to drive information merchandise from insights into the enterprise. And that’s what we’re making an attempt to get to.
Michael Weiss, Product Supervisor
These findings had been constant throughout Nasdaq’s numerous information and capabilities. Whether or not uncooked information from their matching engine, or ready metrics and visuals, important time was spent on amassing context, slowing supply on key insights and information merchandise.
In search of a beyond-the-numbers understanding of stakeholder sentiment, the survey requested for qualitative suggestions, resulting in the belief that siloed and misplaced information had been key issues to handle. With out robust possession of knowledge belongings, or constant documentation, Michael’s stakeholders discovered it troublesome to select up the place one other member of Nasdaq’s group left off.
“It was not unusual that you’d have somebody working in a single a part of the group, understanding one a part of one thing. Then a knowledge product proprietor would are available in, and doesn’t perceive why choices had been made, and even what was obtainable, in order that they’re again to sq. one,” Michael defined.
Driving Widespread Understanding of Nasdaq’s Knowledge
“Among the early outcomes have been actually astonishing,” Michael shared. Regardless of being early of their Atlan adoption journey, a follow-up survey reveals indicators that Nasdaq’s alternative of Atlan for energetic metadata administration is yielding constructive outcomes.
“The one remark that retains developing is ‘That is like having Google for our information,’” Michael defined. “Letting them begin with column names and baseline descriptions has enabled our customers to actually perceive what’s in our lakes and warehouses already, after which begin asking the correct questions.”
Having made the correct know-how selections, and with a staff construction that helps their enterprise models distinctive information wants, the introduction of Atlan into the Nasdaq ecosystem has catalyzed a standard understanding of their information and the instruments at their disposal. And with a rising array of knowledge, insights, and capabilities at customers’ fingertips in Atlan, confidence in Michael and his staff’s information technique is enhancing.
“Despite the fact that we’re early into this journey there’s already a little bit of a shift within the sentiment of our staff. Whereas we nonetheless have quite a bit to do, this journey is already proving precious and has been important for the chief staff having confidence we are able to proceed to ship on our promise.”
