Bettering Developer Productiveness and Supply Velocity with Atlan
The Lively Metadata Pioneers collection options Atlan clients who’ve not too long ago accomplished a radical analysis of the Lively Metadata Administration market. Paying ahead what you’ve realized to the following knowledge chief is the true spirit of the Atlan neighborhood! In order that they’re right here to share their hard-earned perspective on an evolving market, what makes up their fashionable knowledge stack, modern use circumstances for metadata, and extra.
On this installment of the collection, we meet Abraham Tom, Director of Information at Generate Capital, who shares his 20-plus-year historical past in Information & Analytics, why Information Governance issues a lot to his group, and the way he and his staff use Atlan to enhance developer productiveness.
This interview has been edited for brevity and readability.
Would you thoughts introducing your self, and telling us the way you got here to work in Information & Analytics?
I’ve been a knowledge skilled for over 20 years, constructing out Netezza, Oracle and SQL server warehouses, shifting on to Hadoop, and now Cloud methods. I’ve labored on all elements of knowledge, starting from motion, governance, engineering, science (earlier than it was horny), BI analytics, graphical illustration, and specializing in ground-up implementations of recent knowledge warehouse initiatives. My motivation is displaying to executives why data-driven resolution making is efficacious, then constructing a roadmap to attain that state.
May you describe Generate Capital, and any vital initiatives you and your staff are engaged on?
Generate Capital PBC is within the enterprise of re-building the world to save lots of humanity. We construct, personal, function, and finance reasonably priced and dependable infrastructure options for clear power, water, waste, transportation, and good cities’ infrastructure applied sciences. Information Governance must be the cornerstone of our knowledge warehouse program so that everybody that must be knowledgeable or that should decide has the proper info to take action.
What does your stack appear like, and the way has that advanced over time?
I began because the second tech particular person within the group. Generate Capital has the distinctive problem the place we get knowledge from completely different strains of enterprise and coalescing all that knowledge into an mixture view. This required shortly devising an answer that may scale, and implementing a set of instruments that allowed for that. Our major repository was Snowflake as a cheap scalable platform, we use Streamsets as our major knowledge ingestion, dbt as our transformation engine, and Tableau as our major visualization and supply expertise. We additionally deal with a whole lot of the info integration between our ERP and CRM methods, in addition to being the hub for knowledge transport to different methods.
How did you come to search for an Lively Metadata Administration resolution? Why did Atlan stand out?
Initially, I needed to discover a easy knowledge dictionary software. Whereas dbt has lineage capabilities and a few documentation constructs, it was restricted to what dbt is conscious of, and never all of the methods we use. Atlan is a good resolution to not solely to accommodate our dictionary, however the lineage and glossary capabilities have elevated our staff’s information in order that they will see the impression of their new improvement.
What have you ever constructed with Atlan, to this point? What worth have you ever been in a position to drive?
Proper now, we’re utilizing Atlan as extra of a tech useful resource than a enterprise useful resource, and have discovered good worth.
Our product staff has grow to be our first non-developer customers of the Atlan platform which has improved general ticket and specification writing. Tickets, characteristic requests, and onboarding have all vastly improved. Our Product Managers can do analysis on how the info flows and may write very particular requests with higher acceptance standards, which has resulted in improved developer effectivity, much less back-and-forth with questions, and improved general supply, all of which elevated our velocity.
Now we have additionally built-in Atlan’s impression evaluation with our dbt pull requests as a GitHub merge dependency test. This offers our builders consciousness of the downstream impacts of the modifications they plan to merge earlier than the ultimate commit occurs. This has helped us guarantee any new characteristic requests don’t adversely have an effect on current manufacturing gadgets. Normally, this requires senior sources to evaluate code, and whereas we nonetheless require peer code evaluate, that evaluate is lots quicker, and everyone seems to be knowledgeable of the potential upstream change and impression.
As a part of our Governance initiative, we’re additionally beginning to leverage Atlan’s glossary operate as a seize mechanism for the enterprise definitions. Atlan assists us in tying that enterprise definition to the system dictionary.
Any remaining ideas?
Atlan is a good software for builders to see knowledge lineage and to assist them work out what must be finished and the way their work impacts downstream methods. All of my builders repeatedly use this to know what they’re constructing. It additionally supplies different improvement groups transparency of the work we do and the place their work finally ends up.
Photograph by Bastian Pudill on Unsplash
