dbt is a DWA (Data Warehouse Automation) market player?

Hello everybody.

As for now I have been working with DBT for several months on MS SQL Server.

First of all, have to say it is absolutely amazing, just to name a few areas:

  • SQL templates on Jinja - beautiful and promising. Saw a lot of implementations of templates for SQL: SQL + customs scripts, SQL + SQL, SQL + C# etc. - always had a feeling that it does not fit well to each other. Jinja just looks like it always has been here
  • Graph tasks (DAG) management - is what we always needed
  • Environments management/CI/CD/DevOps - the ideas of DevOps in analytics started to crystallize finally into something mature
  • Integrated Data Catalog - when Gartner marked data catalog as an obsolete technology Gartner Hype Cycle for Data Management Positions Three Technologies in the Innovation Trigger Phase in 2018 it provided a lot of confusion about the future of data catalogs at all. Now it is just exciting to see working elements of Data Catalog maturity level 4 by Deloitte https://www2.deloitte.com/content/dam/Deloitte/de/Documents/deloitte-analytics/Deloitte_Analytics_CatalogingData-Whitepaper.pdf
  • Open Source (and the answer to that confusing question “how to get it deployed?” - “pip install”) is a huge adoption catalyst
  • And all that beauty is integrated of course together with logging, snapshots etc

The question is why DBT is standing somehow apart from DWA (Data Warehouse automation) market although it solves a lot of tasks in the area?

For example, I could not find any mention of “DWA” in the forum. And digging through the DWA market for years it was quite hard to figure out that DBT is actually doing a big part of the same job.

A few links from the DWA market:

https://en.m.wikipedia.org/wiki/Data_warehouse_automation

DWA comparison tool

https://www.datawarehouseautomation.guide/dwa-tools/

Most famous DWA market player

https://www.wherescape.com/

DWA solution Integrated with Microsoft Visual Studio:

https://www.dimodelo.com/ (it was one of my first attempts to use DWA tool in end-to end solution http://sergeyavdovin.com/contents/resume/AdventureWorksDWHandReportingSolutionHROverview.pdf )

Microsoft about DWA market

https://azure.microsoft.com/en-us/blog/accelerate-your-cloud-data-warehouse-with-automation-tools/

And sure, I do agree that the community is a big part of the DBT adoption. Worth to mention that my own adoption of DBT started when I found PSA (persistent staging) template discussion on the forum https://discourse.getdbt.com/t/implementing-a-persistent-staging-area-with-a-dynamic-incremental-model/252.
So, I hope this post will help to grow the community a bit by extending it to the bigger DWA area.

Cheers
Sergey

1 Like

Hey Sergey, Excellent post (sorry I am 2 years late :grinning:)

As a former WS employee, couldn’t agree more. Infact, we are getting plenty of inquiries from our NZ based clients as they get ready to adopt cloud and SaaS model to data warehouse and data transformation. dbt definately is the evolutionary next step.

A bit of self promotion: we have built some materilizations that deliver the same functionality as Stage Tables, Datastores and Type 2 Dimensions and Facts/ other 3NF tables that can be found in WS.

happy to talk more to the community, if there’s an interest

Peeyush

1 Like