Hello everybody.
As for now I have been working with DBT for several months on MS SQL Server.
First of all, have to say it is absolutely amazing, just to name a few areas:
- SQL templates on Jinja - beautiful and promising. Saw a lot of implementations of templates for SQL: SQL + customs scripts, SQL + SQL, SQL + C# etc. - always had a feeling that it does not fit well to each other. Jinja just looks like it always has been here
- Graph tasks (DAG) management - is what we always needed
- Environments management/CI/CD/DevOps - the ideas of DevOps in analytics started to crystallize finally into something mature
- Integrated Data Catalog - when Gartner marked data catalog as an obsolete technology Gartner Hype Cycle for Data Management Positions Three Technologies in the Innovation Trigger Phase in 2018 it provided a lot of confusion about the future of data catalogs at all. Now it is just exciting to see working elements of Data Catalog maturity level 4 by Deloitte
https://www2.deloitte.com/content/dam/Deloitte/de/Documents/deloitte-analytics/Deloitte_Analytics_CatalogingData-Whitepaper.pdf
- Open Source (and the answer to that confusing question “how to get it deployed?” - “pip install”) is a huge adoption catalyst
- And all that beauty is integrated of course together with logging, snapshots etc
The question is why DBT is standing somehow apart from DWA (Data Warehouse automation) market although it solves a lot of tasks in the area?
For example, I could not find any mention of “DWA” in the forum. And digging through the DWA market for years it was quite hard to figure out that DBT is actually doing a big part of the same job.
A few links from the DWA market:
https://en.m.wikipedia.org/wiki/Data_warehouse_automation
DWA comparison tool
https://www.datawarehouseautomation.guide/dwa-tools/
Most famous DWA market player
https://www.wherescape.com/
DWA solution Integrated with Microsoft Visual Studio:
https://www.dimodelo.com/
(it was one of my first attempts to use DWA tool in end-to end solution http://sergeyavdovin.com/contents/resume/AdventureWorksDWHandReportingSolutionHROverview.pdf
)
Microsoft about DWA market
https://azure.microsoft.com/en-us/blog/accelerate-your-cloud-data-warehouse-with-automation-tools/
And sure, I do agree that the community is a big part of the DBT adoption. Worth to mention that my own adoption of DBT started when I found PSA (persistent staging) template discussion on the forum https://discourse.getdbt.com/t/implementing-a-persistent-staging-area-with-a-dynamic-incremental-model/252
.
So, I hope this post will help to grow the community a bit by extending it to the bigger DWA area.
Cheers
Sergey