When we should select dbt over ETL tools such as Talend?
Hello Saurabh!
These questions can be opinionated, however I’ve used both tools over the years.
DBT is perfect when your pipeline is ELT with the (T)ransforms are written in SQL and executed on warehouse (Snowflake, Redshift, BigQuery etc) or a “lakehouse” (Databricks, Athena etc).
Talend may be useful for the (E)xtract and the (L)oad, but these days less and less so, unless you have some obscure, legacy, on premise applications that you need to extract from. But in those cases you’ll probably have to write a lot of custom code anyway which defeats the purpose of a GUI ETL tool like Talend.
So in summary for the (E) and the (L) maybe Talend, but there are better options these days. Most can do far better with services like Fivetran or Stitch.
For the (T) definitely DBT, it’s not even a contest.
1 Like
Hi Saurabhb,
I have used both Talend and DBT and I would say that there is no contest.
And I also have a few Talend answers in Stackoverflow, so I know what I am talking about.
Like Krishna said the only thing I would use Talend for is to move the data -unchanged- to the data lake / cloud platform.
Then just use DBT to do the transformations, there is no contest there in quality of development (think tests, how the hell you do them in Telend?), clarity and maintainability of the result.
With DBT you can use a proper SW Engineering way of working, with Talend not.
In Talend basic things like reusing logic are not trivial at all and you can not even clearly understand what is the difference between two versions, let alone making code reviews or pull requests.
Box based tools are great for watered down problems that you see in demos with well prepared inputs, but are quite a pain to use in real life, when you do, then change, then change again… before coming to the final solution.
Ciao, Roberto