Understanding DBT

[CharanND]

Hi, I want to know the cost benefits of using dbt as it is open-source.

Consider the below scenario. Let’s say I have a data in my ADLS and consider the below cases : Scenario - 1 : I pull the data using Azure ADF, perform transformations using the ADF activities and store the data in azure synapse tables. Scenario - 2: I pull the data using Azure ADF, perform transformations using dbt and store the data in synapse tables.

Here, I want to understand all the advantages while using dbt. Can you help me with respect to cost association here ? How can I leverage dbt here to minimize costs ?

[brunoszdl]

i am not familiar with azure pricing. But what dbt do is to compile your SQL code and send it to synapse to run the queries.

So you have to compare the pricing of adf activities and synapse computing, and see what makes more sense to your use case.

[CharanND]

Ok, can you please help if you are either familiar with snowflake or databricks. I’m Ok to proceed with any of these

[brunoszdl]

Snowflake and databricks will charge you for how long your clusters/warehouses are running. The cluster/warehouses will run the queries.

So it depends on your use case, if you need to be constantly running queries and keep them active all the time it will cost more, if you just need to run some queries e stop them, it will cost less. It also depends on the size of the cluster/warehouse, the region and the cloud provider.

They also have a storage cost.

But as I said it depends on your use case. Try to simulate the time you need the clusters/warehouses active, the size you will use, the region and the provider and compare them.

There are some calculators people made for this purpose

Snowflake:

Snowflake Cost Calculator | DataDrive

Cleartelligence | SNOWFLAKE COST CALCULATOR

Databricks
https://www.databricks.com/product/pricing/product-pricing/instance-types

[CharanND]

Thanks for the response, @brunoszdl. My use case is simple. For a beginner, who is just starting the journey with dbt, please refer to this topic which i’ve posted in another title. If you can please help with the below use case, please do that…

POC projects with DBT

Hi community, I am just getting started with dbt and learning it. I learnt dbt is an open-source ELT tool which takes care of Transformation part. I would like to demonstrate the same by developing the projects (one using dbt and the other without using dbt) which will prove that dbt is more cost efficient than any cloud services available. The primary objective of this project would be to showcase that using dbt would reduce my cost. Let it be in snowflake or Databricks, can you please help me…

[brunoszdl]

It is dangerous to make a pricing comparison if you are not familiar with the tools, if you make some wrong configurations you can end up spending a lot of money.

I would suggest you making a snowflake trial account, create your project there, see which resources you need to run your project, and then estimate the costs based on your usage. This is the safer way. The trial account is good for you to be used to tool as well.