Pros & cons of using dbt incremental as an ingestion tool from Cloud SQL

In-Depth Discussions
1

hi dbt community,

a question on using dbt as data ingestion tool (with the incremental model)

i am trying to find out the pros & cons using 1) dbt incremental model to bring data from or production database under GCP cloud SQL into BigQuery vs 2) using GCP datastream to bring data directly from the production database into BigQuery

if the production database is hosted under GCP cloud SQL, BigQuery can query the production db directly via the ‘EXTERNAL_QUERY()’ feature. as such, one in theory can build an incremental model in dbt with syntax 'select * from external_query(, ‘select * from production db table name’) to bring incremental changes from the prod db.

what is the benefit and disadvantage over using GCP datastream?

2 Likes