I’m not able to read data from MySQL and write them to a BigQuery DB
I’m stuck at the very beginning because seems that the 2 connetors dbt-bigQuery and dbt-mysql depend on different versions of dbt-code
dbt-bigquery 1.4.3 requires dbt-core~=1.4.0, but you have dbt-core 1.1.4 which is incompatible.
dbt-core 1.1.4 is installed by dbt-mysql
can someone suggests a way to make dbt to work together with BigQ and MySQL ?
dbt is not a data movement tool (the Extract/Load in ELT), it is only for transforming data once it lands in a warehouse.
To move data from one warehouse to another, you’ll want to look into tools like Fivetran, Stitch, Meltano, Airbyte etc.
thanks, I’m aware of the scope of dbt, and indeed we’re using it on BigQ for data trasnformation. It happens now that one raw datasource is not on BigQ itself, but rather on a different db (MySQL) and seem it should be possible to just read from MySQL and write to BigQ.
Do you say that is not possibile at all or that dbt is just not the best tool for that ?
dbt can only do things that your warehouse can already do in SQL. So you can do something like unload a CSV to a GS bucket, but unless there is some native way to connect to MySQL from BQ then no it can’t do it at all.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.