I am running Cloud DBT (version 1.3) with PG14 and trying to understand how DBT handles table materialization without any down time. The introduction of grant support in the configuration with version 1.2 has gotten ride of any permission issues that happened during the life of a job due to hooks.
Furthermore, as mentioned in several threads, DBT materializes tables in such a way that the original table is accessible while running the computation and is swapped with the temporary file once it has been written.
From these two I would assume that I should be able to access my tables at all time, even when the hourly job is running. However, in practice whenever I run a job the associated tables become inaccessible.
It seems I have either misunderstood how DBT handles this process or I am unaware of an interaction that I might be causing.
Thank you for your time!
In debug log you can see the tmp table as they are created … you could use that info and try to trouble shoot … in general you are doing something risky … look for blue green deployment strategies
Thanks for the feedback, I will look into blue green deployment. Could you elaborate in what regard I am doing something risky ? Are you referring to not using blue green deployment ? Specifically within DBT I am not doing anything out of the ordinary
user can query table that are under refresh - the risky bit is that
with B/G you do atomic switch and avoid that possibility Performing a blue/green deploy of your dbt project on Snowflake
Thank you for the link. It seems to me that the way DBT handled the replacement of tables to be similar in its procedure as blue/green. I understand that there are no tests and validations but the swapping should be the same, no ? If that is the case then either I should see the same performance as green/blue and have no downtime or green/blue will have the current issues with inaccessible tables every hour.