Hi,
I have divided my models into separate folders motivated via How we structure our dbt projects 10
Unfortunately my recent runs are taking forever and not even throwing an error. The jobs are expected to be completed in seconds but go on forever around ~40mins.
I had found out the problem to be based with exclusive lock being set by dbt for redshift Prevent locks from blocking queries in Amazon Redshift, which means this should be a problem specific to redshift.
My staging outputs are materialized as views but internally looking at the logs (usually in the format dbt.log.1), it seems like they are being converted into tables where the ALTER statement is being used which causes the AccessExclusiveLock in redshift.
Is there a way to handle this issue?
Should dbt not be handling this by itself?
This issue does not occur of course if I use THREADS=1 but then there is not point using THREADS=1 forever right?
If it was a lock created between dbt and something else, it is understandable but apparently it is created within the same dbt run --models staging statement because it uses threads=4 by default and 2 threads lock on a table which is not an expected behaviour.