Suggestion for dbt glue - improvement

Hello, I’m developing in dbt using glue and iceberg.

However, I’ve noticed that when there’s an error in some part of the code and it creates the table with the prefix tmp_.

The next time I run it, I always get an error saying that the table already exists. Looking at the source code, I noticed that in the file .\Lib\site-packages\dbt\adapters\glue\impl.py
In the code part:

if session.credentials.glue_version == "4.0": head_code += f'''outputDf.createOrReplaceTempView("tmp_tmp_{target_relation.name}") 
spark.sql("CREATE TABLE tmp_{target_relation.name} LOCATION '{session.credentials.location}/{target_relation.schema}/tmp_{target_relation .name}' AS SELECT * FROM tmp_tmp_{target_relation.name}") '''  

The command is missing before the deletion below:
*spark.sql("DROP TABLE IF EXISTS tmp_{target_relation.name}")*
So this way we won’t have a problem in the next execution.

It would look like this:

if session.credentials.glue_version == "4.0": head_code += f'''outputDf.createOrReplaceTempView("tmp_tmp_{target_relation.name}") 
**spark.sql("DROP TABLE IF EXISTS tmp_{target_relation.name}")** 
spark.sql("CREATE TABLE tmp_{target_relation.name} LOCATION '{session. credentials.location}/{target_relation.schema}/tmp_{target_relation.name}' AS SELECT * FROM tmp_tmp_{target_relation.name}") ''' I use version 1.8.1
1 Like