Hello, I’m developing in dbt using glue and iceberg.
However, I’ve noticed that when there’s an error in some part of the code and it creates the table with the prefix tmp_.
The next time I run it, I always get an error saying that the table already exists. Looking at the source code, I noticed that in the file .\Lib\site-packages\dbt\adapters\glue\impl.py
In the code part:
if session.credentials.glue_version == "4.0": head_code += f'''outputDf.createOrReplaceTempView("tmp_tmp_{target_relation.name}")
spark.sql("CREATE TABLE tmp_{target_relation.name} LOCATION '{session.credentials.location}/{target_relation.schema}/tmp_{target_relation .name}' AS SELECT * FROM tmp_tmp_{target_relation.name}") '''
The command is missing before the deletion below:
*spark.sql("DROP TABLE IF EXISTS tmp_{target_relation.name}")*
So this way we won’t have a problem in the next execution.
It would look like this:
if session.credentials.glue_version == "4.0": head_code += f'''outputDf.createOrReplaceTempView("tmp_tmp_{target_relation.name}")
**spark.sql("DROP TABLE IF EXISTS tmp_{target_relation.name}")**
spark.sql("CREATE TABLE tmp_{target_relation.name} LOCATION '{session. credentials.location}/{target_relation.schema}/tmp_{target_relation.name}' AS SELECT * FROM tmp_tmp_{target_relation.name}") ''' I use version 1.8.1