The problem I’m having
We have files in Raw Json format in s3. I used a crawler to generate a table in Glue Database. When trying to create a new table from the Json table dbt is throwing error that column does not exist. But the column can be found in the schema in Glue Table.
Some example code or error messages
AnalysisException: Column 'metadata' does not exist. Did you mean one of the following? [spark_catalog.dbt_db.crawler_4127_indb_ahl_test_topic_flexi_indb_gonogo_test_gonogocustomerapplication.date, spark_catalog.dbt_db.crawler_4127_indb_ahl_test_topic_flexi_indb_gonogo_test_gonogocustomerapplication.hour];
The hour and date columns are partitions and it seems dbt is only recognising these two columns.