The problem I’m having
When running the snapshot on my iceberg tables, I am getting the Error
ERROR: output size exceeded 2097152 bytes
Despite of the error, the snaphots are still being created. I believe it is referencing to the iceberg property write.parquet.dict-size-bytes which has default value of 2097152
The context of why I’m trying to do this
I am trying to create a snapshot tables for my slow changing dimension model.
I am using dbt-glue.
What I’ve already tried
I know that the iceberg property can be overridden but it doesn’t look like snapshot config block supports table_properties option.
Some example code or error messages
22:34:52 Glue adapter: status = error
22:34:52 Glue adapter: Glue returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.7.3", "profile_name": "dbt_glue_demo", "target_name": "dev", "node_id": "snapshot.dbt_glue_demo.events_snapshot"} */
select * from glue_catalog.dbt_metrics.events_snapshot
'''), ERROR: output size exceeded 2097152 bytes