The problem I’m having
is creating external tables using dbt_external_package with infer_schema option true generate DDL where I have column names as keywords in the parquet file. I will have enclose them in quote.
The context of why I’m trying to do this
It would be better If we create external tables from dbt to keep all of our pipeline objects in one place
What I’ve already tried
I tried creating external tables using below yml file syntax:
- name: homely
database: sample_db
schema: raw_schema
loader: S3
tables:-
name: agent1
description: “agent related details”
external:
location: “@AGENT__S3_STAGE/” ##reference an existing external stage
file_format: “my_parquet_format”
auto_refresh: false
infer_schema: true
partitions:- name: "year" data_type: varchar expression: (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 6), '=', 2)) - name: "month" data_type: varchar expression: (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 7), '=', 2))
-
Some example code or error messages
generate ddl:
SQL compilation error: syntax error line 9 at position 16 unexpected ‘order’. syntax error line 9 at position 33 unexpected ‘0’. syntax error line 9 at position 36 unexpected ‘as’.