I am fairly new to dbt
I have a dbt model that query a staging table (where the incremental works).
When I run the model without incremental it takes only a few seconds to finish and query 90K rows.
When I add the incremental part ( I added it below) it never stop running (more than 10 minutes ) as it is looking for the newest rows in the staging table.
The staging table comes from a partitionned external table from GCS.
{{
config(
materialized='incremental'
)
}}
WITH parsed_json AS (
SELECT
payload as json_data,
`date`
FROM {{ ref("stg_table") }}
),
incremental_load AS (
SELECT
p.json_data,
p.`date`
FROM parsed_json p
{% if is_incremental() %}
WHERE p.`date` > (SELECT MAX(`date`) FROM {{ this }})
{% endif %}
)
--SELECT QUERIES FROM incremental_load
Have you ever had this issue ?
Thanks in advance !