Incremental model runs only like “create or replace table…”
Hi all, I have some incremental model in dbt-databricks with mehod merge. If I run “dbt run --full-refresh --select <table_name>”, then is everything OK. But when I run it like “dbt run --select <table_name>”, there is no difference - it is run like full-refresh…
The main peculiarity is that my colleague, who uses everything the same way, it works for him, the problem is only with me.
Here is my code…
{{ config(
materialized = 'incremental',
incremental_strategy = 'merge',
unique_key = ['server_name','stock_date','hash_guid']
) }}
SELECT
hash_guid
, MAX(stock_date) AS max_date
FROM
{{ ref('table_name') }}
WHERE
stock_date >= '{{full_refresh_date}}'
AND stock_date <= '{{last_stage_date}}'
{% if is_incremental() %}
AND stock_date >= DATEADD(day, -31, '{{last_stage_date}}')
{% endif %}
Terminal output looks OK:
11:12:08 1 of 1 START sql incremental model schema.table_name [RUN]
But in Databricks it is runing like “create or replace…”
Not entirely sure here but wondering if it is a parsing issue - is it possible it is caching something with regards to the command ? Also - does your table already exist when you run it without the full-refresh switch ?
The main issue is that the model starts without merging, even if I run it without full-refresh.
The procedure for creating a table is that I run it for the first time in full-refresh mode, then the table remains - it is not deleted. On the second run, I need to start WITHOUT full-refresh, only merge, but I always get dbt run as “crate or replace table…”
It should be like this, but in my case “dbt_INTERNAL_SOURCE” like …__dbt_tmp does not appear:
That is a bit odd for sure - the only other thing I can think of since it works for your coworker would be to compare project.yml and make sure there is not any delta in there that would affect this table - it should not since you are explicitly setting it in config in the model which should override the project settings but it might be worth a look just to confirm you are both in alignment. There is a part of me that still feel like maybe that initial run is being cached somewhere in the manifest and just being re-executed on subsequent runs.
