Incremental model runs only like "create or replace table..."

Incremental model runs only like “create or replace table…”

Hi all, I have some incremental model in dbt-databricks with mehod merge. If I run “dbt run --full-refresh --select <table_name>”, then is everything OK. But when I run it like “dbt run --select <table_name>”, there is no difference - it is run like full-refresh…

The main peculiarity is that my colleague, who uses everything the same way, it works for him, the problem is only with me.

Here is my code…

{{  config(
        materialized = 'incremental',
        incremental_strategy = 'merge',
        unique_key = ['server_name','stock_date','hash_guid']
        ) }}  

        , MAX(stock_date) AS max_date
        {{ ref('table_name') }} 
        stock_date >= '{{full_refresh_date}}'
        AND stock_date <= '{{last_stage_date}}'
    {% if is_incremental() %} 
        AND stock_date >= DATEADD(day, -31, '{{last_stage_date}}')
    {% endif %}

Terminal output looks OK:
11:12:08 1 of 1 START sql incremental model schema.table_name [RUN]

But in Databricks it is runing like “create or replace…”

Many thanks for any reply!!


Not entirely sure here but wondering if it is a parsing issue - is it possible it is caching something with regards to the command ? Also - does your table already exist when you run it without the full-refresh switch ?


thanks for your reply.

The main issue is that the model starts without merging, even if I run it without full-refresh.

The procedure for creating a table is that I run it for the first time in full-refresh mode, then the table remains - it is not deleted. On the second run, I need to start WITHOUT full-refresh, only merge, but I always get dbt run as “crate or replace table…”

It should be like this, but in my case “dbt_INTERNAL_SOURCE” like …__dbt_tmp does not appear:

That is a bit odd for sure - the only other thing I can think of since it works for your coworker would be to compare project.yml and make sure there is not any delta in there that would affect this table - it should not since you are explicitly setting it in config in the model which should override the project settings but it might be worth a look just to confirm you are both in alignment. There is a part of me that still feel like maybe that initial run is being cached somewhere in the manifest and just being re-executed on subsequent runs.

Thank you very much for the suggestions. We will try to find out in the team.


Finally, creating a new schema in the unity catalog helped.

Thx, PR.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.