Have you considered an incremental model? This is a perfect use case - you can put a check in for new data, something like
with upstream as (
select * from {{ ref('upstream_complex_table') }}
)
select
-- complex transformations here...
from upstream
{% if is_incremental() %}
where upstream.loaded_at > (select max(this.loaded_at) from {{ this }} as this)
{% endif %}
It is possible, but you need to set the default
property described above. When you do you’ll see this output:
$ dbt run
Running with dbt=1.4.1
[...]
Using default selector everything_except_the_monthly