The problem I’m having
In a macro i have - i want to get some values from model config of a ref model i use in other target_model.sql
I want to build generic dbt macros that will automatically backfill missing data depending on the model config variables. For this - i set the configuration for each model in the models.yml files
For example:
- name: L0__game_runs
config:
materialized: incremental
incremental_strategy: append
time_interval: hour
time_duration: 1
elementary:
timestamp_column: time_started
- name: L1__drafts
config:
materialized: incremental
incremental_strategy: merge
time_interval: hour
time_duration: 2
elementary:
timestamp_column: created_at
Say my source is L0__game_runs
and the target (=“this”) is L1__drafts
I want to get the source timestamp_column but i don’t find a proper way to do it.
my macro:
Code
L1__drafts .sql:
with
{% if is_incremental() %}
{% set self_max_timestamp = get_table_max_timestamp(this) %}
drafts as {{ (generate_source_query(ref('L0__game_runs'), self_max_timestamp)) }}
{% endif %}
drafts as (select * from {{ ref('L0__game_runs') }}),
...........
My macro:
{% macro generate_source_query(source_model, self_max_timestamp) %}
{%- set source_timestamp_column = source_model.get('elementary').timestamp_column -%}
{% set source_query %}
SELECT *
FROM {{ source_model }}
WHERE DATE('{{ source_created_at }}') >= DATE_ADD(DATE('{{ self_max_timestamp }}'))
{% endset %}
{% endmacro %}
The problem:
source_timestamp_column
is not correct and just returns blank. I didn’t find a way to retrieve it.
The question:
How can i do it?
I hope i was clear.
Thank you !