In my python model I am using session.sql(my_query).
I can’t use dbt.ref because that will load the entire upstream dataset at once, which is too big for my memory limitations. Instead, I loop over segments of this data set by updating my_query with a dynamic where clause. (If anyone has a nicer suggestion for this, let me know). This all works well, but because I use session.sql instead of dbt.ref, the upstream view I use isn’t added to the lineage. So when I call dbt build -s +this_model, the upstream view won’t run.
Now, for sql models you can add the ref to a comment, like
– depends_on: {{ ref(‘upstream_parent_model’) }} - from About ref function | dbt Developer Hub
So my question is: is there an equivalent for python?
Thanks!