Issue passing a ref table name to a custom materialization

Archive
2

Hey @daniele.frigo - this is a super funky issue and it’s definitely confusing! The thing happening here is that dbt runs in two different modes:

  • parsing
  • running

During parsing, dbt is going to try to find all of the ref(), source(), and config() function calls in your model code in order to determine the shape of the graph and the configurations for your models.

The big problem is that at parse time, dbt has not necessarily walked your entire project to find models yet! This means that when you have some code like:

ref('OUT_AREAS_LIST')

dbt is going to have no idea:

  1. if this OUT_AREAS_LIST model exists
  2. what the configured identifier/schema/database is for the model
  3. what the materialization is (eg. is it a table/view, or is it ephemeral?)

As such, dbt is just going to return a junk/placeholder Relation object for refs called at parse-time (basically just this). The big issue is that dbt captures the values provided to config() at parse-time too, so these junk values are being saved as configs which will then be utilized at run-time :confused:

dbt doesn’t really have a good way to represent the thing you’re trying to do today, but it is a great idea, and I’d very much like to support it! I think we might want to re-process the config() call at model runtime, capturing the true config values and then passing them on to the materialization.

What do you think about something like that?

1 Like