Using dbt core 0.25 and Bigquery,
when creating a new model/table (B) I can’t reference (ref()) a table (A) from a different dataset
even if they are on the same database (gcp project) and the same dbt project (in an other subfolder).
SELECT
column
FROM
{{ ref('model_A') }}
is not working
note: if i use {{ source(‘dataset_name’, ‘table_name’) }} it works but ref() should be used instead since the first model (A) i want to reference is already using the source.
|S(other_gcp_project)|–>|A(in_dataset_A)–>B(in_dataset_B)|_same_project
I am using the command to run model_B referencing model_A
dbt run --full-refresh --select 'model_B' --target name_of_connexion_B
having the name_of_connexion_B in
profiles.yaml
name_of_profile:
outputs:
name_of_connexion_B:
type: bigquery
method: oauth
project: "name_of_the_same_gcp_project"
dataset: the_dataset_destination_B_named_in_dbt_project_yaml
location: EU
threads: 8
...
name_of_connexion_A: (used when dbt run the model_A)
type: bigquery
method: oauth
project: "name_of_the_same_gcp_project"
dataset: the_dataset_destination_A_named_in_dbt_project_yaml
location: EU
threads: 8
and having in
dbt_project.yml
profile: 'name_of_profile'
...
models:
name_of_the_dbt_project:
subfolder_A_under_model_folder:
+database: "name_of_the_same_gcp_project"
+materialized: table
subsubfolder_A_containing_model_A:
+schema: suffix_destination_dataset_A (prefixed by subsubfolder_A)
subfolder_B_under_model_folder_containing_model_B:
+materialized: table
+database: "name_of_the_same_gcp_project"
and got
the error
Runtime Error in model model_B
(path_of_model_B)
404 Not found:
Dataset database: subfolder_B_suffix_destination_dataset_A
was not found in location EU
detailed noticed in error:
the ref() seems to search the wrong dataset, and badly build the dataset name by concatenating the ‘subfolder_B’ (found for the in dbt_project.yml) used in profiles (used by –target) with the suffix_destination_dataset_A (found also in dbt_project.yml)
Maybe i am missing something, can you help me to make ref() work on this scenario ?
Thanks a lot to the dbt community for the help you can provide to help me on this