The problem I’m having
We are dealing with some inexplicable errors in our main job. The error always says “404 Not found: Dataset datawarehouse-xxxxxx:_dev_dataproducts_mf; reason: notFound, message: Not found: Dataset datawarehouse-xxxxxx:_dev_dataproducts_mf”,
and each time it happens it’s on completely different models.
This BigQuery dataset “_dev_dataproducts_mf” is not referenced at all in any of our dbt models, tests, snapshots, macros or anywhere else in our repository. Additionally, we went through the debug logs of the affected runs and the dataset name also does not appear anywhere other than the error messages themselves.
We have already deprecated a job before because this error kept happening in it, but now it’s also happening in the new main job. There doesn’t seem to be any action we can take to address it on our side given that the repository does not reference the problematic dataset at all, as I mentioned.
The context of why I’m trying to do this
We are executing our main transformation using dbt cloud and executing a bunch of models and snapshots.
What I’ve already tried
Only thing that solved it was creating a new job but we don’t want to do this as we then need to do some changes.
Some example code or error messages
12:15:20 Runtime Error in model <model_name>
(.../.../.../.../.../.../<model_name> .sql)
404 Not found: Dataset datawarehouse-xxxxxx:_dev_dataproducts_mf; reason: notFound, message: Not found: Dataset datawarehouse-xxxxxx:_dev_dataproducts_mf
Location: EU
Job ID: <uuid>