I’m building a big new dbt project and developing in the cloud IDE. The models I’m developing depend on intermediate tables that take hours to run.
Is it possible to use
defer or another feature of the dbt CLI to source data from the production version of tables from the cloud IDE but build the target models in my personal dev environment?
You can definitley use
defer within the IDE, you’ll just need to have the relevant, up-to-date artifact to defer to. Here’s an example of a github action that’s triggered when code is merged to your main branch. It’s doing several things:
- Running a dbt Cloud job. The job I have configured will run any modified models and anything downstream. You can also do something really simple here like
dbt compile --exclude fqn:*, which will still generate the
manifest.json file but not actually run any queries.
- Retrieve the
manifest.json file from the completed run
- Save the
manifest.json file into a directory in your project. Once there, you’ll now be able to defer to that artifact within the IDE.
Another option is using the upstream_prod package. This package overrides the built-in
ref function and allows you to select from production objects when appropriate. No github action or deferral necessary here!
Hope that helps!