Possible to use defer to testing time in cloud IDE?

Help
1

I’m building a big new dbt project and developing in the cloud IDE. The models I’m developing depend on intermediate tables that take hours to run.

Is it possible to use defer or another feature of the dbt CLI to source data from the production version of tables from the cloud IDE but build the target models in my personal dev environment?

2

You can definitley use defer within the IDE, you’ll just need to have the relevant, up-to-date artifact to defer to. Here’s an example of a github action that’s triggered when code is merged to your main branch. It’s doing several things:

  • Running a dbt Cloud job. The job I have configured will run any modified models and anything downstream. You can also do something really simple here like dbt compile --exclude fqn:*, which will still generate the manifest.json file but not actually run any queries.
  • Retrieve the manifest.json file from the completed run
  • Save the manifest.json file into a directory in your project. Once there, you’ll now be able to defer to that artifact within the IDE.

Another option is using the upstream_prod package. This package overrides the built-in ref function and allows you to select from production objects when appropriate. No github action or deferral necessary here!

Hope that helps!

2 Likes
3

I did the following:
Commit a manifest.json file into states/prod/manifest.json
Then I tried to execute the following command in dbt Cloud IDE:

dbt run --select example.my_first_dbt_model --defer --state states/prod --favor-state

and it seems like defer and state ignored.

I there a way I can use dbt Cloud IDE and defer functionality?

1 Like
4

Please try using below command. I was able to successfully use --defer in dbt runin Cloud IDE. In my case, I created prod directory with manifest.json file

dbt run --select test_dep --defer --state prod