dbt defer to production manifest

Help
1

Hello, I’m trying to set up my local development environment so that I don’t have to rebuild upstream models every time I need to make a change or create a new model that uses models that are already built in the production environment.

I just learned about the defer feature and it looks like it will solve all my problems, but I’m a noob at this so I want to make sure that I set up everything correctly and don’t mess up my data. :slight_smile:

Here’s how I run my project at the moment:

  • Daily job builds all models through dbt Cloud (free account) on the production environment
  • I use Snowflake with two databases - dbt_PROD and dbt_DEV
  • dbt_PROD runs from the master branch of the project repo on Github
  • I use dbt Core for local development on dbt_DEV database, and this is done through branches other than master, as I have disabled pushing new code directly to master branch
  • “target” folder is included in .gitignore, so manifest is the same on every branch

I would like to use the production manifest.json in the defer flag so that I don’t have to rebuild upstream models.

My question is - is it enough to just checkout to master branch and pull the latest data from the repo, and then run “dbt compile” before doing any development on other branches where I use defer flag? Will that ensure that I use the production manifest file for --state?

Is this the right way to do what I’m trying to do, or is there a better way to get the production manifest for deferring purposes?

Also, is there a way to make sure all dbt build/run commands use the defer method by default on the dev environment? Would that be a bad practice? I would like to make this easier for future collaborators on the project and avoid them having to add this flag every time.

Thanks in advance to the amazing dbt community! :slight_smile:

2

I tried testing this and it doesn’t work, it still builds upstream models. So I figured that it does “dbt compile” when I build the new model, so I tried copying the manifest into a separate folder and tried running “dbt build -s +my_model --defer --state prod_manifest” and it is still building the upstream models.

Even if I run “dbt compile --target prod” and fetch the production manifest, it still doesn’t work. :S

3

This is how I implemented state selection

  1. compile the project to a production folder (include manifest.json only)
  2. re-compile when pushed into master branch
  3. run with defer to production folder
  4. copy new manifest to production → commit (github actions)
  5. pull the changes

On pull-request
3. run and test with defer to prod manifest & target dev profile
4. merged if passed all the test

hope this help