Running dbt unit tests in CI/CD

Help
1

The problem I’m having

I’m trying to run the new unit tests feature as part of my CI/CD pipeline, as you would do in software engineering. However, the tests are failing with the error

Not able to get columns for unit test 'projects' from relation "dwh"."marts"."projects"

Clearly the test database that is spun up inside the CI/CD pipeline has no knowledge of how the model’s upstream sources are structured.

The context of why I’m trying to do this

I was hoping that the unit tests feature would allow me to introduce more software engineering best practices into the team, and we would only merge thoroughly tested models. Maybe unit tests are not 1-to-1 comparable between the software and data engineering paradigms

What I’ve already tried

I could work around it by using the sql format for all data sources, but that makes it much more difficult to maintain. I guess I could try to download the manifest.json from the main branch, but this would have the issue that my PR has obviously changed the models, so the manifest.json would risk being out of date.

Has anyone run into this issue? Or does anyone have an idea on the best practices for setting up unit tests / CI/CD in dbt?

2

I believe you are using different profile (target) in CI/CD vs Production.
Option1
You need to have to build model that unit test depending on in CICD as well before running unit test.
Option2
Using --defer when running unit test so dbt will use production model instead