The problem I’m having
dbt opening multiple connections to target database where number of connections per user is restricted.
The context of why I’m trying to do this
We have restricted number of connections to target Postgres Database. Would like to strictly limit the number of connections dbt makes.
What I’ve already tried
Tested with different versions of dbt-bigquery as well as dbt-postgres adapters. It seems like even with threads=1 dbt is initating one master connection and another set of connections for each schema defined in the dbt_project.yaml. In our project we cannot use more two connections concurrently to target database. is there a way or configuration that we can use to ensure strictly the number of connections made to the target database. From logs we could see the connections are initiated like below
[0m16:42:27.610328 [debug] [MainThread]: Acquiring new bigquery connection 'master'
[0m16:42:27.671724 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_gdp-project-id'
[0m16:42:27.674011 [debug] [ThreadPool]: Opening a new connection, currently in state init
[0m16:42:32.247486 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_staging'
[0m16:42:32.249455 [debug] [ThreadPool]: Opening a new connection, currently in state init
[0m16:42:32.250455 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_pre_consumption'
[0m16:42:32.252455 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_consumption'
Some example code or error messages