all subsequent dbt-core invoke calls from python fail after one failure

Help
1

The problem I’m having

Calling dbt from python using dbt-core causes all subsequent calls to fail thinking the database has been invalidated, despite calling via subprocess.run working fine.

The context of why I’m trying to do this

Working on a project that calls dbt many times via subprocess.run to insert into different sub schemas in duckdb. Trying to be able to analyze failures more easily, I tried to replace the subprocess.run with calls to dbt-core’s dbtRunner(). However, as soon as one call fails, all subsequent calls fail, even when re-instantiating dbtRunner with the error:

“Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.”

Re-running the subsequently failing calls work, so it’s not actually that the DB is in a bad state.

Some example code or error messages

for vars in varList:
        sp_dbt_run = dbt.invoke(
            [
                "run",
                "--select",
                "models1",
                "models2",
                "--vars",
                json.dumps(jinja_vars),
            ]
        )
    # process success/errors
2

After lots of experimentation, I discovered that it may have something to do with using multiple threads. If I run with only 1 thread instead of 8, the first run fails but it doesn’t cause issues with the second run.

3

Hello,
Thanks for asking.