all subsequent dbt-core invoke calls from python fail after one failure

The problem I’m having

Calling dbt from python using dbt-core causes all subsequent calls to fail thinking the database has been invalidated, despite calling via working fine.

The context of why I’m trying to do this

Working on a project that calls dbt many times via to insert into different sub schemas in duckdb. Trying to be able to analyze failures more easily, I tried to replace the with calls to dbt-core’s dbtRunner(). However, as soon as one call fails, all subsequent calls fail, even when re-instantiating dbtRunner with the error:

“Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.”

Re-running the subsequently failing calls work, so it’s not actually that the DB is in a bad state.

Some example code or error messages

for vars in varList:
        sp_dbt_run = dbt.invoke(
    # process success/errors

After lots of experimentation, I discovered that it may have something to do with using multiple threads. If I run with only 1 thread instead of 8, the first run fails but it doesn’t cause issues with the second run.

Thanks for asking.