Leaked Semaphore Objects causing dag to never finish

Help
1

The problem I’m having

We’ve connected our dbt cloud instance to redshift. We have 1 large run that used to take about 5 hours, but it recently started reaching 12 hours before it was canceled by the max run time rules. All but 1 task will have finished (154/155). The logs show this at the end. (I’ve changed the model names to be generic.)

08:30:08 155 of 155 OK created sql table model datavault.model_155 .. [SELECT in 1884.13s]
13:14:08  CANCEL query model.datavault.model_a ................... [CANCEL]
13:14:08  CANCEL query snapshot.datavault.model_b .................... [CANCEL]
13:14:08  CANCEL query model.datavault.model_c ..................... [CANCEL]
/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Why isn’t the dbt run finishing anymore?
Is the error a red herring? Is redshift not marking some queries as done?

2

I have not. Are you trying to resolve an issue related to it? I recall it being a python bug and upgrading seems to be the fix

Note: @Charlie Hahn originally posted this reply in Slack. It might not have transferred perfectly.

3

Note: @Charlie Hahn originally posted this reply in Slack. It might not have transferred perfectly.

4

Upgrade python or dbt?

Note: @pauldrabinski originally posted this reply in Slack. It might not have transferred perfectly.

5

Charlie Hahn

Upgrade python. Lemme find the related thread
Slack
Not sure if this is related but hope it helps!

(Manually cross-posting because the integration breaks when the discussion moves entirely to Slack)