Accessing run history for development jobs

Help
1

As a relative newcomer to dbt I am finding there’s a problem when I run jobs in my development environment in Cloud IDE, if I am not active in the session it times out and there is no access to the logs when I restart the IDE.

For Prod runs you have access to run history which is perfect, but I need to see this for development too.

Is there something I missed in the setup perhaps?

I’d appreciate some help with this.

2

Hey @piquet00! The Cloud IDE creates a development pod for you while you’re working, but once it times out (after 1 hour of inactivity I think?) the pod is torn down and so any logs it contains are discarded too, so you start each development session with a clean slate.

It’s possible that we’ll persist the logs in the future - I’ve added this as a note to the feature request.

3

Hi @joellabes

Thank you so much for your response. I understand why you might not want to persist development session logs given the sheer volume.

Is there a way to increase the timeout to keep the session alive? Most of our models take more than the timeout period to run so as things stand you more or less have to keep checking in to be sure of being able to see the logs at the end of the job.

Cheers,
Phil

4

Ahh I see! There isn’t a way to keep the session alive, but a couple of options:

  • My main recommendation: while developing, transform just a subset of data to avoid it taking an hour. We give some examples here
  • I believe that the dbt Cloud CLI does persist logs (and we’re planning to migrate the Cloud IDE to be backed by the same service one day, which is why I said we might persist logs in the IDE in future), so if you’re open to using VS Code and the Cloud CLI then AFAIK it will download logs after an invocation completes.

But really the main thing to aim for would be not having your transformations take an hour while you’re iterating, because that’s a pretty painful development loop! So either using the limit above or trying the new --sample flag once that’s built would be my first approach.

5

Hi @joellables

This is really helpful advice and makes a lot of sense - I do run models in isolation (no up and down-stream models) and use the preview option too so this is only really a problem when running a --full-refresh to test the whole lineage at the end.

I will check out the VScode / Cloud CLI approach and look out for the --sample flag which would definitely help in my use case.

Thank you so much,

Phil

1 Like