# Running two dbt Cloud jobs back to back

**URL:** https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017
**Category:** Help
**Tags:** orchestration-and-deployment, dbt-cloud
**Created:** [October 6, 2022, 9:25am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017 "2022-10-06T09:25:32Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![obar1](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/obar1/32/1221_2.png) [@obar1](https://discourse.getdbt.com/u/obar1)
#### Post date: [October 6, 2022, 9:25am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/1 "2022-10-06T09:25:32Z")

</div>

I have job1 and job2  
I need job1 to be scheduled and it takes 3H to finish  
after job1 completes I want job2 to be executed and it take 2H

if I just schedule them and they run in parallel I have problem on TABLE looking

how can I do that in dbt cloud without external scheduler?

job1 → job2 → job1 etc for ever

---

<div class="post-metadata">

### Author: ![joellabes](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/joellabes/32/3980_2.png) [@joellabes](https://discourse.getdbt.com/u/joellabes)
#### Post date: [October 6, 2022, 6:49pm UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/2 "2022-10-06T18:49:25Z")

</div>

Hey @obar1, dbt cloud jobs don’t have dependencies between one another by default. We rely on the dependency graph that dbt builds for itself internally to make sure things run in the correct order and without conflict.

Why do they need to be two jobs? Could you add the steps from job 2 into job 1’s definition?

Before `dbt build` was released, my old team did have a job for snapshots which ran separately to the rest of the project. We only ran it once a day and scheduled it far enough in advance of the other runs that they never overlapped, but it doesn’t sound like that solves your use case.

I’d be interested to see the specific problem you’re trying to solve

---

<div class="post-metadata">

### Author: ![obar1](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/obar1/32/1221_2.png) [@obar1](https://discourse.getdbt.com/u/obar1)
#### Post date: [October 7, 2022, 6:58am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/3 "2022-10-07T06:58:26Z")

</div>

hi @joellabes I have 1 model that is big let’s call it _big\_job_ so the idea was to split the original 1 job we had in 2 jobs like so

JOB\_Y

```auto
dbt build --select +big_job+

```

and another  
JOB\_OTHERS

```auto
dbt build --exclude +big_job+

```

but there are issues with the intermediate layer now:  
as JOB\_OTHERS will not materialize the intermediate that in the lineage graph are ancestor on big\_job (exclude cmd) so I need to run first JOB\_Y to build them and then JOB\_OTHERS  
Having 1 in all take too much time … to get some big\_job done and we cannot increate AWS spending for new 😛 so in this way we prioritize big\_job over the rest …  
any suggestion how to do it better ?

---

<div class="post-metadata">

### Author: ![joellabes](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/joellabes/32/3980_2.png) [@joellabes](https://discourse.getdbt.com/u/joellabes)
#### Post date: [October 10, 2022, 12:15am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/4 "2022-10-10T00:15:21Z")

</div>

I’m a bit confused - you said that having them all in a single job takes too much time, and so you want to prioritise the `big_job` over the rest, but in your original message you asked for them to run in series. Splitting it out won’t reduce the total time to run if you’re only going to start job 2 after job 1 is complete anyway.

> [@obar1](#):
>
> job1 → job2 → job1 etc for ever

How many threads do you have in your job definition? You said AWS spend so I assume you’re using Redshift. Too many threads in Redshift can lead to resource contention and the job actually running _slower_. I would aim for 4 (maybe try with up to 8 and see where you get a decreasing return).

You might be able to create a single job that contains several steps, something like this:

```auto
dbt build -s +big_job --exclude big_job --threads 4
dbt build -s big_job --threads 1
dbt build -s big_job+ --exclude big_job --threads 4

```

The one thing to keep in mind here is that any nodes that aren’t related to `big_job` won’t be built using this selector. If that’s a problem, you might be able to express it with a [YAML selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors) but I haven’t spent enough time using those to have any useful guidance.

---

<div class="post-metadata">

### Author: ![obar1](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/obar1/32/1221_2.png) [@obar1](https://discourse.getdbt.com/u/obar1)
#### Post date: [October 12, 2022, 11:53am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/5 "2022-10-12T11:53:38Z")

</div>

**tagging** can be handy as well when I have more the once big job 🙂

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex020/uploads/getdbt/original/1X/a7a7ca1fe379aaf90952b0e13118a817babcd14f.png) [@system](https://discourse.getdbt.com/u/system)
#### Post date: [October 19, 2022, 11:54am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/6 "2022-10-19T11:54:21Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

### Author: ![joellabes](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/joellabes/32/3980_2.png) [@joellabes](https://discourse.getdbt.com/u/joellabes)
#### Post date: [April 8, 2024, 12:26am UTC](https://discourse.getdbt.com/t/running-two-dbt-cloud-jobs-back-to-back/5017/7 "2024-04-08T00:26:39Z")

</div>

dbt Cloud now supports triggering one job when another job finishes - I still don’t think this is necessarily the right solution for the use case described in this thread, but it ranks highly in Google for the search term so I’m adding a link for others who come along to it:

> **[Deploy jobs | dbt Developer Hub](https://docs.getdbt.com/docs/deploy/deploy-jobs#trigger-on-job-completion)**
>
> Learn how to create and schedule deploy jobs in dbt Cloud for the scheduler to run. When you run with dbt Cloud, you get built-in observability, logging, and alerting.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/getdbt/original/2X/3/3211b91e1bd14c976e4106fdbb2738880abbf3b5.png)
