Combining results of multiple transformations into a single final table

[daniele.frigo]

I’m trying to convert an ODI project into dbt (moving from Oracle target to Snowflake).
In one datamart I have 5 different “sections” of data coming from different transformation processes that at the end I need to “merge” in the target table simply appending them.
The original approach was to create 5 different workflows, executed in parallel, with 5 different INSERT INTO in the same table as last step.
In dbt this is not possible (as far as I know); I can prepare 5 work tables and then do the final insert with 5 UNION ALL, but the original approach is probably better parallelized.
How do you guys handle this kind of situations?
Thanks
Daniele

[fabrice.etanchaud]

Hi @daniele.frigo, with dbt’s built-in parallelization, work tables should be prepared in parallel; why do you think it could be worse ? And should “inserting into separate datasets”, and then “unioning all” be a better option concerning locks ?

Best regards,

[daniele.frigo]

The locking could be an issue on some databases, it strongly depends on the lock implementation of the specific vendor.
The original implementation essentially skips one step, the last one where I actually do UNION ALL and insert in the final table (I did not explain that correctly in my previous post, it’s not a matter or parallelization, which is absolutely perfect with dbt).