We’ve been using dbt for a quite a while now and loving it! However as great as it is for working inside of the data warehouse, there’s still a lot stuff we need to do before the data gets into the data warehouse and into domain of dbt.
We’ve been benchmarking the data orchestration tools, and we’re considering implementing either Dagster or Prefect. Both of them seem really great and hugely popular inside the scene. And now both of them support dbt as well.
My initial thoughts:
- They both seem to have the same standard functionality and great code usability. They work very similarly. However Dagster has a bit more versatility with integrations (jupyter/papermill is appreciated)
- Dagster seems to have better UI and tools for debugging data pipelines locally. This is hugely beneficial as data pipelines grow more complex.
- Prefect has better cloud operations and less maintenance with native Prefect Cloud service, which is appreciated. We’re happy to pay some premium for less work in maintenance.
Does anybody have any hands-on experience and could give some thoughts? Or any direct recommendations? Or should we consider something else entirely?