I’ve been at the same crossroads just weeks ago, and your thoughts are spot on. It was a tough call, but we ultimately went with Dagster, mostly due to superior tooling (Dagit) + flexible programming model + community. I honestly thing you’ll make a good decision either way, but for us it just seemed like Dagster “thinks” holistically about the process and challenges of making data applications, whereas Prefect solves for developing and executing pipelines in a very ergonomic way, but it’s not as complete.
I saw this post from Nick Schrock on the Dagster slack community, that I think gets at the core of the difference between them:
Dagster pipelines are more structured and constrained. This allows us have a lot of additional features (a type system, a config management system, a system-managed context object that flows through the compute, among other things). By constrast Prefect pipelines are more minimal and dynamic.
Another way of framing this difference is that dagster is very interested in what the computations are doing rather than only how they are doing it. We consider ourselves the application layer for data applications (rich metadata, type system, structured events with semantic meaning etc), whereas prefect frames their software in terms of “negative” and “positive” engineering. This negative/positive framing is more exclusively about the “how” of pipelines: retries, operational matters etc.