Moving away from manual end to end testing
My current pipeline looks like this: legacy MSSQL DB → FiveTran → Snowflake → dbt → Snowflake marts → proprietary dashboard UI.
We are trying to avoid a huge overhead of manual end to end tests of the data pipeline
We are trying to achieve scalability and ease of use that automation brings
Because the data is ostensibly the same from here (legacy MSSQL DB → FiveTran - Snowflake) and with a few exceptions the same here (Snowflake marts → proprietary dashboard UI) we think that dbt is our friend in this scenario and we can leverage dbt packages to reconcile the data from start to end point (e.g. sample test this field contains 50 at the beginning of the process, does it contain 50 at the end of the process?)
What We’ve already tried
dbt unit test packages - which provide a degree of comfort, but are not specifically reconciling the data from source to endpoint
Do dbt data rec packages exist?
Or has some tried to carry out data reconciliation similar to this using dbt and a programming language eg Python?