Hey there!
Context
- Data warehouse: Snowflake
- Ingestion tools: Airbyte, custom code
- Transformation: dbt (of course!)
- Our cloud otherwise: AWS
What I want to do
I have a Stripe source and a Paypal source, and I want to use dbt to get it into a unified Payments table.
Then, I want to enrich that data, including from web-hosted ML models (like OpenAI), and from web geocoding services.
I then want to use dbt’s Python integration to do some complicated aggregation logic.
So what’s the problem?
I am curious what the recommended practice is to do the middle step, of enriching data with external REST APIs. It doesn’t look like I can do this in dbt itself, and I’m wondering if having an orchestration framework like Airflow is a requirement. I would rather avoid that complexity if I can!
Thank you in advance 