How to enrich data via APIs?

Hey there!


  • Data warehouse: Snowflake
  • Ingestion tools: Airbyte, custom code
  • Transformation: dbt (of course!)
  • Our cloud otherwise: AWS

What I want to do

I have a Stripe source and a Paypal source, and I want to use dbt to get it into a unified Payments table.
Then, I want to enrich that data, including from web-hosted ML models (like OpenAI), and from web geocoding services.
I then want to use dbt’s Python integration to do some complicated aggregation logic.

So what’s the problem?

I am curious what the recommended practice is to do the middle step, of enriching data with external REST APIs. It doesn’t look like I can do this in dbt itself, and I’m wondering if having an orchestration framework like Airflow is a requirement. I would rather avoid that complexity if I can!

Thank you in advance :pray: