dbt with AWS Athena and Python models.

Help
1

From Python models | dbt Developer Hub , seems only the following 3 platforms supports Python models.

  • Databricks, Snowflakes, BigQuery.

I am thinking to have dbt get source tables from Athena, doing some transform, then store the result in S3. However, I cannot find a way if I can use dbt-python models on this? Or it must involve one of above 3 platforms.

Any examples would be super helpful.

Thanks
yanan

2

Hi Yanan,

At the moment, those three platforms are the only ones that support dbt Python models natively.

However, you might be able to use the fal-dbt adapter to run your Python workloads.

1 Like
3

Thanks Joel for the help. I did a quick try on dbt-fal, by following the link:

  1. The profile seems file, with fal in the picture:
        target: dev
        outputs:
          dev:
            type: fal
            db_profile: dev_athena
          dev_athena:
            type: athena
  1. I have a very simple sql and py file, the py ref to the sql:
      import pandas
      def model(dbt, fal):
          df = dbt.ref("test_query")
          return df
  1. dbt run got the success on sql, but not the py:
       $ dbt run
        19:14:51  Running with dbt=1.3.2
        19:14:51  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 282 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics
        19:14:51
        19:14:55  Concurrency: 1 threads (target='dev')
        19:14:55
        19:14:55  1 of 2 START sql table model dbt.test_query .................................... [RUN]
        19:15:14  1 of 2 OK created sql table model dbt.test_query ............................... [**OK -1 in 18.75s**]
        19:15:14  2 of 2 START python view model dbt.demo_athena_python .......................... [RUN]
        19:15:14  2 of 2 ERROR creating python view model dbt.demo_athena_python ................. [**ERROR in 0.03s**]
        19:15:14
        19:15:14  Finished running 1 table model, 1 view model in 0 hours 0 minutes and 23.07 seconds (23.07s).
        19:15:14
        19:15:14  Completed with 1 error and 0 warnings:
        19:15:14
        19:15:14  Runtime Error in model demo_athena_python (models\demo_athena_python.py)
        19:15:14    **Materialization "materialization_view_athena" only supports languages ['sql']; got "python"**
        19:15:14
        19:15:14  Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2

feels like it still may not work on my scenario ‘athena’ case. Or something I missed?

4

Hey there, this is Meder from fal. We indeed do not support Athena yet (thanks for pointers @joellabes). I just created a ticket for it, so we should get it working pretty soon. In the meantime, @yzhang, if you have time, you can create an issue in our github repository: GitHub - fal-ai/fal: do more with dbt. fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.

2 Likes
5

Hi again! So we just released dbt-fal 1.3.13 that enables support for dbt-athena-community. To install, run pip install dbt-fal==1.3.13. @yzhang, your Python models should compute as long as you set materialized config to table:

import pandas
def model(dbt, fal):
  dbt.config(materialized="table")
  df = dbt.ref("test_query")
  return df

Let me know if it works!

2 Likes
6

Thank you! I just confirmed, with dbt.config(materialized=“table” , it went through my code. Again, thank you very much for the help.

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.