DBT in Azure Function

Archive
1

We are using AZURE Function to trigger a DBT project and run. we are using ADLS to store the DBT Project.

we are getting the below error and the execution fails:

[3/11/2020 2:55:18 AM] Running with dbt=0.15.2
[3/11/2020 2:55:19 AM] Error importing adapter: cannot import name ‘BlockBlobService’ from ‘azure.storage.blob’ (C:\Users\XXXXXX\AzureFunctions\jaffeldbtfunapp.venv\lib\site-packages\azure\storage\blob_init_.py)
[3/11/2020 2:55:19 AM] Encountered an error while reading profiles:
[3/11/2020 2:55:19 AM] ERROR Runtime Error
[3/11/2020 2:55:19 AM] Credentials in profile “jaffle_snowflake”, target “dev” invalid: Runtime Error
[3/11/2020 2:55:19 AM] Could not find adapter type snowflake!

Can you please help us if you faced this issue ? in the requirements.txt we added all the below packages.

azure
azure-functions
azure-storage-file-datalake
dbt==0.15.2
azure-storage-blob==2.1
azure-storage-common==2.1

2

Hi All,

The issue resolved when used Blob storage instead of ADLS Gen 2

3

Hi sudhirnune,

Are you able to offer any advice or guidance on how you set up your Azure Function. Are you using a Python function? Are you using a container?

We are struggling with where you actually host the dbt runtime and how you call it. Any advice/code would be much appreciated.

Regards,
Troy

4

Any Update on above request? Even we are trying to schedule the Dbt run using Function.

5

Hi @dallyhorton ,

Sorry for delayed response, we dropped the usage DBT from function and we using docker Image with container instance approach.

DBT import into python does not work as its not yet stable to be user as a library in python

We run the DBT from python as a sub-process and capture the execution status and stderr & stdout.

6

Hi, Even we are using docker based solution to run DBT in Production ,It runs fine untill we rebuild the docker image and do the deployment,after deployment first run of DBT always get stuck and there is no Output messge. Are you facing issue after deployment? However if I kill first run after deployment consecutive runs are all good.

7

@PayalP - we have been using docker based approach, but did not see this type of issue, I have few questions:

  1. what is the CMD in the docker file ?
  2. How is the docker executed ? (Local / Cloud - using ?)
  3. what is the command used to run the DBT ?
  4. Is the project inside the Docker ?
  5. when the docker is stuck are you able to see the login to DOCKER Run, see what the DBT Log has ?
8

@sudhirnune We are using solution given in below link with some change.

We build the docker from Devops CD pipeline and deploy to function app every day. This function app is executed from ADF pipeline using HTTp trigger. I am not Function app person but trying to fix the first run after deployment. There is no error return from From function app or DBT but first run never get finished. the log say something like below . Now m not sure where its going wrong…When I cancel the run and start again it works fine… but every first run after deployment is stuck. The DBT Command I am running is dbt run -m +XXXX +YYY+

When I check DBT Log I can see it has successfully finished few model and suddenly DBt worker stops with Below message…:

Stopping task hub worker. IsGracefulStop: False. InstanceId: . Function: . HubName: bidbtprodfa. AppName: bi-dbt-prod-fa. SlotName: Production. ExtensionVersion: 2.3.1. SequenceNumber: 8.

Task hub worker stopped. IsGracefulStop: False. Latency: 00:00:00.0688840. InstanceId: . Function: . HubName: bidbtprodfa. AppName: bi-dbt-prod-fa. SlotName: Production. ExtensionVersion: 2.3.1. SequenceNumber: 9.

Singleton lock renewal failed for blob ‘bi-dbt-prod-fa/host’ with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 0001-01-01T00:00:00Z (-2147483648 milliseconds ago) with a duration of 0 milliseconds. The lease period was 15000 milliseconds.

Failed to renew host lock lease: Another host has acquired the lease. The last successful renewal completed at 2021-07-06T08:08:49.603Z (12016 milliseconds ago) with a duration of 12 milliseconds.

9

Hi @PayalP,

Any reason why are you using both Docker and Azure Function ? If you are using ADF you can Run a docker image on a ACI, using WEB Activity of ADF. Can you give it a try ?