Could someone please tell me if I can access MS Fabric Lakehouse parquet tables from dbt? We have created a Fabric Lakehouse and would like to use those parquet tables as a source to the Fabric Warehouse models.
1 Like
You can read them from the SQL endpoint of the lakehouse.
Note: @Josh originally posted this reply in Slack. It might not have transferred perfectly.
I have a similar question and am also wondering if there are any solutions. Is it possible to submit Spark jobs to Fabric Lakehouse using dbt?
We have a bunch of Spark jobs that migrated from Hadoop. By shifting to Synapse Spark, It could be best compatible with the Hive SQL, UDFs, and Spark RDD APIs.
Update: I’ve just found a project named dbt-fabricspark intended for this and tried to connect Lakehouse Spark by Livy endpoint, but there are no docs and connect failed on SSL CERTIFICATE even though I fixed the legacy APIs into new dbt-core’s.(BTW project dbt-fabric works well for the same tenants and subscriptions)
I’ve fixed this SSL verification failure issue, It was caused by my local network Zscaler rules. The dbt-fabricspark adapter connected successfully either for legacy APIs or my modification version.