Error when attempting to convert a dataframe to pandas.

Help
1

Hi everyone,

I’m getting the error ‘Schema has to be provided to write_pandas when a database is provided’ when attempting convert a dataframe to pandas in a python script. I’m using snowflake.

I’ve seen the following error related to using a custom schema name: [CT-1813] [CT-1378] [Bug] Python models not picking up custom schema · Issue #393 · dbt-labs/dbt-snowflake · GitHub however when testing without a custom schema name the error persists.

Anyone else experienced this?

Thanks

2

Hey @darren.hickey, are you able to post the code you’re trying to run?

3

Hey Joel,

Yeah the offending piece of code is the following: final_df = tickets_df.to_pandas()

Thanks

4

The good news is that issue you linked above has been resolved and will come out in dbt-snowflake 1.4.1.

We’ve only seen this with custom schema names configured - is it possible that when you changed to testing without a custom schema, a config file didn’t save or something?

5

+1 on this-one! Any timeline for 1.4.1 ?

6

I am having the same issue.
Running with dbt=1.4.1

7

dbt-snowflake v1.4.1 came out earlier today, have you checked that you have the patch for dbt-snowflake installed, not just dbt-core 1.4.1?

8

I just tested dbt-snowflake 1.4.1 and can confirm the issue still persist @joellabes . The changelog of 1.4.1 also does not address this issue, so I am not surprised.

9

Is there any other alternative to “to_pandas()”

10

Hi Joel, we have now tried on 1.4.1 and having fully removed the custom schema name and we are still getting the error.

11

OK - we don’t really have enough to reproduce this, can you post the full code file (not just the final line of code) for the model you’re trying to run as well as the logs from when you try doing a dbt run?

12

@troyel yes my mistake, it didn’t go out in 1.4.1, but confirmed with the PM that it is actively being worked on! I don’t have a version to share sorry

13

We’re experiencing the same issue for one of our Python dbt models that uses a custom schema. Wanted to hop on this thread to be notified of when this would be resolved! (Should also mention that we are on dbt-snowflake==1.4.1)

14

cross-posting from: [CT-1813] [CT-1378] [Bug] Python models not picking up custom schema · Issue #393 · dbt-labs/dbt-snowflake · GitHub

@patkearns10 and I managed to get to the bottom of this by live-debugging with a very helpful & generous user who was running into the issue!

I’m not sure why this bug is cropping up for some Snowflake users, and not others; I believe it should have been solved at the source in snowflake-connector-python==3.0 (included in dbt-snowflake>=1.4).

For anyone still experiencing the issue, this seems to be a valid workaround:

def model(dbt, session):
    dbt.config(schema="custom_schema")
    pandas_df = dbt.ref("my_model").to_pandas()
    
    # add these lines
    session.use_database(dbt.this.database)
    session.use_schema(dbt.this.schema)
    
    return pandas_df

We’ll see if there’s a way to include those session.use_* calls within the dbt materialization code, so that you don’t need to write it in every Python model that returns a Pandas dataframe.

2 Likes
15

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.