dbt run error: Maximum number of tokens exceeded

Help
,
1

The problem I’m having

My company’s daily production run halted during the dbt run command with this error on two models:

11:36:44 Completed with 2 errors, 0 partial successes, and 0 warnings:

11:36:44

11:36:44 Failure in model ()

11:36:44 Maximum number of tokens exceeded ().

11:36:44

11:36:44 Failure in model (<model 2 path>)

11:36:44 Maximum number of tokens exceeded ().

11:36:44

11:36:44 Done. PASS=XXX WARN=0 ERROR=2 SKIP=27 NO-OP=0 TOTAL=XXX

`

Any help on what these tokens are and how to overcome the error would be most appreciated.

The context of why I’m trying to do this

I need to fix the error and get the production run to complete so that my project’s data can be delivered to its end users.

What I’ve already tried

I’ve been unable to find documentation on the error message or what tokens these are.

Some example code or error messages

11:36:44  Completed with 2 errors, 0 partial successes, and 0 warnings:
11:36:44  
11:36:44  Failure in model <model name> (<model path>)
11:36:44    Maximum number of tokens exceeded (<quantity>).
11:36:44  
11:36:44  Failure in model <model2 name> (<model 2 path>)
11:36:44    Maximum number of tokens exceeded (<quantity>).
11:36:44  
11:36:44  Done. PASS=XXX WARN=0 ERROR=2 SKIP=YY NO-OP=0 TOTAL=XXX
2

On my team we currently suspect that we need to install sqlparse 0.5.4 instead of the more recent version 0.5.5. I will update once we pass the error point on our production build.

3

I can confirm that using sqlparse 0.5.4 is a fix for this error. We added the “Install sqlparse 0.5.4” action to our workflow.

  - name: Install sqlparse.0.5.4

    run: |

      pip install sqlparse==0.5.4
4

Thanks @Bill ! This is definitely what happened to us last night. Same setup, dbt airflow snowflake, this was the fix.

looks like 0.5.5 was pushed last night too.