How to debug an issue occurring within threads

Help
,
1

Hey folks!
I have built a custom macro (post-hook) for incremental models that does the following if dbt run -m <incremental_parent>+:

  1. Check the checksum of the <incremental_parent> .
  2. If the checksum of the model has changed, drop all of the children of that model that are also built incrementally.
  3. Because the incrementally built children have been dropped, running dbt run -m <incremental_parent>+ should result in the children being built from scratch - since they should now not exist in dbt.

I was able to get 1. and 2. working but I end up with a database error for the third point, where the thread that builds out the children doesn’t seem to identify that the child has been dropped.
I’m not sure why this is happened as if I run the command dbt run -m <incremental_parent>+ again (after the first time which produces the error), the children do indeed get built from scratch.
I would genuinely appreciate any advice on why this might be happening so I can materialize this effort, and thanks in advance for the same!
Kindly find the macro here:

{% macro drop_modified_incremental_downstream(modified_incrementals) %}

    {%- set default_schema = target.schema -%}
    {% set models_to_drop = [] %}

    {%- for node in graph.nodes.values()
     | selectattr("resource_type", "equalto", "model")
     | selectattr("config.materialized", "equalto", "incremental") %}

     {% if modified_incrementals[0] in node.refs[0] %}
        {% do models_to_drop.append(node.name) %}
     {% endif %}

    {% endfor %}

    {% for rel in models_to_drop %}
        {% set drop_query %}
            DROP TABLE IF EXISTS {{ default_schema }}_{{ env_var("DBT_USER") }}.{{ rel }};
        {% endset %}
        {% do run_query(drop_query) %}
    {% endfor %}
    {% do return(['Done']) %}

{% endmacro %}

{% macro is_incremental_modified(model_name) %}

    {%- set modified_incrementals = [] %}
    {% set foo = false %}

    {% if execute %}
    {%- for node in graph.nodes.values()
     | selectattr("name", "equalto", model_name) %}

            {% set last_stored_checksum = 'ad0b6192abb7548da303224afb8663544b25e18e213c756117aaa3685d6cc19a' %}
            {%- set current_model_checksum = node.checksum %}

        {%- if current_model_checksum != last_stored_checksum %}
            {% do log("Checksum modified since last run", info=true) %}
            {% do log("Modified incrementals: " ~ node.name, info=true) %}
            {% do modified_incrementals.append(node.name) %}
        {% endif %}

    {% endfor %}

    {%- if modified_incrementals | length > 0 %}
        {% do log("Modified Incremental models found in run.", info=true) %}
        {% do log("Dropping downstream models", info=true) %}

        {% set status = drop_modified_incremental_downstream(modified_incrementals) %}
    {% endif %}
    {% endif %}

{% endmacro %}
2

This is a very interesting idea! I think you’re having trouble because you are dropping a table after the dbt invocation started. I’m pretty sure that during its setup/compilation phase, dbt goes through and finds all the models it’s going to need to build, checks if they exist and generates the appropriate sql.

Because you are dropping the downstream models in a post hook, dbt is expecting to find them and then panics when it can’t.

I don’t think this is a threading error, but you could conclusively prove that by doing dbt run ... --threads 1. If that succeeds (I don’t think it will), then I would open a bug on GitHub.

Instead I think you’d need to do this as two separate dbt invocations - firstly build the new one and drop the downstream children, and then a second job which will rebuild the children.

Another option could be to override the should_full_refresh() macro, which has the final say on whether a model should be dropped and rebuilt or not. Not 100% sure if that’ll work, but it feels more dbtonic than manually dropping things!