Cannot Create Custom Adapter—Assumes schema

Hi There,

I am trying to put together a barebones adapter for MemSQL. I have a very simple adapter plugin set up and installed. When I run dbt compile I get the following error:

AttributeError: 'MemsqlCredentials' object has no attribute 'schema'

This is true; like MySQL, MemSQL does not support schemas, only databases and tables.

Looking closer, I can see the error stems from File "/usr/local/lib/python3.7/site-packages/dbt_core-0.14.0-py3.7.egg/dbt/context/common.py", line 410, in generate_base "schema": config.credentials.schema which has the following method:

def generate_base(model, model_dict, config, manifest, source_config,
                  provider, adapter=None):
    """Generate the common aspects of the config dict."""
    if provider is None:
        raise dbt.exceptions.InternalException(
            "Invalid provider given to context: {}".format(provider))

    target_name = config.target_name
    target = config.to_profile_info()
    del target['credentials']
    target.update(config.credentials.serialize(with_aliases=True))
    target['type'] = config.credentials.type
    target.pop('pass', None)
    target['name'] = target_name

    adapter = get_adapter(config)

    context = {'env': target}

    pre_hooks = None
    post_hooks = None

    db_wrapper = provider.DatabaseWrapper(adapter)

    context = dbt.utils.merge(context, {
        "adapter": db_wrapper,
        "api": {
            "Relation": db_wrapper.Relation,
            "Column": adapter.Column,
        },
        "column": adapter.Column,
        "config": provider.Config(model_dict, source_config),
        "database": config.credentials.database,
        "env_var": env_var,
        "exceptions": dbt.exceptions.wrapped_exports(model),
        "execute": provider.execute,
        "flags": dbt.flags,
        # TODO: Do we have to leave this in?
        "graph": manifest.to_flat_graph(),
        "log": log,
        "model": model_dict,
        "modules": get_context_modules(),
        "post_hooks": post_hooks,
        "pre_hooks": pre_hooks,
        "ref": provider.ref(db_wrapper, model, config, manifest),
        "return": _return,
        "schema": config.credentials.schema,
        "sql": None,
        "sql_now": adapter.date_function(),
        "source": provider.source(db_wrapper, model, config, manifest),
        "fromjson": fromjson,
        "tojson": tojson,
        "target": target,
        "try_or_compiler_error": try_or_compiler_error(model)
    })

    return context

This function makes an assumption about the presence of schema in credentials. Is there any way to work around this?

Hey there - cool to hear you’re making an adapter for MemSQL - I’d love to learn more about that!

dbt-core definitely does assume that the warehouses it works with will have a database and schema layer. That may change in the future, but for now, I think it’s the kind of thing you’ll need to work around.

We did something similar for SparkSQL. Spark was weird because it has a notion of a database and a schema, but they both refer to the same thing and can be used interchangeably. If you want to poke around in the repo, you can do so here: https://github.com/fishtown-analytics/dbt-spark

Check out how we include both a database and schema in the credentials contract and then mutate the the database config in the Credentials constructor.

I’m not exactly sure what the best way to hack this it - maybe you can supply a phony value for the schema? You’ll probably also want to create a new Relation class with a custom include_policy. This should make it possible to exclude the schema from the relation name when dbt renders it out.

Hope this helps!