Cannot Create Custom Adapter—Assumes schema

Archive
1

Hi There,

I am trying to put together a barebones adapter for MemSQL. I have a very simple adapter plugin set up and installed. When I run dbt compile I get the following error:

AttributeError: 'MemsqlCredentials' object has no attribute 'schema'

This is true; like MySQL, MemSQL does not support schemas, only databases and tables.

Looking closer, I can see the error stems from File "/usr/local/lib/python3.7/site-packages/dbt_core-0.14.0-py3.7.egg/dbt/context/common.py", line 410, in generate_base "schema": config.credentials.schema which has the following method:

def generate_base(model, model_dict, config, manifest, source_config,
                  provider, adapter=None):
    """Generate the common aspects of the config dict."""
    if provider is None:
        raise dbt.exceptions.InternalException(
            "Invalid provider given to context: {}".format(provider))

    target_name = config.target_name
    target = config.to_profile_info()
    del target['credentials']
    target.update(config.credentials.serialize(with_aliases=True))
    target['type'] = config.credentials.type
    target.pop('pass', None)
    target['name'] = target_name

    adapter = get_adapter(config)

    context = {'env': target}

    pre_hooks = None
    post_hooks = None

    db_wrapper = provider.DatabaseWrapper(adapter)

    context = dbt.utils.merge(context, {
        "adapter": db_wrapper,
        "api": {
            "Relation": db_wrapper.Relation,
            "Column": adapter.Column,
        },
        "column": adapter.Column,
        "config": provider.Config(model_dict, source_config),
        "database": config.credentials.database,
        "env_var": env_var,
        "exceptions": dbt.exceptions.wrapped_exports(model),
        "execute": provider.execute,
        "flags": dbt.flags,
        # TODO: Do we have to leave this in?
        "graph": manifest.to_flat_graph(),
        "log": log,
        "model": model_dict,
        "modules": get_context_modules(),
        "post_hooks": post_hooks,
        "pre_hooks": pre_hooks,
        "ref": provider.ref(db_wrapper, model, config, manifest),
        "return": _return,
        "schema": config.credentials.schema,
        "sql": None,
        "sql_now": adapter.date_function(),
        "source": provider.source(db_wrapper, model, config, manifest),
        "fromjson": fromjson,
        "tojson": tojson,
        "target": target,
        "try_or_compiler_error": try_or_compiler_error(model)
    })

    return context

This function makes an assumption about the presence of schema in credentials. Is there any way to work around this?

2

Hey there - cool to hear you’re making an adapter for MemSQL - I’d love to learn more about that!

dbt-core definitely does assume that the warehouses it works with will have a database and schema layer. That may change in the future, but for now, I think it’s the kind of thing you’ll need to work around.

We did something similar for SparkSQL. Spark was weird because it has a notion of a database and a schema, but they both refer to the same thing and can be used interchangeably. If you want to poke around in the repo, you can do so here: https://github.com/fishtown-analytics/dbt-spark

Check out how we include both a database and schema in the credentials contract and then mutate the the database config in the Credentials constructor.

I’m not exactly sure what the best way to hack this it - maybe you can supply a phony value for the schema? You’ll probably also want to create a new Relation class with a custom include_policy. This should make it possible to exclude the schema from the relation name when dbt renders it out.

Hope this helps!

1 Like
3

Hi,
I was wondering if you’ve made any progress in porting dbt for MemSQL?

4

I spent a couple of days on this, but did’t make it very far. From what I recall, the main hurdle is that dbt assumes some concept of multiple schemas / databases, but this paradigm doesn’t work well with MySQL / MemSQL.

@greenlitmysql — I’m not sure of your use case, but one thing you might consider is using dbt with Presto (which, itself, seems to have issues), and then use Presto to federate MemSQL. From my testing, you will probably be able to use do this, but only if you’re using the recently released MemSQL 7. The main drawback except for the complexity of adding Presto to your stack is creating COLUMNSTORE tables. If you’re doing data warehousing, this is highly desirable.

Happy to chat further on this. We have love, love, loved MemSQL as a fast data warehouse solution, but often if just doesn’t quite work with other systems in our stack (Presto, Kafka, dbt, Metabase) that we either have to find a workaround or ditch the idea entirely.