how can dbt read files and use it when refresh model

Help
1

i have some configuration files. inside the configuration file, it has different attribute value like account_id, name. how can i read the file during model refresh? found below solution, but looks like dbt doesn’t have read_file function.

image
image642×679 21.4 KB

configuration files is like below:
xx.config:

BINARY_TAB_FILE=TRUE
USE_END=TRUE
ACCOUNT=10000

Anyone done this before? thanks so much in advance.
ZX

2

Are all your configurations just key value pairs and do you have just 3 configs ?
If so then you could use either environment variables or dbt variables.

3

How we are handling this currently is to store large configs in a macro.

For example:

{%- macro get_configs() -%}
  {{- return({
    "key": "value",
    "key2": "value2",
    "obj": {
      ...
    },
    ...
  }) -}}
{%- endmacro -%}

But it would definitely be nice to be able to load these configs from a YAML file instead!

4

Actually, we have now moved to storing these configs in a project variable inside of dbt_project.yml!

Like this:

vars:
  configs:
    key: value
    key2: value2
    key3:
      a: 1
      b: 2
      ...