Just to add on this, I use
MY_MODELS=$(dbt -q ls -m my/path/to/models --output name | xargs -I{} echo -n ' "{}",' | sed 's/,*$//' | sed 's/\(.*\)/[\1]/')
# returns ["model_a", "model_b"]
where dbt -q trims out any logs and leaves just the output, and the two consecutive sed remove the last trailing comma and wraps the result in square brackets, respectively.
then,
dbt -q run-operation generate_model_yaml --args '{"model_names": '"$MY_MODELS"' }'
notice that using single quotes renders the variable as a literal value, to expand it we need to close the quotes before the variable and reopen them again after using it, like 'my'"$VAR"'is cool'. see shell - Expansion of variables inside single quotes in a command in Bash - Stack Overflow for details.
wonky
May 31, 2024, 5:10pm
4
Hi I’ve expanded the capability of this here
dbt-labs:main ← dbt-labs:feat/justfile-improve-dx
This is more likely a feature request for codegen its self but this is a feature… I added to our justfile
```suggestion
# Generate docs blocks markdown for all unique columns in a folder. For making DRY documentation. Inspired by https://docs.getdbt.com/blog/generating-dynamic-docs-dbt
dbt-generate-column-docs-md folder=default_folder:
#!/usr/bin/env bash
models=$({{dbt}} ls -m {{folder}} -q --output name | xargs -I{} echo -n ' "{}",' | sed 's/,#$//')
models="[$models]"
model_name="_$(echo {{folder}} | cut -d'/' -f2- | tr '/' '_')"
{{dbt}} run-operation generate_model_yaml --args "{ \"include_data_types\": false, \"model_names\": $models}" > /tmp/${model_name}.tmpyml
awk '/models:/{p=1} p' /tmp/${model_name}.tmpyml > /tmp/temp${model_name} && mv /tmp/temp${model_name} {{folder}}/${model_name}.yml.tmp
grep ' \- name:' {{folder}}/${model_name}.yml.tmp | cut -c 15- | sort -u > {{folder}}/${model_name}.md.tmp
if [ ! -d "generated" ]; then \
mkdir -p generated; \
fi
if [ -f {{folder}}/${model_name}.md ]; then \
merge=true; \
output_folder="generated"; \
else \
output_folder="{{folder}}"; \
fi
while read line; do
echo "{% docs column_${model_name}__${line} %}" >> ${output_folder}/${model_name}.md;
echo "" >> ${output_folder}/${model_name}.md;
echo "{% enddocs %}" >> ${output_folder}/${model_name}.md;
echo "" >> ${output_folder}/${model_name}.md;
done < {{folder}}/${model_name}.md.tmp
rm {{folder}}/${model_name}.md.tmp
rm {{folder}}/${model_name}.yml.tmp
echo "Docs blocks ${model_name} generated in ${output_folder}/${model_name}.md"
if [ "$merge" = true ] ; then
file1="${output_folder}/${model_name}.md"
file2="{{folder}}/${model_name}.md"
output="{{folder}}/${model_name}.md"
if [[ "$OSTYPE" == "darwin"* ]]; then
echo "Warning: If you are running this script on a Mac, please make sure you have gawk installed. You can install it using 'brew install gawk'"
awk_cmd="gawk"
else
awk_cmd="awk"
fi
$awk_cmd '
/^{% docs / {tag=$0; next}
/^{% enddocs %}/ {a[tag]=doc; doc=""; next}
{doc=(doc==""?"":doc "\n") $0}
END {n = asorti(a, b); for (i=1; i<=n; i++) print b[i] "\n" a[b[i]] "\n{% enddocs %}\n"}
' "$file1" "$file2" > /tmp/${model_name}.md.tmp && mv /tmp/${model_name}.md.tmp $output
echo "Docs blocks ${model_name} merged in ${output}"
fi
```