dbt deps fails to retrieve module from github when ran in cloud dbt

Help
,
1

The problem I’m having

Libraries listed in packages.yml are failing in cloud dbt. The libraries are located in a github private repository.

The context of why I’m trying to do this

Retrieving model code that has been packaged into a module.

What I’ve already tried

For testing, the github username and access token are directly in the URL. dbt deps works locally but when the code runs in cloud dbt it fails.

Some example code or error messages

This is the error message with ellipses replacing some of the pathing:

Error checking out spec=‘None’ for repo https://xxxxxxxxxxxxxxxxxxxxxx@github.com/....feeds/....bingads.git
remote: Repository not found.
fatal: repository ‘https://github.com/…-feeds/…bingads.git/’ not found

2

Hey @dandoyle, I understand that you can’t show the actual tokens in the public message, but it makes it tricky to debug! Could you reach out to the dbt Cloud support team in-app? They will be able to have a closer look at logs etc and be able to be more helpful

1 Like
3

Hi @dandoyle, I’m facing the exact issue with importing private packages from Github in dbt Cloud. I was wondering whether you have managed to find a solution or workout for the issue yet :pray:

4

This might be the cause of your issue: Private packages must be cloned using access tokens provided by environment variables | dbt Developer Hub

7

We did that too and it failed. The debug message makes it very difficult to know if we configured it incorrectly if indeed Cloud dbt is looking for variable substitution. Well, I guess I will give it go again after a week with a fresh mind. Maybe I will not repeat a syntactical error if that is what was tripping things up.

As a heads up, we contacted support. They told us to go back to the community. I am back checking on this thread after so long because support just determined they could not support us - after more than 10 days. I am a little frustrated and disappointed. Thank you for your help.

10

After some testing, I determined that whatever was failing is now fixed in Cloud dbt. So the rules are not a thing and everything is working as documented.

Here are the tests I ran after the above success which used the access token env var from the page Joel linked as DBT_ENV_SECRET_GIT_CREDENTIAL:

- git: "https://{{ env_var('DBT_ENV_SECRET_GIT_CREDENTIAL') }}@github.com/feeds/feed-site-googleanalytics.git"
  1. passed: substituted in DBT_ENV_SECRET_GITHUB_ACCESS_TOKEN for DBT_ENV_SECRET_GIT_CREDENTIAL:
- git: "https://{{ env_var('DBT_ENV_SECRET_GITHUB_ACCESS_TOKEN') }}@github.com/feeds/feed-site-googleanalytics.git"
  1. passed: added back {{ env_var(‘DBT_GITHUB_USERNAME’) }} to the form that failed for the original post to the community:
- git: "https://{{ env_var('DBT_GITHUB_USERNAME') }}:{{ env_var('DBT_ENV_SECRET_GITHUB_ACCESS_TOKEN') }}@github.com/feeds/feed-site-googleanalytics.git"

Make sure your URL for github uses all forward slashes.

One thing I am not sure of, which may not be a thing, I could not reset the “environment” between tests and I don’t know how Cloud dbt functions behind the scene so all the success after the first success may have had state that allowed the other forms to pass.

Good luck!

1 Like
11

Thank you very much @dandoyle! This is extremely helpful! :slight_smile: :pray:

12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.