I am not sure why this file cannot be found.
I am using Dagster to run the DBT Docs Generate command, upload those files to S3 so that they can be deployed via CI/CD to an ngnix server.
The documentation will sometimes load after a fresh deploy, but quickly will throw the above error. I can confirm, though, that the debug and docs generate CLI commands produce no error(s), and all the files (including the catalog.json) are present on the ngninx server.
I am not sure how to debug this weird issue. Thanks!
I tried running the docs generate command with the --no-compile flag, but it produces the same result.
I haven’t deployed the docs site anywhere other than dbt Cloud, but a few guesses:
I assume that the directory structure is the same on your S3 bucket as when you test locally? (i.e. the catalog file isn’t in an unexpected subdirectory)
Does your S3 bucket have any sort of permissioning/CloudFront signed URL requirements?
Can you access the catalog file by pasting its URL into the address bar, or accessing it with curl/wget etc?
Have you tried manually generating a manifest, catalog, etc and copying those files onto your S3 bucket and trying to serve them?
Are there any other errors in your devtools console, e.g. a 404 showing the path it tried to access?
Thanks for responding. Let me say, the S3 bucket is just an intermediary. I push the files to S3 first, but use Ci/CD to pull those files down from S3 onto a deployed webserver.
I assume that the directory structure is the same on your S3 bucket as when you test locally? (i.e. the catalog file isn’t in an unexpected subdirectory)
Correct. All files live in a single folder without any structure.
Does your S3 bucket have any sort of permissioning/CloudFront signed URL requirements?
N/A. Files are uploaded to S3 but downloaded to the webserver that serves these files.
Can you access the catalog file by pasting its URL into the address bar, or accessing it with curl/wget etc?
I can. If I tack catalog.json on the end of the URL, the JSON loads in the browser.
Have you tried manually generating a manifest, catalog, etc and copying those files onto your S3 bucket and trying to serve them?
Curious what you mean by manually? Currently, I’m using Dagster to issue the CLI command dbt docs generate.
Are there any other errors in your devtools console, e.g. a 404 showing the path it tried to access?
No 404s, just the error in the console which reflects the above error message. (503 - Service Unavailable)
One thing I’m curious is the suffix on the filename shown in the message. i.e. cb=1669728929765
I meant doing the entire flow without involving any other tools (Dagster, your CI/CD tool), i.e.
Run dbt docs generate locally
Copy the files to the ngnix server
Try loading the files
If that works, then something somewhere in the automated pipeline is misbehaving. If it still fails, then I would point the finger at your ngnix server.
The fact that you’re getting a 503 error does imply that it’s an oddity of the webserver - I missed that in your original screenshot and assumed it was going to be a 4XX error. If it’s a server issue then I am going to be doubly unable to help sorry!
I think that is a cachebuster to ensure that changes to the json files are picked up when you refresh the page. That looks like a Unix timestamp in milliseconds, and maps to 13 hours ago which is when you posted.
Does the webserver need the project_dir and profile to be present? I only have the generated target folder on the webserver. Does the docs need to have a connection to the database, as well?
