The problem I’m having
After upgrading from dbt-core v1.2.3 to v1.4.6 our orchestration (Airflow) and CICD pipelines (Gitlab) began randomly (not every execution) crashing. The crashes are happening shortly after the command is given (run, test, docs) but only after the parsing has started. The crash will result in an error similar to:
[2023-07-07 10:24:06,421] INFO - b'/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown\n'
[2023-07-07 10:24:06,421] INFO - b" warnings.warn('resource_tracker: There appear to be %d '\n"
[2023-07-07 10:24:09,004] INFO - b'/bin/bash: line 28: 125 Segmentation fault (core dumped) dbt test --profiles-dir profile --target prod --models sources.omamori+ --exclude edm_snapshot\n'
But those errors may just be a result of the crash and not the cause. Does any one have any insight as to what may be the casue of these crashes? We have had to revert back to v1.2.3 to attempt to regain stability in out production environment.
dbt Specs
dbt-snowflake = "1.4.2"
dbt-core= "1.4.6"
Cluster Specs
The dbt task via Airflow runs on below spec of OS. machine type : n1-highmem-8
- Kernel version - 5.10.162+
- OS image - Container-Optimized OS from Google
- Container runtime version - containerd://1.6.18
- kubelet version - v1.24.12-gke.500
- kube-proxy version - v1.24.12-gke.500
- CPU :- 8 CPU
- RAM Memory :- 54.76 GB
This is spec of 1 machine and we have 5 machine running in parallel where dbt task can run.
Container-Optimized OS from Google is Operating system from google to run the containers in GKE cluster.
For CI Job machine type : n1-standard-2
- Kernel version - 5.10.162+
- OS image - Container-Optimized OS from Google
- Container runtime version - containerd://1.6.9
- kubelet version - v1.24.10-gke.1200
- kube-proxy version - v1.24.10-gke.1200
- CPU :- 2
- RAM Memory :- 7.82 GB
This is single node GKE Cluster