I am trying to load seed file to data warehouse. But it took 67 minutes to load,what can we do to decrease the this time.
My file size is 3gb.
I’ve seen a rule of thumb here somewhere that said if a seed is more than one or a couple MB it should not be a seed.
You can also refer to this FAQ
Seeds should not be used to load raw data (for example, large CSV exports from a production database).
Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees.
Loading CSVs using dbt’s seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse.