hey everyone, hope you guys are having fun.
in our company we are planning to create a medallion based architecture, we’ve been storing the bronze layer in iceberg for a while now and we are planning to store our silver layer on minio as well, since we are in a big data spectrum and we can’t really afford to move this layer to starrocks.
I was wondering how we should move forward with the data modeling, we have some data models that are providing services to other teams and they are mostly created using pure spark.
we considered using starrocks for gold layer but the main problem we are facing right now is that how could we make this data modeling happen with the help of dbt and starrocks.
There is a dbt-starrocks package, I’ve used it and I love it, although I couldn’t get it working when it came down to a solution when we got to the incremental microbatch strategy, unfortunately the compiled sql files didn’t contain any bounds for the data, everytime we ran it was a full refresh and it brought all of the data in.
Also it was a bit problematic for us to get it back and store it again in iceberg, by default it stores the new data in starrocks, any idea for a workaround on these problems?