dbt 1.6 is out today and I think this was fixed by Do not rewrite `manifest.json` during `docs serve` command by jtcohen6 · Pull Request #7554 · dbt-labs/dbt-core · GitHub - would you mind giving it a go?
Edit: it should actually be available from 1.5.1 and onwards!