This has been problematic for me after upgrading from 18.04 to 20.04. Every time I apt update cuda or some nvidia package my x server fails to start for several different reasons.

Just today I've already spent 30 minutes trying to start x with this latest cuda update. Too bad I can't switch back to the open source nouveau driver.