but graphics library affects a lot of other components including desktop. Without sources it is in my view impossible to not have unwanted side effects.
And you have lots of software using it.
To do a clean-room rewrite and modernization of graphics library not breaking lots of software and components sounds to me like a huge and complicated project