But X cannot live on its own: it needs a Window Manager and a widget toolkit to do anything. More millions of lines of code, layered on top of an already overweight system. GNOME, KDE, even Motif exact a huge performance hit and add further complexity. Worse yet, no one in the Unix community seems to be able to agree on any kind of GUI standard, so you have an explosion of window managers, desktops, file managers, and widget toolkits which are mostly inoperable with each other.
Dealing with the which Window Manger, how to configure, and which widget set to write to is the biggest pain in the butt of Linux/UNIX. So much duplication of effort to achieve the same end result.