The BSDs ditched them, too. NetBSD was tne last, but they finally made the move.
With hindsight, I would say that people ditched XFree more on the unsatisfaction of the development progress. Only a few had commit access and they were quite slow at adding new features that people wanted. For those who didn't knew, on-the-fly resolution switching was added something like three years ago, maybe four. Furthermore, few people were familiar with the build tools used (imake) and development was tedious because it's a huge monolithic tree. X.Org 7.0 is going to fix these two issues, BTW.
There are other reasons. X-Windows even covers a whole
chapter of the
UNIX Hater book...