There is a need for an X environment for some builds to succeed.
The nature of the X server is irrelevant.
For this, a script could be devised to launch an Xvfb display with as much
safety as possible.
There are two possible modes of operation for such a script.
1. Run a single command under the Xvfb environment, then shutdown the server.
2. Launch an Xvfb server and dump environment variable settings for DISPLAY,
XAUTHORITY and a variable for Xvfb pid or pidfile in form of shell commands on
standard output. Usage for this is similar to that of ssh-agent:
eval `xvfb-start`
# do stuff
... 
xvfb-shutdown  # this could be another script that picks up the variables

start-stop-daemon could be helpful in starting/stopping the Xvfb server reliably.