IMHO all applications should default to /home, as I have never once been looking to open a recently used file, but they don't and GNOME inpaticular seems to have every program ever start with the recently used list. 

When on the Recently Used option, you can hit / then start typing /path/to/filename and hit enter to select the file. From everywhere else you can just start typing the /path/to/filename and hit enter. The button is there, but the only time I could see clicking it would be on a touchscreen?

I'm aware that this probably isn't an XFCE-specific problem.

exmaple screen shot (lacking a file path text box): http://bcaptain.net/filechooser.png