I've done a fair bit of casual off-road on my track bike, but it didn't really seem like abuse, in fact, I think a fixed gear road set-up is pretty good for going off the beaten track. A couple of years ago I did a good chunk of the Pilgrim's way, which involved a fair few bridleways:
It was pretty good.
I've been down a few 'Sandy lanes' which never seem to end well (unsurprisingy, almost always in sand or pikey flytips); a particularly memorable one was last December, in Norfolk, walking across unrideable sand farm tracks in the dark, in the rain, passing marshals, etc.
I found some pretty good territory whilst scouting out the M25 route, lots of poo-filled cul-de-sacs-cum-bridleways-cum-naitonal-cycle-network lanes whichare great forfilth.
Sorry Corlis :-s