Well the player can do only so much.
A video that has been thrashed by excessive compression or downscaling is never in a million years going to look good. Sure you can tinker with options but that is like trying to path up a pair of pants that just went trough the lawnmower.
It's a sad fact that regret about compression and resolution always comes after the fact despite the fact that many warn against it.
Perhaps you could post some information about the video in question using MediainfoXP?