Hard to say without knowing more about your set up.
How is your PS3 connected to the TV ?
What are your video settings in the PS3 ?
Have you tried Multi-AVCHD ?
I have my PS3 connected via HDMI to a 1080p HDTV. I use Multi-AVCHD to convert/author 720p mkv files and burn with Imgburn as UDF 2.50. These play back happily at full res.