How to decrease ref-frames in mkv-container? 1080p/720p

[k1llerunit]

hi,

i just purchased a WD TV HD Media Player and found out that it has problems with files that reach a higher value as 8294400 when you multiplicate "heigth", "width" and the number of ref-frames. the number of ref-frames should be between 1-5 but some of my files have 8,9 or even 13. i either need to decrease the resolution from 1080p to 720p or decrease the number of ref-frames to a maximum of 5.
can somebody tell me what programs i need for that. i can handle mkvtoolnix. i also have MeGUI, AVIsynth and tsmuxergui but i actually am not able to deal with them
may someone help me?

thanks

EDIT: i already have the *.h264-stream from the mkv-container!

[Baldrick]

I use xvid4psp. Load the mkv, choose the mkv profile, change the resolution/size to 720p.

The reference frames is default 3 I think, click on the E-icon next to the video encoding to adjust it.