ok, here's the deal:

assume i have a 1920x800 mkv, encoded with x264, ac3 audio, with a frame rate of 23.976 fps(side note: every editing, encoding and playback software sees this frame rate, but kmplayer adds the caveat that the original frame rate was 24 fps) and it plays back smooth as butter when viewed on my computer.

second side note: i also have the movie on a legit blu-ray, but i can only view it on my PS3 (don't have a BD-Rom), but needless to say that the blu-ray has no flaws, oh the movie in question is Batman Begins.

continuing: when i try and transcode the above mkv to mpeg-2, h264 or wmv AND resize it to another size (let's say 1280x720, 960x540, 1368x768, 1024x728 <--let's not worry about aspect ratio at the moment) and i set the frame rate to 23.976, the outputted file plays back in one of a number of ways: it plays back in a 'slow motion', more frequently it plays back in a very jerky manner (<--almost as if frames have been decimated) or it will play normally for most of the movie but will have a part that has the above characteristics. the above holds true primarily for mkv's encoded with x264, for 1920x800 that were encoded using a different h264 codec, such as apple's h264, elecard's, nero's avc or more commonly main concept's h264, there are no problems of the sort. at times the problem has also shown it's face (primarily the slow motion) when the source file was encoded with microsoft's vc-1.

the problem seems to be independent of the transcoder i use as i have found similar behavior with the following software:

main concept's H264 encoder
main concept's reference
tmpgenc xpress 4.0
tmpgenc authoring works 4
avidemux
elecards converter studio
ffmpeg
procoder 2 and 3

and a few others.

the funny thing is that the above problem doesn't show it's face if i transcode to divx (i only tried it with tmpgenc express 4) or if i use nero for avc encoding or if i use tmpgenc express 4/procoder 3 to convert to apple's H264).

one more thing, i transcoded the above file (it happens with nearly all files with the same characteristics) using main concept's H264 encoder with constant quality set to best and 23.976 fps and the resulting file was under 2 gigs (the original was over 8 gigs and the bitrate was under 2 Mb/s, however the quality of each individual frame was fantastic (it rivaled the commercial blu-ray of the same movie) however as i already stated the playback was jerky, as if frames were missing. when i set the target frame rate to 29.97, the file size of the transcoded file ballooned to about 7 gigs, and the playback was smoother but it almost looked like there were duplicate frames, the bitrate had also ballooned and the quality had tanked.

i was thinking it may be an issue with pulldown, but somehow i don't think so.

anyone have any idea what is going on?