Really odd issue playing some x264

[NotSoEasy]

Last night I noticed out of the Blue that some of my video files are playing oddly.
The issue is noticeable and I know it was never there before.
Only way to describe this is by imagine playing a video in your favorite software and slowing down the video. Has that "jerky" look to it. Now the really odd part is if I make the player go fullscreen, the issue goes away.
Again. I never had this playback issue before yesterday. I haven't installed anything that would hinder playback.

I can play any 1080 .ts file without issue. Which is why I'm confused. The video that are doing this are 720p

Windows 7 Ultimate SP1 - 64-bit
AMD Phenom II X4 2.8 GHz - Quad Core
ATI Radeon HD 4200 - 512MB Share Memory (on-board)
(2) Crucial 2GB, 240-pin DIMM, DDR2 PC2-6400
ASUS M4A785-M AMD 785G Socket AM2+

[mh2360]

Windows 7's in-built decoder uses GPU hardware acceleration. Maybe an updated graphics driver is causing the issues? Do you have Windows updates set to install automatically?

[NotSoEasy]

I do not have auto updates. I pick and choose.

[Bjs]

What is 720p video ? ... use media info for more info ... mention video container involved.

Considering the odds ... do a full defrag on the drive containing the files ... test again

[NotSoEasy]

MKV containers and the drive is 0% fragmented.
Guess it doesn't really matter though anyway since I stream to WDTV and plays fine or just load onto HDD connected to WDTV
Just thought it was odd that all of a sudden I had playback problems.

[NotSoEasy]

This is still bugging me ... OCD I guess ...
I can play 1080i/1080p with no issues but 720p and it's like the video is set to play at a slower speed. That is until I double-click to go fullscreen and the problem goes away. Makes no sense.
Example file would be:

Format : Matroska
Format version : Version 2
File size : 1.70 GiB
Duration : 1h 33mn
Overall bit rate : 2 590 Kbps
Encoded date : UTC 2012-02-28 02:26:53
Writing application : mkvmerge v4.7.0 ('Just Like You Imagined') built on Apr 21 2011 01:13:14
Writing library : libebml v1.2.0 + libmatroska v1.1.0

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings, CABAC : Yes
Format settings, ReFrames : 3 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 1h 33mn
Bit rate mode : Variable
Bit rate : 2 141 Kbps
Maximum bit rate : 40.0 Mbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.097
Stream size : 1.37 GiB (81%)
Writing library : x264 core 120 r2164 da19765
Encoding settings : cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=9 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / slices=4 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=0 / keyint=23 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc_lookahead=23 / rc=2pass / mbtree=1 / bitrate=2141 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=40000 / vbv_bufsize=30000 / nal_hrd=vbr / ip_ratio=1.40 / aq=1:1.00

Audio

Yet I can play this 1080 file with no issues:

Format : MPEG-TS
File size : 12.2 GiB
Duration : 1h 35mn
Overall bit rate : 18.3 Mbps
Maximum Overall bit rate : 35.5 Mbps

Video
ID : 4113 (0x1011)
Menu ID : 1 (0x1)
Format : MPEG Video
Format version : Version 2
Format profile : Main@High
Format settings, BVOP : Yes
Format settings, Matrix : Custom
Format settings, GOP : M=3, N=15
Codec ID : 2
Duration : 1h 35mn
Bit rate mode : Variable
Bit rate : 17.1 Mbps
Maximum bit rate : 80.0 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 29.970 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.274
Stream size : 11.3 GiB (93%)

[hello_hello]

Which media player are you using? Maybe try a different one such as MPC-HC to see if the problem persists. If so, go into MPC-HC's options and disable the internal DXVA filters (decoders) to get the CPU to do the decoding rather than the GUP and see if that makes a difference.

Isn't the second video you posted mpeg2? The GPU may be used for decoding the first video, but not the second.

[NotSoEasy]

I use MPC-HC as well as VLC and same problem. Will check options though.
First text is the 720p MKV example that has issues. The second text is MPEG2 (TS file)

Unchecking H264/AVC DXVA filter worked great for the MKV 720p
I also unchecked MPEG-2 video DXVA and saw a small change (for the better) in playing HD files.

[hello_hello]

Which types of video are hardware decoded probably depends on your video card. For me it's h264/AVC only. When playing a video MPC-HC should display "Playing DXVA" (or something like that) in the status bar if it's using hardware decoding.

If unchecking DXVA decoders fixed the problem, then it points to your video card or it's drivers being the cause, although I'd try at least re-setting MPC-HC's renderer settings back to default. You can do it via the right click menu. Maybe even try a different renderer. Although given the same problem occurs using VLC......
You could try updating or uninstalling/re-installing your video card drivers, although with Windows there's probably other steps in the decoding chain which can cause problems I know nothing about. Unless of course you're happy just to decode with the CPU instead of the video card.

I gave up trying to work things like this out a long time ago. These days after setting up a PC I image it, and when oddities start to occur if they're not something I can easily fix I just restore the image. It tends to be less time consuming. Not much help for you, I know....

One other thought, although it's a long shot..... but try using MKVToolNix to resave one of the problem MKVs without any other streams. No audio, subtitles etc, just to see if when playing "video only" you still have the same problem. I don't see why you shouldn't but then I don't see why you should be having this problem at all....

[hello_hello]

PS Go into your ATI control panel (or whatever it's called) and if there's any "enhancements" enabled, try disabling all of them. I'm an Nvidia guy but if memory serves me correctly, ATI tends to enable that sort of thing by default. Once again, I've no idea why normal playback should be different to full screen playback so I'm just taking a shot in the dark, but it might be worth a look.

Do you have an old DXVA capable video card lying around you could stick in the PC to try?

[jagabo]

If it's not a graphics driver issue there's a miscommunication between the MKV reader/splitter, and h.264 decoder. Try using different combinations.

[NotSoEasy]

Well my video is on-board as I stated in first post. I am, from what I can tell, up to date on drivers and software.
Only 512 shared from my 4GB RAM but it is obviously enough as I just encountered this issue.
It's not a huge deal as I do not watch video (movies) on my PC. I just check to see that everything is good before either streaming or placing on external HDD for my WDTV
It was just bugging me that it all of a sudden happened to 720p and or MKV only.
I have played with 720p Xvid as well as 1080i/p and no issues. I would think if 1080 plays without issue that 720 would also but ...

I did make the changes though in MPC-HC (DXVA) and that seemed to have worked well.

[hello_hello]

It's very likely that the video card only decodes h264/AVC and the 1080i/p videos you're testing are, I'd theorize, mpeg2. So they're being decoded by the CPU, as are the 720p Xvid encodes. Even with DXVA enabled it'll only be used for the formats your video card supports. Now you've disabled DXVA in MPC-HC it's all being decoded by the CPU.

You could try installing the Haali Media Splitter and disabling MPC-HC's internal MKV splitter to see if anything changes, although the fact you have the same problem using VLC does seem to point to it being some sort of video card/driver/Windows issue.
Have you tried changing the renderer used by MPC-HC? Maybe it's using the same one as VLC and it's causing the problem.