Annoying Pixelating/Blocking After Encoding in Handbrake

[Reckoner]

I am trying to encode my Doctor Who blu rays but am having a miserable time. I'm new to all this so hoping that someone can help. I've tried tons of different settings but it always comes out the same. There are these annoying pixels, most noticeably in the title sequence.I could post my handbrake settings but I've tried several different profiles and nothing works.

Here's the kicker...It occurs even when I play the original source mkv ripped from the blu ray in VLC but NOT in the Windows 10 video player. What's the deal with that? the source is 1080i VC-1. Deinterlacing in VLC is turned off. After encoding the pixelating is visible in any player. It's very noticeable and cringe inducing when watching on my TV via Plex so I can't ignore it. Anyone have any ideas? Screenshot posted below.

https://imgur.com/a/zOpFXsO

[poisondeathray]

Likely it's the same VC-1 decoding bug issues that have plagued libavcodec since forever.

You have to use the MS decoder to decode correctly, or purevideo (nvidia) .

There is no way that I know of configuring handbrake to do this. (since it's a decoding issue, no amount of playing with settings you can adjust in handbrake will fix it, those only affect encoding or output, not decoding)

You can use avisynth with a directshow graph to force the MS decoder, or avisynth with dgdecnv (not free) if you have a nvidia card

[sneaker]

since it's a decoding issue, no amount of playing with settings you can adjust in handbrake will fix it

HandBrake offers QuickSync decoding but I don't know if it extends to VC-1 and of course I do not know if OP has suitable hardware/software.

Btw, use of ffmpeg with hw decoding (e.g. DXVA2) is also possible.

https://forum.videohelp.com/threads/388014-Best-codec-packs-for-encoding-decoding-bluray-movies

[Reckoner]

So is there no software out there (paid or free) that I can use to properly convert interlaced VC-1 to x264? I'm not sure i'll be able to figure out this avisynth scripting stuff.

I have an i7 6700k and an AMD GPU so dgdecnv is out.

[poisondeathray]

It's just an assumption that this is the same bug (it certainly looks like it) . But it could be something else too.

There might be other software but the MS decoder and purevideo / cuvid are proven to work for that particular bug. The other software might use those as well. For example many programs actually use avisynth in the background, but how you configure the decoder or source filter might be different

But you can try handbrake with the QS decoding as sneaker suggested since you have a 6700k. VC-1 is supported for QS, but I don't know if QS will work for this particular bug (assuming that's the problem), or if handbrake has implemented support correctly

[Reckoner]

I think that's a safe assumption considering the blu ray source ripped with makemkv has the same pattern of pixelation when played in VLC but not with the Windows Movies and TV app.

[sneaker]

XMedia Recode (it's an ffmpeg based GUI) can also use the ffmpeg DXVA decoder. (Preferences->Decoder->VC-1)

[Reckoner]

Man this really sucks. I've scoured the internet and come up with absolutely nothing (at least within my comprehension)..I wonder if it's worth it to buy an old nVidia card for purevideo...assuming that would actually solve my issue.

[poisondeathray]

MS decoder - it comes with windows. But it's a bit of hassle configuring things. But at least it's free (well not really free because Windows isn't free, but you get the point....)



Some observations (at least with a different vc-1 interlaced source that exhibits similar problems using libavcodec to decode)

In handbrake with QSV, the same blocking /pixellation issues. Not sure if that means it's using SW to decode, QSV to encode, or just that it's not implemented correctly, or if QSV is affected also.

I tried xmedia recode with DXVA as sneaker suggested, and it worked (no block/pixellation artifacts as with libavcodec), but I didn't know if it was using intel or nvidia for DXVA; it didn't give you an option or GPU id control . So I disabled the Nvidia gpu in the control panel and re-ran the test forcing it to use intel - and some other types of glitches occurred (frame disorder, jerky playback, some field issues).

BUT - in potplayer with HW on ( with nvidia card disabled again) the playback is smooth no glitches. The playback info says it's using Intel - so QSV does work. That suggests it must be an implementation issue. There might be some other GUI that has it working correctly for QSV


And again, we are assuming it's the same problem...If you want to upload a small sample someone can test it for you.

[Reckoner]

I'm at work so I couldn't test QSV but just realized my work laptop has a 7th gen i7 so I did a quick encode of the episode, just the min or so that contains the biggest artifacts. Unfortunately QSV didn't make a difference. So maybe it's a different issue? I'm at a loss if it is. Why would the raw bd rip play fine in WMP but have all the same pixelation in VLC as the encoded videos.

Uploaded the sample I encoded here: https://drive.google.com/open?id=1j_NN-2WxDra4Grcj4Stq3WVP7vQ6tr9s

I'm not sure how I'd go about sharing the original file for someone to test as it's an 8gb raw bluray video.

[Reckoner]

OK yeah it's not some different unknown issue. I installed potplayer and played the blu ray rip toggling HW on and off. When it's on the video looks great, with it off I see the same pixelation.

[poisondeathray]

OK yeah it's not some different unknown issue. I installed potplayer and played the blu ray rip toggling HW on and off. When it's on the video looks great, with it off I see the same pixelation.

ok, but what was the specific decoder being used for potplayer on that test? ctrl + f1

We already know nvidia works, but my potplayer test shows that QSV should work too . So some implementation of QSV for decode in an encoding GUI should be working

I'm not sure how I'd go about sharing the original file for someone to test as it's an 8gb raw bluray video.

you can use tsmuxer to cut a transport stream, or mkvtoolnix to cut a mkv

[Reckoner]

It's definitely Intel as that's all my laptop has. Only question is why doesn't QSV make a difference in handbrake?

[poisondeathray]

Only question is why doesn't QSV make a difference in handbrake?

I don't know. But there were problems with xmediarecode too when using QSV under DXVA, but the problems were different

I don't really use HB so it might be misconfigured/user error - but I made sure tools=>preferences=> video => enable quicksync decoding was enabled. I also used QS for encoding , so full HW . But the resulting artifact pattern was the same as SW decoding. It makes me think it's not doing anything in terms of decoding

Maybe AMD/ATI works ? on your home computer ? Worth a try for xmediarecode

[Reckoner]

Did a quick test on my home PC with xmediarecode. Changed decoder to dxva and hellelujah I think it worked! No more obvious ugly pixelation all over the place. Not sure if it's Intel or AMD, haven't checked yet but assuming AMD since you mentioned Intel caused problems for you. Looking forward to fine tuning the settings and finally getting this Blu Ray added to my library. Thank you everyone for your input and for suggesting xmediarecode.

[poisondeathray]

When you get a chance , can you disable the intel gpu to confirm or verify that AMD works for these? (Or it just might be something different about my setup like drivers, and intel actually works with xmediarecode for these, or a even a slightly different problem) . It could help others in the same boat

[Reckoner]

Sure will do. I'll check back in a few hours and let you know.

[poisondeathray]

Sure will do. I'll check back in a few hours and let you know.

Thanks.

And you don't have to encode the whole video - you can cut a small piece around a known area that results in glitches with one of the tools mentioned above (tsmuxer, or mkvtoolnix)

[Reckoner]

I've gone through all the settings and can't find a way to disable it.

[poisondeathray]

I've gone through all the settings and can't find a way to disable it.

Not in the program;

In the Windows control panel; the Device Manager . The GPU's should be under something like display adapters

[Reckoner]

Well it must be the GPU. There's not even an option anymore for dxva in the decoders. Only libav.

[poisondeathray]

The question is which GPU - so if you disable one in the windows Device Manger, the other GPU must be doing the work. If there are artifacts, then you know the enabled one doesn't work. If enable the other, disable the current and there are no artifacts, then you know for sure

[Reckoner]

If I disable my AMD GPU in device manager the option for DXVA disappears. I think in order to get DXVA to work with my CPU's integrated graphics i'd have to disable my AMD GPU and plug my monitor directly into my motherboard which means I have to dig up an hdmi cable so that'll have to wait for tomorrow.

[dannyboy48888]

Yeah earlier and Dr who episodes and specials are interlaced VC1 which requires actual direct show decoding in avisynth. Filed a report 2 years ago under Linux but haven't been fixed yet. Ended up up running virtualbox and encoding just those that way. Season 7 up are avc and dont have the issue

[sneaker]

http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=144ce364cd2df752226115c243d92083291c9355

[dannyboy48888]

Just got a email from ffmpeg saying my issue was fixed 61secs ago, now for handbrake to integrate it. Should of posted the bug here lol. https://trac.ffmpeg.org/ticket/3262#comment:5

[sneaker]

I'm not sure how long it will take for it to get into HandBrake as HandBrake is based on libav, not ffmpeg directly.

[dannyboy48888]

Either way I should be able to process natively in Linux now, I'll take what I can get lol and am grateful either way

[Reckoner]

Hmm interesting. I wonder how long it will take for the fix to be implemented into HB and if I should just wait. I read that encoding with DXVA results in lower quality and bigger file sizes than. I guess I could always just re-encode them once it's working in HB.

[poisondeathray]

I read that encoding with DXVA results in lower quality and bigger file sizes than. I guess I could always just re-encode them once it's working in HB.

Not necessarily true - DXVA is just doing the decoding instead of decoder A,B, or C , or through another system like SW, CUVID, VDPAU(for linux) etc... In theory the same frames should be output to the encoder. If you used the same encoder, same settings, theoretically you'd get identical output

However, in my experience, GPU decoding though any system can be flaky. In might be "good enough" for playback in a media player, but it can pose problems sometimes for encoding. Sometimes it works ok (same result as you'd expect) but sometimes there are frame discrepancies such as dropped frames, sometimes duplicated frames. Sometimes that causes sync issues. Only when it's indexed does it ensure it to work or increase the reliability substantially