How to encode 264 without aliasing (jaggies)

[vertigoanduminas]

I have successfully backed up several blu-rays to my hard drive and demuxed the main streams, but I'm trying to compress them for space.

I've somewhat easily compressed the .vc1 source of "Serenity" using RipBot264 from 18GB to 8GB with 2 pass at ~9305kbps (VBR). However, it introduces aliasing (jaggies) in scenes with detailed lines or textures in motion. This is true even with every quality setting at max, so I'm wondering if there's a fundamental reason.

Please look closely at the windows and upper nose of the large ship in the lower left of the frame. It's hard to see without flipping back and forth, but the windows are dimmer in the encodes because of the aliasing. In motion they shimmer.

To illustrate this specifically, here is the original (20.5mbps):


Now after 2 pass encode (9.3mbps):



Just as a test, I ran this 8s clip at full bore (17mbps):
--level 5.1 --aud --nal-hrd --vbv-bufsize 25000 --vbv-maxrate 25000 --filter 6,6 --ref 16 --bframes 16 --b-adapt 2 --subme 10 --aq-mode 1 --trellis 2 --partitions all --me tesa


Note this has nothing to do with de-interlacing (in case someone jumps to conclusions) since the original is 1080p vc1.
Any ideas?

[poisondeathray]

Is the vc1 stream telecined or native 24p ?

What was the .avs script used? Look at the log file. What vc-1 decoder was used?

If you preview the .avs script on that same frame , are the artifacts visibile?

[vertigoanduminas]

The original is a .mkv repack (mkvmerge) of the .vc1 from the .m2ts stream:
Video #0 : VC-1 at 20.1 Mbps
Aspect : 1920 x 1080 (1.778) at 23.976 fps

avs script:
video=DirectShowSource("G:\_RIPPING\serenityvc1.mk v",audio=false).ConvertToYV12().Trim(121846,122037 )
return video

264 script:
--level 5.1 --aud --nal-hrd --vbv-bufsize 25000 --vbv-maxrate 25000 --filter 6,6 --ref 16 --bframes 16 --b-adapt 2 --subme 10 --aq-mode 1 --trellis 2 --partitions all --me tesa

[poisondeathray]

That can't be the whole script. Your images are cropped, but your script that you posted doesn't even show crop().

Did you preview the script on that frame?

What directshow decoder are you using. Open the mkv in graphstudio.

If you can't figure it out, post a native sample of the source to a free hosting site (e.g. mediafire.com)

[vertigoanduminas]

The script is from the 3rd image (the full bore recode). I is as complete as ripbot will allow one to see. I throw the two samples on rapidshare, but I can use an alternate if you prefer.

[poisondeathray]

If you scroll down the log, there should be more of the script. Ripbot divides up the script into sections

Just the native sample of the source (.m2ts) you can cut with tsmuxer for example, on that section

I would prefer mediafire.com

[jagabo]

Note this has nothing to do with de-interlacing (in case someone jumps to conclusions) since the original is 1080p vc1.

You're wrong. Your video was obviously drop-field deinterlaced somewhere along the line. your source image after discard field and resize in VirtualDub:



Note the exact same aliasing artifacts.

[poisondeathray]

post the *full* log file and .avs and it will tell you what commands were used, whether or not you deinterlaced. Also there could be post processing in the filter chain, depending on what directshow filters you are using - for example you might have enabled deinterlacing in ffdshow and forgot to turn it off. Previewing the .avs will quickly help to show where you are going wrong.

also --me tesa is pretty much useless and is about 4x slower. Your deblock settings of -6,-6 are pretty bizzare as well.

[vertigoanduminas]

I'm game, but how is this possible? vc1 streams from blu ray are always progressive right? The "line we're talking about has about 2 stops here: m2ts stream to .264. The .mkv packaging doesn't change the source whatsoever.

(I'm working on re cutting the m2ts)

[jagabo]

I'm game, but how is this possible? vc1 streams from blu ray are always progressive right?

You can run a deinterlacing algorithm on a progressive source. It's just the wrong thing to do.

[vertigoanduminas]

#1 Remember I said I simply turned everything on for the second run, to make sure I wasn't just short-sheeting my encode.

#2 I realize de-interlacing can ruin progressive video, I just didn't expect it to be enabled without me asking for it.

The log file in the main ripbot folder:
The syntax of the command is incorrect.

G:\>"E:\=DOWNLOADS\PROGRAMS\MEDIA\RIP\RIPBOT\tools\bepipe\bepipe.exe" --script "import(^G:\temp\RipBot264temp\job1\job1.avs^) " |

G:\>"E:\=DOWNLOADS\PROGRAMS\MEDIA\RIP\RIPBOT\tools \avs2yuv\pipebuf.exe" "E:\=DOWNLOADS\PROGRAMS\MEDIA\RIP\RIPBOT\tools\avs 2yuv\avs2yuv.exe" "G:\temp\RipBot264temp\job1\job1.avs" -raw - : "E:\=DOWNLOADS\PROGRAMS\MEDIA\RIP\RIPBOT\tools\x26 4\x264_x64.exe" --crf 18 --stats "G:\temp\RipBot264temp\job1\job1.stats" --fps 24000/1001 --min-keyint 24 --keyint 240 --frames 192 --sar 1:1 --level 5.1 --aud --nal-hrd --vbv-bufsize 25000 --vbv-maxrate 25000 --filter 6,6 --ref 16 --bframes 16 --b-adapt 2 --subme 10 --aq-mode 1 --trellis 2 --partitions all --me tesa --output "G:\temp\RipBot264temp\video.264" - 1920x1080 : 2

x264 [info]: 1920x1080 @ 23.98 fps

x264 [info]: using SAR=1/1

x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64

x264 [info]: profile High, level 5.1
G:\temp\RipBot264temp\job1\job1.avs: 1920x1080, 10000000/417083 fps, 192 frames

x264 [info]: frame I:3 Avg QP:17.37 size:221419

x264 [info]: frame P:82 Avg QP:20.17 size:128643

x264 [info]: frame B:107 Avg QP:22.67 size: 57374

x264 [info]: consecutive B-frames: 1.1% 65.6% 19.0% 6.3% 7.9% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

x264 [info]: mb I I16..4: 26.2% 65.3% 8.5%

x264 [info]: mb P I16..4: 0.8% 9.3% 0.2% P16..4: 40.5% 18.5% 7.2% 0.3% 0.1% skip:23.1%

x264 [info]: mb B I16..4: 0.1% 1.5% 0.0% B16..8: 49.7% 0.8% 1.5% direct:10.4% skip:35.9% L0:38.3% L1:48.8% BI:13.0%

x264 [info]: 8x8 transform intra:85.4% inter:74.3%

x264 [info]: coded y,uvDC,uvAC intra: 88.2% 79.2% 54.9% inter: 41.0% 20.8% 4.8%

x264 [info]: i16 v,h,dc,p: 84% 9% 4% 3%

x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 7% 18% 10% 13% 11% 12% 10% 14%

x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 11% 5% 9% 14% 12% 14% 11% 18%

x264 [info]: Weighted P-Frames: Y:0.0%

x264 [info]: ref P L0: 70.9% 7.8% 11.2% 2.8% 1.9% 1.3% 1.1% 0.6% 0.5% 0.4% 0.4% 0.4% 0.2% 0.2% 0.2% 0.1%

x264 [info]: ref B L0: 86.4% 5.8% 2.8% 1.2% 0.8% 0.7% 0.6% 0.3% 0.3% 0.2% 0.3% 0.2% 0.1% 0.1% 0.1%

x264 [info]: kb/s:17334.63

encoded 192 frames, 0.63 fps, 17334.63 kb/s

G:\>"E:\=DOWNLOADS\PROGRAMS\MEDIA\RIP\RIPBOT\tools \mkvtoolnix\mkvmerge.exe" -o "J:\CQ18extreme.mkv" --title "CQ18extreme" --default-duration 0:24000/1001fps -d 0 -A -S "G:\temp\RipBot264temp\video.264"

mkvmerge v3.0.0 ('Hang up your Hang-Ups') built on Dec 12 2009 15:20:35

'G:\temp\RipBot264temp\video.264': Using the AVC/h.264 ES demultiplexer.

'G:\temp\RipBot264temp\video.264' track 0: Extracted the aspect ratio information from the MPEG-4 layer 10 (AVC) video data and set the display dimensions to 1920/1080.

'G:\temp\RipBot264temp\video.264' track 0: Using the MPEG-4 part 10 ES video output module.

The file 'J:\CQ18extreme.mkv' has been opened for writing.

The cue entries (the index) are being written...

Muxing took 0 seconds.

-------------------------

Elapsed Time: 00h:05m:08s

This is job1.avs in the "Temp" folder:
#MT

#VideoSource

video=DirectShowSource("G:\_RIPPING\serenityvc1.mk v",audio=false).ConvertToYV12().Trim(121846,122037 )

#Deinterlace

#Decimate

#Crop

#Resize

#Colors

#Denoise

#Subtitles

#AudioSource

#DownMix

#Delay

#Tempo

#Normalize

#AudioDub

#Triming

#AVSameLength

#ColorSpace

return video

Here are the first and third scene cuts. The first is .mkv because tsmuxer wouldn't play nice and after 2 hours I could only get 0kb files out of it (trying to scene cut). I do promise you the .vc1 is untouched. The other file is the one encoded with max settings.

http://www.mediafire.com/file/omtwnny0gfn/Serenity scene cut - CQ18 RipBot x264 Maxed.mkv
http://www.mediafire.com/file/zvdozzjimdj/Serenity scene cut - vc1 repack.mkv

[poisondeathray]

Your mkv sample works fine for me , no aliasing , using default settings. Ill try again with your weird settings but I doubt it will make a difference. Again, an easy way to figure out what is going wrong is to preview your .avs script. e.g. save the script and open it up in vdub, or avsp. If there is aliasing artifacts there, you know it's more likely a decoder issue (ie. before the frame gets fed to the encoder) , and likely a configuration issue on your system

You must have a filter somewhere in your filter chain doing that. Again, use graphstudio, and examine the filters. e.g if you are using ffdshow for wvc1 decoding, make sure the deinterlacing is disabled.

If you want to use a different source filter, and avoid the directshow issues, you could use ffmpegsource2 instead

[vertigoanduminas]

Unbelievable. I thought it was just my encode, since it looked the same in zoomplayer with every video render.

However, the problem was it defaults to "use ffdshow to control video de-interlacing". Even though it was unchecked in ffdshow's control panel, zoomplayer was still getting some action when it shouldn't have.

Since I never watch interlaced video, I don't need it. Thanks for the help.

Hopefully anyone else with this problem will google "jaggies" or whatnot!

EDIT: One question though, what is wrong with my encodes that is telling the player to use de-interlace? It just unnerves me to think I'm ending up with a broken format...

[jagabo]

However, the problem was it defaults to "use ffdshow to control video de-interlacing". Even though it was unchecked in ffdshow's control panel, zoomplayer was still getting some action when it shouldn't have.

Keep in mind there are two video subsystems in Windows, DirectShow and VFW. So there are two sections in ffdshow. Be sure to turn off the deinterlacer in both of them (Video Decoder Configuration, VFW Configuration).

[vertigoanduminas]

I see that, but they were both off, so I dunno.

[jagabo]

Is ZoomPlayer using an internal version of ffdshow or a DirectShow installed version? Do you see the ffdshow icons in the system tray while playing?

[vertigoanduminas]

Apparently zoomplayer has ffdshow controls within it's menus, but it can manipulate the filter without altering your system's configuration. So for me, at least, it comes down to that one check box in the Playback > Video page under "Video Post-Processing".

The only icon I see when playing is Haali Media Splitter showing which streams are selected.

EDIT: So apparently the entire issue is that the player correctly detected a VC1 stream (the original), but mistakenly de-interlaced a progressive AVC (my encodes), right?

[jagabo]

So apparently the entire issue is that the player correctly detected a VC1 stream (the original), but mistakenly de-interlaced a progressive AVC (my encodes), right?

Apparently. Both files look the same on my computer.

[poisondeathray]

Apparently. Both files look the same on my computer.

The links he posted above are the same file, same url

[jagabo]

Apparently. Both files look the same on my computer.

The links he posted above are the same file, same url

LOL. I didn't notice that.

[FTW]

Hopefully anyone else with this problem will google "jaggies" or whatnot!

I recently discovered the same issue with a backup encode I did, those little jaggie lines are annoying as all hell.
So I double checked the settings in ffdshow (VFW config & video decoder config) to ensure that deinterlace wasn't activated (which it wasn't).
I use MPCHC for my video playback, so I went into View > Options > Playback > Output > Directshow Video, then changed the output from VMR7 (renderless) which was the default setting, for me anyway, to Haali Renderer. Problem solved!
Sweet!!

[vertigoanduminas]

Sorry about the links, but thanks for the solution!