VideoHelp Forum
+ Reply to Thread
Results 1 to 18 of 18
Thread
  1. Hi guys,

    I am new to this forum and also new to the art of encoding. I am having a severe problem with banding. I know I am not the first one and I have already read lots of posts about this and how to fight it. I would like your help to see if I can add anything to my workflow to improve the video quality or if I am doing something wrong here.

    The source file is a 10bit avi v210 file with no visible banding (the sequence is 04min 44s long and has size of 46Gb) but Iíll show only the most problematic scene. Hereís a screenshot showing how it looks like in the 10bit source file (the picture has been stretched so it can exagerate the fading):

    http://i.imgur.com/NQf0g7F.jpg

    The first thing I made with this file was making it 8bit with dither loading a .avs file into Virtualdub using the following code:

    LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
    FFvideosource("D:\Arq\encoding\timelapse 30fps master 10bit.avi",enable10bithack=true)
    f3kdb_dither()

    Then, the output is a 8bit avi (52Gb size) with a zero (or very very little and way acceptable) banding visible as you can see in this screenshot:

    http://i.imgur.com/DlTdvae.jpg

    So I used HandBrake 0.9.9.5530 - 64 bit version to encode a .mp4 file via x264. One important thing to note is: I am aiming the best quality possible that I can output for a 5Gb file. 5Gb is the largest size Vimeo will accept my video so that`s my limit. I don`t mind if the file takes 2 weeks to encode to gain just one pixel better. The video will be online for years to come so encoding time is definetly out of question. The important thing is the quality above everything that can fit this size.

    Settings:

    Picture - Anamorphic strict (1920x1080), custom croppping (0 for top, left, right and bottom).
    Filters - Detelecine default, Ddecomb default. Deinterlace, Denoise and Deblock Off.
    Video - Codec x264. Framerate constant (30fps in the source and the output). 2-pass encoding with average bitrate 145000 (that will give me a file around 4.9Gb).
    Advanced - ref=16:bframes=16:b-adapt=2:direct=auto:me=tesaubme=11:merange=64:an alyse=all:trellis=2:deblock=-1,-1

    Here`s a screenshot of the advanced tab on Handbrake:

    http://i.imgur.com/cDYsuLD.jpg

    This settings takes around 12h to encode. Fine for me.
    This is the output as seen from virtualdub:

    http://i.imgur.com/JJ6dU6U.jpg

    And this is the same file but played with Media Classic Player:

    http://i.imgur.com/VHBlX0N.jpg

    When I play the 8bit dithered uncompressed file on Media Player Classic, the output range is 0-255. x264 changed the output it to 16-235.
    Question #1: Is this normal?

    I had to set the output range on MPC to 16-235 to match virtualdub colors. But on virtualdub absolute black is darker.
    Question #2: Is Virtualdub transforming my video file in 0-255 to preview it?

    Question #3: Does Vimeo do the same, i.e. expect a 16-235 and transform into 0-235? I had problems with gamma shift before, when uploading to vimeo, so it`s an important thing to consider. Is x264 outputting a 16-235 file the correct workflow for vimeo? (PS. I have read vimeo guidelines, and there`s no metion for the output range)

    The MPC screenshot above is just an extra: all of the screenshots below will be made using Virtualdub so we can compare with the 10bit file.

    As you can see, obvious and strong banding after x264, even with dither. The problem seem to be on x264. So I started reading on which values to tweak to try to improve this. I have read a lot and have made a few experiments.

    Firstly, I changed AQ Streght to 0.8 and add the no-mbtree parameter. Here`s a sreenshot on the output file with these changes:

    http://i.imgur.com/0dC6Kzy.jpg

    Not a change. And then I also set deblock to 3,3, used the No-DCT-Decimate with the AQ strenght 0,8 and no-mbtree. The result is:

    http://i.imgur.com/2zGc7WZ.jpg

    Not a change also.

    Question #4: What I am doing wrong here? Anyhow I can tweak x264 to make this fading better with less banding?

    I don`t know what else I can do to reduce this. I don`t think adding noise to the source file will help, since the 8bit source has already no banding.
    Quote Quote  
  2. To decrease banding you want to increase aq-strength, not decrease it. Try around 1.8 or 1.9.

    The normal range of RGB video is 0-255. The normal range of YUV video is 16-235 for Y, 16-240 for U and V. Most programs automatically adjust the range when converting between the two. Keep in mind that what your monitor displays is RGB. So when watching YUV video, somewhere along the line it's converted to RGB.
    Quote Quote  
  3. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    When I play the 8bit dithered uncompressed file on Media Player Classic, the output range is 0-255. x264 changed the output it to 16-235.
    Question #1: Is this normal?
    Are you referring to YUV or RGB ?

    x264 doesn't change anything, unless you tell it to. This is more likely a playback configuration issue

    Question #2: Is Virtualdub transforming my video file in 0-255 to preview it?
    Yes, YUV is converted to RGB 0-255 with a Rec601 matrix.


    Question #3: Does Vimeo do the same, i.e. expect a 16-235 and transform into 0-235? I had problems with gamma shift before, when uploading to vimeo, so it`s an important thing to consider. Is x264 outputting a 16-235 file the correct workflow for vimeo? (PS. I have read vimeo guidelines, and there`s no metion for the output range)
    Vimeo passes through YUV levels when re-encoding. It's upon the flash display that converts YUV to RGB(0-255) using Rec709 . The video itself is still YUV , same levels as you uploaded

    Again, x264 doesn't alter levels , unless you tell it to with the --range and --input-range parameters


    Question #4: What I am doing wrong here? Anyhow I can tweak x264 to make this fading better with less banding?
    145Mb/s should be more than plenty for your file. Something else is going on. I would clean up and filter the original more as well (there is already macroblocking in your filtered AVI ) - but part of that is probably jpeg compression from your screenshot

    In general , for encoding a dithered product:

    A) You want to increase AQ strength, not decrease

    B) Try grain retention settings to retain the dither . Especially increasing psy-rd , psy-trellis, lowering deadzones . Basically along the lines of --tune grain, but increasing AQ instead of decreasing it

    C) If that section is still problematic, you can use zones to regulate a higher bitrate and different settings in that section (unfortunately, some parameters like AQ cannot be effected by zones)

    D) If vimeo accepts 10bit AVC, I would use that. Banding will significantly be reduced on the uploaded file (But it will probably re-appear when vimeo re-encodes it)

    *EVEN if your uploaded file is perfect , there is a large chance that there will be banding when vimeo re-encodes it due to lower bitrates it uses
    Last edited by poisondeathray; 18th Apr 2014 at 20:50.
    Quote Quote  
  4. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Can you upload 1 frame from your filtered AVI from vdub (using video => direct stream copy) , from that section ? If it's uncompressed AVI, you can zip it up as well (eg. 7zip , winrar)

    You can tell if your pre-filtering has done a good job or not, but looking at it with histogram("luma") . It's an enhanced view, that will show defects. The problem with doing it on a jpeg, is I don't know what is real, or jpeg artifacts from your screenshot. I suspect much of them are real compression artifacts from the original timelapse footage

    This "master AVI" looks poorly dithered to me, and those blocks will predispose you to farther banding when you re-encode, and again when vimeo re-encodes

    Click image for larger version

Name:	0.png
Views:	24
Size:	1.28 MB
ID:	24612
    Quote Quote  
  5. Originally Posted by poisondeathray View Post
    Can you upload 1 frame from your filtered AVI from vdub (using video => direct stream copy) , from that section ? If it's uncompressed AVI, you can zip it up as well (eg. 7zip , winrar)

    You can tell if your pre-filtering has done a good job or not, but looking at it with histogram("luma") . It's an enhanced view, that will show defects. The problem with doing it on a jpeg, is I don't know what is real, or jpeg artifacts from your screenshot. I suspect much of them are real compression artifacts from the original timelapse footage

    This "master AVI" looks poorly dithered to me, and those blocks will predispose you to farther banding when you re-encode, and again when vimeo re-encodes
    Sure, I have uploaded both the master 10bit (before dithering) and the 8bit dithered file using the avisynth code (both with direct stream copy). The file is attached, thanks for the help.

    Ok, I tried increasing the AQ Strenght to 1,8 and re-encode. File looks good but I am confused now. I am using my notebook (I am traveling and unable to use my desktop). When I open the x264 video files in vdub, all of them seems with less banding now. Here are the screen captures:

    #1a: The same original x264 145kbs video file opened in vdub in my notebook: http://i.imgur.com/1KYLCal.jpg
    #1b: The same original x264 145kbs video file opened in vdub in my desktop: http://i.imgur.com/JJ6dU6U.jpg

    #2a: The same original x264 145kbs video file opened in Media Player Classic in my notebook: http://i.imgur.com/ZbpuzDD.jpg
    #2a: The same original x264 145kbs video file opened in Media Player Classic in my desktop: http://i.imgur.com/VHBlX0N.jpg

    #3a: New encode using aq strenght 1,8 but opened in vdub on my notebook: http://i.imgur.com/zjFXeHs.jpg (compare with #1a)
    Not conclusive.

    The new encoded file (#3a) looks way sharper on the buildings, but only after upload to imgur. Imgur is messing with the files somehow. Both are equally sharp while I am viweing them on Photoshop.

    I have to open this file in my desktop now to have a screenshot to compare with #1b

    I am using DShowInputDriver.vdplugin to open x264 files in vdub. I don't know what I am using to open in my desktop where the screenscaptures were first made. I don't know what MPC is using to display the images in each computer. Could this be a display problem that is causing the banding on the preview rather than encoding problem?
    Attached Files
    Quote Quote  
  6. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Yes, likely you have display configuration issues

    You made a mistake on "teste 8bits dithering.avi", it has been converted to RGB. If that was cut with direct stream copy, probably you forgot to use video=>fast recompress in vdub on the original file when filtering the 1st time, but otherwise dithering pattern looks fine

    You're using lossy jpeg screenshots for comparisons, and different frames to compare. There are too many variables , for example did you use the same jpeg compression setting? Is this an I-frame vs. b-frame comparison ? etc... You should be using lossless comparisons of the same frame (png), using the same decoder, same renderer
    Last edited by poisondeathray; 20th Apr 2014 at 14:42.
    Quote Quote  
  7. Originally Posted by poisondeathray View Post
    Yes, likely you have display configuration issues

    You made a mistake on "teste 8bits dithering.avi", it has been converted to RGB. Probably you forgot to use video=>fast recompress in vdub, but otherwise dithering pattern looks fine

    You're using lossy jpeg screenshots for comparisons, and different frames to compare. There are too many variables , for example did you use the same jpeg compression setting? Is this an I-frame vs. b-frame comparison ? etc... You should be using lossless comparisons of the same frame (png), using the same decoder, same renderer
    Thanks for taking a look a the files. I did not use video=>fast recompress, I used video=>direct stream copy. (on the original file that I used avisynth to dither from 10 to 8 bit too). Should I have used fast recompress when loading avisynth code? Ok, that makes sense, and a silly error now.

    I am aware of the different frames, but somehow when I open the x264 file on vdub the frame count is much more than the real frames, so I have to catch a frame somewhere near. I don't know if I am using b frames or i frames. Sorry, I am very new to this world, anywhere I can know the difference between them. I exported them as jpg with maximum quality (12 in Photoshop). I have exported pngs also, but the differences we are seeing are in another order of magnitude in comparison with jpg artifacts or png. They represent consistently what I was seeing on the screen during the entire scence, I am not analysing only these frames, so that's why I accepted those. But, I'll start again and do in a proper way the comparisons after the display issues are sorted.

    Anyway to know which decoder and redering are my notebook and desktop using so I can match both? Thanks
    Quote Quote  
  8. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    I edited my post above . I meant to say you probably forgot to use fast recompress the first time when filtering the video (not when cutting the sample) . ie. If you used direct stream copy for the sample, then the original filtered version is in the wrong color space . The dithered output from the avs script should be in YUV not RGB

    Anyway to know which decoder and redering are my notebook and desktop using so I can match both?
    Look in the options in mpchc (options => output)

    Also you have to configure your graphics cards , drivers, display/monitor levels - set the levels properly, calibrate them as well . Make sure the GPU is set to the highest bit depth . There are many factors that can affect what you see, or the screenshot taking ability

    If you want to be more consistent in vdub display, options=>preferences=>display and uncheckmark "use directx for display panes"

    Also, taking screenshots in vdub, use video=>copy output frame to clipboard (ctrl+2) , and paste into your image editor . The screenshots should have the correct levels, but will be using Rec601 for the conversion (colors will be shifted slightly, as the conversion from YUV to RGB for the screenshot should be using Rec709 )

    I am aware of the different frames, but somehow when I open the x264 file on vdub the frame count is much more than the real frames, so I have to catch a frame somewhere near.
    Then use something frame accurate , like avisynth ffms2 to open the videos
    Quote Quote  
  9. Thanks for telling me the correct workflow to take screenshots, didn't know about the crtl+2 option and disable directx. I have calibrated displays both in desktop and notebook (I have the colormunki smile calibrator, an entry level one).

    So, I have uploaded 2 screenshots from vdub saved as png using crtl+2 and disabling directx at display, both taken in my notebook. One is the original x264 145kbps and the other is the same with AQ strenght 1,8. Same frame. Not a difference, after all.

    Looks like I have a display problem in my desktop, since my notebook shows no ugly banding (although not perfect, fine for me). I am very surprised on discovering that, since both virtualdub and mpc show exagerated banding on my desktop. Does virtualdub use the same decoder and renderer as MPC?

    MPC option=>output settings shows everything "system default" on my notebook. Should I change it to anything else, like null? If the settings in my desktop also shows "system default" how can I check what is causing these displays differences, assuming the GPUs are at the highest bit depth (I'll check that aswell, but it is likely to have)?

    Also, thanks for pointing out the "fast recompress" option. I forgot to check that when loading the avisynth script and that should let the problem worse also. I'll generate another 8 bit file with fast recompress checked and encode via x264 again in my desktop and report the results.
    Attached Thumbnails Click image for larger version

Name:	Correct comparison 145 original.png
Views:	27
Size:	2.58 MB
ID:	24649  

    Click image for larger version

Name:	Correct comparison 1,8aqstrenght.png
Views:	34
Size:	2.58 MB
ID:	24650  

    Last edited by alfas; 20th Apr 2014 at 15:50.
    Quote Quote  
  10. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    You don't need to upload another proper AVI sample - it was just to rule out other problems (and to get better idea of what is going on, hard to tell from a lossy jpeg) - but it looks like you have most of it figured out already except for the display issues


    Does virtualdub use the same decoder and renderer as MPC?
    It depends how file is loaded into vdub, and what decoder is configured to be used. Same with MPC. You can configure them with different decoders . But the decoder choice usually won't be the culpirt for "banding" - techincally all decoders should output the same thing (but in the real world it's not always true)



    MPC option output settings shows everything "system default" on my notebook. Should I change it to anything else like null? If the settings in my desktop also shows "system default" how can I check what is causing these displays differences, assuming the GPU are at the highest bit depth (I'll check that aswell, but it is likely to have)?
    A lot of it is personal preference. There can be large differences between some renderers. Some have higher quality upsampling (e.g madvr), some have higher CPU usage, some are able to convey info from stream flags

    But if between the 2 systems, the MPCHC settings were the same, then check for the GPU configuration differences next

    Since both applications are affected on your desktop system, that suggests a global issue like a GPU setting. Sometimes cheap display panels can be the problem (they come in 6,8,10 bit varieties) . But I would expect the laptop to have a cheaper display than the desktop system
    Last edited by poisondeathray; 20th Apr 2014 at 16:18.
    Quote Quote  
  11. I think you have VirtualDub set to 16 bit display mode. Change it:

    Options -> Preferences -> Main (left pane) -> Output Color Depth (right pane) -> Match Display Depth or 24 bit TrueColor.

    Also, turn off DirectX:

    Options -> Preferences -> Display (left pane) -> deselect Use DirecteX for display panes (right pane)

    Save and restart.

    Go to your graphics card's setup applet and make sure it's set to 24 bit or 32 bit RGB. Go to the video proc amp settings and disable all automatic features like sharpening, contrast enhancement, skin tone enhancement, etc. All they do is mess up the video.
    Last edited by jagabo; 20th Apr 2014 at 17:37.
    Quote Quote  
  12. Hello guys, sorry for the late reply. I have made all the adjustments recommended but didn't solve the problem. I don't know if there's much more to do, I guess I have to deal with it. Maybe try another stronger debanding filter? Or maybe the problem is on my decoders, i don't know.

    Anyway I am facing another problem now with aliasing. I have included 2 screenshots attached: the first is the master file 8bit before encoding (34Gb size) and the after x264 on Constant Quality 20 (250Mb size). The aliasing is very strong and much more than we can see on screenshot as camera moves. Images must be seen at full size, not scaled down or else the comparison doesn't work. This alias movement effect is not seen on the master file 8bit.

    Is it the case on start looking at anti-aliasing filters and put on avisynth script? Or is there any setting I can tweak on x264 that helps minimizing it? If the first the answer, I started to search some of them, but it seems that most are made for anime. Is there any good anti-aliasing filters for real footage and temporal?

    Thanks
    Attached Thumbnails Click image for larger version

Name:	frame 3348 Master 8bits dithered.png
Views:	37
Size:	3.87 MB
ID:	25029  

    Click image for larger version

Name:	frame 3348 x264.png
Views:	38
Size:	4.47 MB
ID:	25030  

    Quote Quote  
  13. That's not caused be x264. Look elsewhere. Did you deinterlacer?
    Quote Quote  
  14. Hi, I have not deinterlaced. I have created the source file on 1080p and 30fps in After Effects. But that's interesting to point out since I remember to have already enceded the same footage with no ugly temporal aliasing like that. And as far as I remember with the same settings. Maybe there was a problem with the framerate somehow along the way. Do the filters Detelecine and Ddecomb being on "default" or on "off" can have some effect on that?
    Quote Quote  
  15. Detelecine and Decomb are deinterlacing techniques.
    Quote Quote  
  16. Hi jagabo, thanks. Didn't know about that. I have read somewhere to have them set always on default, but that was just not right. It seems ok now.
    Quote Quote  
  17. Hi guys, thanks to everyones input I could export a video that has not to horrible banding (but still a little in one rapid scene) and no aliasing. I appreciate all the help. You can have a look at the output if you wish here:


    Critics and opnions about the encoding (or anything else) will be very welcome and taken into account on my next projects.
    Quote Quote  
  18. Very nice work.
    Quote Quote