Direct 8-bit Conversion Using QTGMC

[taigi]

Hi everyone,

I'm using a Canon Legria HFG25, which records interlaced video with some halo effects.

I've been processing this footage with AviSynth using QTGMC and dehaloing.

I converted the material to 10-bit, but now I want to archive it in 8-bit at 15 Mb/s.

Question: Can I achieve this directly in one process, converting straight to 8-bit at 15 Mb/s, without first going through the 10-bit step? I'm concerned about quality - will it be preserved?

Thanks for any advice!

[Selur]

Curious: Up to now, I have not seen one example that shows that using QTGMC in 10bit would show any better quality than sticking with 8bit (unless the source already was 10bit and the loss you see is through converting to 8bit), so I wonder where does the idea come from that it would be better to convert 8it to 10bit before feeding it to QTGMC?

[taigi]

Curious: Up to now, I have not seen one example that shows that using QTGMC in 10bit would show any better quality than sticking with 8bit (unless the source already was 10bit and the loss you see is through converting to 8bit), so I wonder where does the idea come from that it would be better to convert 8it to 10bit before feeding it to QTGMC?

The idea of processing 8-bit sources in 10-bit with QTGMC comes from the general principle that higher bit depths can reduce rounding errors in complex operations. This approach was suggested by GPT, mentioning "bound headroom" as a potential benefit. But yes, concrete evidence of improvement is lacking.

[Selur]

Ah, okay. I would have been surprised to see any (even synthetic) example where this would help with QTGMC.
=> in my option: I see no reason to up convert to 10bit (or even 16bit or 32bit) before deinterlacing with QTGMC.
So I doubt that you will lose anything when deinterlacing 8bit content in 8bit QTGMC.

[rgr]

Curious: Up to now, I have not seen one example that shows that using QTGMC in 10bit would show any better quality than sticking with 8bit

Yes, converting to 10 bits improves quality (in fact, you shouldn't go below 16 bits). This is especially noticeable if you use other filters (and that's what QTGMC does). I once even posted an example of how well it reduces banding (of course due to the lack of rounding when moving from filter to filter).

[Selur]

Using 10/16bit helps with:
a. debanding and similar (from what I see in QTGMC none of the filters used there would gain from upsampling to 10bit before processing)
b. compression
no question about that. But, have not seen that it helps at all with QTGMCs processing.
=> if you can show that is does, please do
(so please show that 8bit->10/16bit->QTGMC is better than 8bit->QTGMC->10/16bit, if you can, which I doubt, since like I wrote: I do not think any of the filters/methods in QTGMC will improve when fed upsampled 8bit (=10/16bit) instead of the original 8bit content)

Cu Selur

[rgr]

Hi everyone,

I'm using a Canon Legria HFG25, which records interlaced video with some halo effects.

I've been processing this footage with AviSynth using QTGMC and dehaloing.

I converted the material to 10-bit, but now I want to archive it in 8-bit at 15 Mb/s.

Question: Can I achieve this directly in one process, converting straight to 8-bit at 15 Mb/s, without first going through the 10-bit step? I'm concerned about quality - will it be preserved?

Thanks for any advice!

You can put ConvertBits(8, dither=-1) at the end of the script - that's probably the best way to convert to 8 bits. A codec won't do it any better.

[rgr]

The second screenshot also shows how QTGMC at 8 bits is more likely to go beyond the 16-235 range.

[rgr]

xx

[Selur]

Without knowing what scripts created those screenshots, they do not really say much.
Both images in your second screenshots violate 16-235,...

[Sharc]

In your second example it's the white text which exceeds 235. It should theoretically be at 255 exact (sRGB 255,255,255) in 8 bit realm or YCbCr(255,128,128) in YUV for sRGB matrix.

[Selur]

You can put ConvertBits(8, dither=-1) at the end of the script - that's probably the best way to convert to 8 bits

Unless any dithering was added beforehand, this will add banding to banding prone areas,....

[rgr]

Without knowing what scripts created those screenshots, they do not really say much.
Both images in your second screenshots violate 16-235,...

Sample examples from the net. You can make your own, the effect should be similar.
In the second example both exceed, but with 8 bits even more.

[rgr]

In your second example it's the white text which exceeds 235. It should theoretically be at 255 exact (sRGB 255,255,255) in 8 bit realm or YCbCr(255,128,128) in YUV for sRGB matrix.

I meant rather the range 0-16. After zooming in you can see that at 8 bits the overshoot is stronger.

[Selur]

Sample examples from the net.

Link?

[rgr]

https://files.fm/u/2pyf63r4kh

[Selur]

That file is progressive,... why would you apply QTGMC to that?
Given an 8bit source, the question was, whether there is a difference in quality comparing:

ConvertBits(16, dither=-1).QTGMC().ConvertBits(8, dither=-1)

vs.

QTGMC()

or

ConvertBits(16, dither=-1).QTGMC().ConvertBits(10)

vs.

QTGMC().ConvertBits(10)

and as such, is there a scenario where it makes sense to convert to a higher bit depth before applying QTGMC?

A progressive source does not make sense to test deinterlacing with QTGMC,....
And since you didn't care to post what did to the file, your screenshots do not really say much.

Cu Selur

[Selur]

I was hoping more for something like:

8bit Output:

ClearAutoloadDirs()
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\DGDecode.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\nnedi3.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mvtools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\masktools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\RgTools.dll")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mtmodes.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\Zs_RF_Shared.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\QTGMC.avsi")
# loading source: G:\TestClips&Co\files\interlaceAndTelecineSamples\interlaced\bff.m2v
# color sampling YV12@8, matrix: bt601, scantyp: bottom field first, luminance scale: limited
Source = MPEG2Source(d2v="J:\tmp\m2v_478bc6d007ec94bfc67367d30d9093a4_853323747.d2v",fieldop=0)
# current resolution: 720x480
# deinterlacing
Source = Source.AssumeBFF()

Source8bitFiltered = Source.QTGMC(PrevGlobals="Replace")
Source16bitFiltered = Source.ConvertBits(bits=16).QTGMC(PrevGlobals="Replace").ConvertBits(bits=8)

Source8bitFiltered = Source8bitFiltered.Histogram("classic")
Source16bitFiltered = Source16bitFiltered.Histogram("classic")
sub =Source16bitFiltered.subtract(Source8bitFiltered)
substrong = sub.levels(122,1,132,0,255)

StackVertical(Source8bitFiltered.Subtitle("8Bit"), Source16bitFiltered.Subtitle("16bit"), substrong.Subtitle("Diff"))
ConvertToRGB32(matrix="Rec601")
AssumeFPS(30000,1001)
PreFetch(16)
#  output: color sampling RGB32@8, matrix: bt601, scantyp: progressive, luminance scale: limited
return last.BicubicResize(Ceil(last.Width*0.888889) - (Ceil(last.Width*0.888889) % 2), last.Height)

16bit Output:

ClearAutoloadDirs()
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\DGDecode.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\nnedi3.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mvtools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\masktools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\RgTools.dll")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mtmodes.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\Zs_RF_Shared.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\QTGMC.avsi")
# loading source: G:\TestClips&Co\files\interlaceAndTelecineSamples\interlaced\bff.m2v
# color sampling YV12@8, matrix: bt601, scantyp: bottom field first, luminance scale: limited
Source = MPEG2Source(d2v="J:\tmp\m2v_478bc6d007ec94bfc67367d30d9093a4_853323747.d2v",fieldop=0)
# current resolution: 720x480
# deinterlacing
Source = Source.AssumeBFF()

Source8bitFiltered = Source.QTGMC(PrevGlobals="Replace").ConvertBits(bits=16)
Source16bitFiltered = Source.ConvertBits(bits=16).QTGMC(PrevGlobals="Replace")

Source8bitFiltered = Source8bitFiltered.Histogram("classic")
Source16bitFiltered = Source16bitFiltered.Histogram("classic")
sub =Source16bitFiltered.subtract(Source8bitFiltered)
substrong = sub.Levels(31354, 1, 33924, 0, 65535)

StackVertical(Source8bitFiltered.Subtitle("8Bit"), Source16bitFiltered.Subtitle("16bit"), substrong.Subtitle("Diff"))
ConvertToRGB32(matrix="Rec601")
AssumeFPS(30000,1001)
PreFetch(16)
#  output: color sampling RGB32@8, matrix: bt601, scantyp: progressive, luminance scale: limited
return last.BicubicResize(Ceil(last.Width*0.888889) - (Ceil(last.Width*0.888889) % 2), last.Height)

At least to me, the 16bit-filtering process for QTGMC doesn't seem to produce better quality to me.
Attached the original source so, others can reproduce the results.
If you think otherwise, please prove it through a script that allows others to reproduce it. Thanks!

Cu Selur

[rgr]

That file is progressive,... why would you apply QTGMC to that?

My mistake. But what does it matter how it was labeled if every frame will be the same anyway?

When marked as interlace: