About using Merge() to improve quality

[Aludin]

I decided to do some more experiments to find out how useful this technique might be. It's already established that combining multiple clips cancels out noise and other temporal problems. Then I got an idea in my head and thought that since different levels of quantization morph the noise differently each time, could the same noisy clip be compressed with different settings and then Merged to essentially remove all noise and improve quality?

I encoded a grainy clip from crf17 to crf30 x264 and then Merged them altogether into one average. It worked very well. All noise was gone and the quality was very good considering that most of the clips had a rather high CRF and looked crappy on their own.

Unfortunately, this doesn't do better than traditional denoisers and Neat Video preserved more legitimate detail but this could be a neat alternative in scenarios where a proper noise profile can't be selected.

So to survey how much damage to the original source this method does, I encoded a noiseless animation clip from crf18 to crf30 and Merged them together. As expected, moving objects were a little flawed. With objective metrics, the Merged clip was rated better quality than the best of the encoded clips. Better quality than crf18 despite it being the average of clips as high as crf30.

That's the part I found interesting. How does introducing 10 crappier quality clips into the equation increase the quality of the best one in the mix? Does this defeat the common wisdom of Shannon's theorem or the proverbial "garbage in, garbage out"?

[jagabo]

I tried it with a noisy vhs cap of a cartoon (crf 18, 20, 22...32, then merging all 8 videos, equally weighted) and it worked surprisingly well. But this source had little detail. I'll have to play around with it some more.

[poisondeathray]

What metric did you use ? What did per-frame metrics (or summary of the trend) look like ?

Did the final clip subjectively look better than the lowest crf clip ?

What about playing with higher weighting to the lower crf clips ?

[jagabo]

What metric did you use ?

Eyeballs.

Did the final clip subjectively look better than the lowest crf clip ?

The results weren't perfect but better than I expected. The merged clip looked better than the source and any of the individual clips -- in terms of noise reduction. The source had lots of VHS noise, dot crawl artifacts, oversharpening halos, horizontal time base jitter, etc. It was all substantially reduced up but there was some loss of fine detail. Of course, more conventional noise reduction techniques would cause some loss of detail too.

before and after:

[Attachment 39512 - Click to enlarge]
[Attachment 39513 - Click to enlarge]

What about playing with higher weighting to the lower crf clips ?

I suspect that may reduce noise a bit more but would likely cause more loss of fine detail and more jagged edges.

I'm going to play around with it some more and compare to more conventional denoising. And some different clips. Lots of things to play around with...

Encoding batch file (a faster preset may work too, better? worse?):

x264.exe --preset=slow --crf=18 --output %1.18.mkv %1
x264.exe --preset=slow --crf=20 --output %1.20.mkv %1
x264.exe --preset=slow --crf=22 --output %1.22.mkv %1
x264.exe --preset=slow --crf=24 --output %1.24.mkv %1
x264.exe --preset=slow --crf=26 --output %1.26.mkv %1
x264.exe --preset=slow --crf=28 --output %1.28.mkv %1
x264.exe --preset=slow --crf=30 --output %1.30.mkv %1
x264.exe --preset=slow --crf=32 --output %1.32.mkv %1

Merge Script:

v0 = LWLibavVideoSource("aticap.avs.18.mkv") 
v1 = LWLibavVideoSource("aticap.avs.20.mkv") 
v2 = LWLibavVideoSource("aticap.avs.22.mkv") 
v3 = LWLibavVideoSource("aticap.avs.24.mkv") 
v4 = LWLibavVideoSource("aticap.avs.26.mkv") 
v5 = LWLibavVideoSource("aticap.avs.28.mkv") 
v6 = LWLibavVideoSource("aticap.avs.30.mkv") 
v7 = LWLibavVideoSource("aticap.avs.32.mkv") 

v01 = Merge(v0,v1)
v23 = Merge(v2,v3)
v45 = Merge(v4,v5)
v67 = Merge(v6,v7)

v0123 = Merge(v01,v23)
v4567 = Merge(v45,v67)

Merge(v0123, v4567)

[Aludin]

So what were your conclusions, jagabo?

[pandy]

I will appreciate if someone will explain to me why such approach as encoding same source with different crf and later combining result...? See no point in this.
btw i understand general idea for noise average (assumption is that noise is not correlated) but simply do not get this idea (as video compression is correlated).

[jagabo]

The idea is to use the encoder's discreet cosine transform and quantization to remove low amplitude, high frequency data (which includes noise). Of course, encoding with high quantizers results in a large loss of detail and the creation of artifacts. But by averaging the results of several low to high quantizers you keep more detail and you average away the artifacts.

Here's the results from another test I ran. Starting with a 640x480 losslessly encoded YV12 video (bad VHS source inverse telecined and resized), encoded with the following batch file:

"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 16 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out1.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 20 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out2.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 24 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out3.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 28 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out4.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 32 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out5.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 36 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out6.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 40 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out7.mp4"
"G:\Program Files\QSVEnc\QSVEncC\x86\QSVEncC.exe" -i %1 --codec h264 --quality best --gop-len 25 --b-pyramid --scenechange --cqp 44 --ref 4 --sar 1:1 --colormatrix smpte170m -o "out8.mp4"

Merged with the following AviSynth script:

v0 = LSmashVideoSource("out1.mp4") 
v1 = LSmashVideoSource("out2.mp4") 
v2 = LSmashVideoSource("out3.mp4") 
v3 = LSmashVideoSource("out4.mp4") 
v4 = LSmashVideoSource("out5.mp4") 
v5 = LSmashVideoSource("out6.mp4") 
v6 = LSmashVideoSource("out7.mp4") 
v7 = LSmashVideoSource("out8.mp4") 

v01 = Merge(v0,v1)
v23 = Merge(v2,v3)
v45 = Merge(v4,v5)
v67 = Merge(v6,v7)

v0123 = Merge(v01,v23)
v4567 = Merge(v45,v67)

Merge(v0123, v4567)

And compared to the same video filtered with MCTemporalDenoise(settings="high").

Source frame:


MCTD result:


The individual cqp=16 result looked much the same as the original, the cqp=44 result looked very bad, and the others between those extremes. Here's the cqp=44 result:


And the final merged QSVEnc result:


MCTD gave a better result (less noise, more detail, sharper). But the merged QSVEnc result is surprisingly good.

My conclusion is that using a real noise reduction filter is superior than merging the results of several h.264 encodings. But the latter technique is interesting.

[pandy]

The idea is to use the encoder's discreet cosine transform and quantization to remove low amplitude, high frequency data (which includes noise). Of course, encoding with high quantizers results in a large loss of detail and the creation of artifacts. But by averaging the results of several low to high quantizers you keep more detail and you average away the artifacts.
MCTD gave a better result (less noise, more detail, sharper). But the merged QSVEnc result is surprisingly good.

My conclusion is that using a real noise reduction filter is superior than merging the results of several h.264 encodings. But the latter technique is interesting.

Thx! A bit bizarre IMHO using codec as lowpass filter, isn't better to use dedicated DCT noise remove filter and eventually stack few results or use better filter (NLM or wavelet decomposition with adaptive filtering on particular spectrum) ?
I would always recommend such technique for analog capture where it can lead to really good results as noise is usually not correlated with signal but for deterministic distortion from compression which is highly correlated with distortion it may loose quality quickly even sometimes highly exaggerating particular distortions - common issue with broadcast signal flow over multiple codecs.

[jagabo]

A bit bizarre IMHO using codec as lowpass filter, isn't better to use dedicated DCT noise remove filter and eventually stack few results

Of course. But you would have to write your own code to do that.

[pandy]

A bit bizarre IMHO using codec as lowpass filter, isn't better to use dedicated DCT noise remove filter and eventually stack few results

Of course. But you would have to write your own code to do that.

Or perhaps use something already existing...

https://www.ffmpeg.org/ffmpeg-filters.html#dctdnoiz

(miss photoshop interface) then http://www.ximagic.com/d_index.html could be nice to try.

[jagabo]

A bit bizarre IMHO using codec as lowpass filter, isn't better to use dedicated DCT noise remove filter and eventually stack few results

Of course. But you would have to write your own code to do that.

Or perhaps use something already existing...

https://www.ffmpeg.org/ffmpeg-filters.html#dctdnoiz

You can put together some examples of how to use it.

[Aludin]

btw i understand general idea for noise average (assumption is that noise is not correlated) but simply do not get this idea (as video compression is correlated).

Each different crf will transform the noise a different way, so it isn't correlated. Signal will largely stay the same while noise will be drastically altered each time. So the merge destroys the noise but also some of the signal. Jagabo's examples didn't do so well, I should post mine. The results were just a step below Neat Video's quality of noise reduction which is saying a LOT. Just a simple, retarded trick that any idiot can do competes with an advanced noise remover developed over a decade. No noise profile selected to target the noise, no fine-tuning, no tuning of any kind.

It's unbelievable. I'm still not sure what lesson to draw from this.

[pandy]

Each different crf will transform the noise a different way, so it isn't correlated. Signal will largely stay the same while noise will be drastically altered each time. So the merge destroys the noise but also some of the signal. Jagabo's examples didn't do so well, I should post mine. The results were just a step below Neat Video's quality of noise reduction which is saying a LOT. Just a simple, retarded trick that any idiot can do competes with an advanced noise remover developed over a decade. No noise profile selected to target the noise, no fine-tuning, no tuning of any kind.

It's unbelievable. I'm still not sure what lesson to draw from this.

Well - each crf will transform noise in same way as codec behaviour is deterministic and it is highly correlated (encode same source twice with same settings - you should get in proper codec bitexact stream) - trust me - being electronics engineer i know how to process weak signal buried bellow noise floor and that's why i was bit surprised to see this idea... have no problem with statistical processing concept and i can understand goal of this exercise but provided sample can be processed IMHO better with help of classical approach (noise combined with some periodical distortion - just check frequency domain) - this concept looks like blind multi-scale processing and that's why i think that wavelet decomposition probably will be faster and better...

[pandy]

You can put together some examples of how to use it.

Well... i'm not convinced to method (too high computational cost).