Dealing with strong deblocking

[Marcoh_]

Hey everyone,
I just got a copy of the video I posted below, and noticed it has a considerable number of issues.
After doing some tests, I managed to get rid of most of them, but I haven't found any way to remove the blockiness, which gets really annoying in some shots.

Can anybody help me to find out how to get rid of it?
Thanks a lot.

Sample video: https://mega.nz/#!QhZT0I7A!WSV09zeXINGicMKoGB0wzDckRANRpVTw_fpAiIaMsRw

[jagabo]

You could use DgMpgDec's built in deblocker. Or you could use deblock_qed() in AviSynth -- which gives you more control over the strength of the deblocking.

[Marcoh_]

Or you could use deblock_qed() in AviSynth -- which gives you more control over the strength of the deblocking.

Did some tests with it, as it's always been very useful for what I needed. But this time I couldn't set it in a satisfying way. Can you help me please?

[jagabo]

Did you deinterlace first? The docs for deblock_qed() give an example of how to handle interlaced video. As for deblock_qed() within that sequence, try something around deblock_qed(quant1=40, quant2=50).

[Marcoh_]

I've tried with this one:
Deblock_QED(quant1=50, quant2=60, aOff1=4, aOff2=4, bOff1=6, bOff2=6, uv=3)

EDIT: Then I raised to about 10/15 the aOff/bOff parameters. The blockiness has gone, but some new annoying artifacts have been created.
Would you be so kind to do a little test to help me with this problem?

[LMotlow]

I think you mean block noise, not "deblocking". I don't see much block noise, if any. I do see what looks like RF noise, or gray vertical striping.

And some other problems. Looks like a bad NTSC conversion (?), with duplicate frames every 6th frame. Most of the duped frames have displaced pixels along the bottom, splitting the picture into two parts. This looks like field blending. The white arrow in the image below points to the split line:



And blacks clipped at y=16.

[Marcoh_]

Unfortunately I couldn't get a better copy of this video. Between the other issues you mentioned, it needs some colour correction as well, which I will be testing in the next few days.

BTW, since I'm quite a newbie in this field, would you try to help me to improve this footage as much as we can?
Thanks.

[jagabo]

some new annoying artifacts have been created.

Post a sample image showing the problem. And your entire script.

[jagabo]

I didn't think it would work -- but those torn frames can mostly be removed with TDecimat(mode=2, rate=25).

[LMotlow]

The problem with removing the torn frames, after looking at it more closely, is that it isn't the frame that's torn -- it's the field. The frame containing the torn field isn't a duplicate, only looks that way if you go too fast with a bob deinterlace. Think of the top field of the frame as instant #1, and the bottom field as instant #2. The top (even) field consists of the complete image of instant #1. The bottom (odd) field consists of the top part of instant #2, but still has some of the bottom portion of instant #1. If you want the complete image of instant #2, it's in the top field of the next frame.

       FRAME 1                   FRAME 2                   FRAME 3    
even field  odd field     even field  odd field     even field  odd field
    |           |             |           |             |          |
instant #1  instant #2    instant #2  instant #3    instant #4  instant #5
            ---------- 
            instant #1

This happens every 6 frames. TDecimate removed some good frames but kept all of the split-field frames. As the above table shows, there are no duplicate frames and no duplicate fields. The split in the field at every 6th frame is unfortunately never exactly half of the next field, but varying portions of it.

Between the other issues you mentioned, it needs some colour correction as well

Color correction for live concerts is really really really tricky. You have to decide whether the color balance in a shot is the result of colored stage lights, strobes and camera angles, or if it's just bad color. Often you have to work it shot by shot. The problems I saw in your sample are undersaturated chroma (except dark red, which is oversturated), not enough blue, a slightly dim image that needs some luma gain, and grimy looking darks. I used a ContrastMask filter for the latter to bring up shadow detail, with ColorYUV and SmoothTweak for the rest.


I don't see how deblock_qed could fix the RF noise bars. I used a VirtualDub filter named "Frequency Suppressor of the Noise," at 1/3 power. The name sounds like a bad Russian translation. It's formally called FSN.vdf, which you can get here: http://acobw.narod.ru/file/fsn.zip (link updated thanks to jagabo, post #17) . ContrastMask.avsi is attached along with the VariableBlur plugin, which is required for it.

I ran the script below to get the attached mp4. Probably other tweaks could be used. What could be done with that glitch at every 6th frame is anyone's guess. Maybe SelectEvery(6,0,2,3,4,5), but what if there's a cadence change along the way? And you'd have to resync audio.

vid=MPEG2Source("path to d2v project file\Example.d2v")
aud=NicAC3Source("path to audio file\Example T80 2_0ch 384Kbps DELAY -98ms.ac3", Channels=2)
AudioDub(vid,aud)
ContrastMask(enhance=2.0)
ColorYUV(gain_y=10,cont_u=35,off_u=2)
SmoothTweak(saturation=1.5)

LoadVirtualDubPlugin("path to VirtualDub\plugins\fsn.vdf", "FSN", 1)
ConvertToRGB32(interlaced=true)
FSN(45,45,2,1,1)
Crop(20,10,-16,-2).AddBorders(18,6,18,6)
ConvertToYV12(interlaced=true)

AssumeTFF()
SeparateFields()
TemporalSoften(3,3,8,15,2)
MergeChroma(awarpsharp2(depth=20).awarpsharp2(depth=10))
LimitedSharpenFaster(edgemode=2)
weave()
return last

[jagabo]

Yes, you have to remove the torn frames after (smart) bobbing.

[LMotlow]

Hm. Actually, it won't work because the cadence does change. After about 50 frames it's one in every 5, then later back to 6, back to 5, etc.
Well, it was an idea...
I won't even hazard a guess about what caused it.

[EDIT] Nope, again. Something screwy with that mpg file. How was it made? Every proc I use to open that mpg (FFMS2, LSMASH, DGIndex, DircetSHowSoiurce) gives me a different video and frame structure. If it was a DVD/VOB structure, the sample should be an m2v made with DGIndex.

Hey, Marcoh_, how'd you make that mpg example?

[Marcoh_]

Hey, Marcoh_, how'd you make that mpg example?

Actually, I cut it in "copy mode" using AviDemux. I thought it was the easiest way to cut a 10 seconds sample video.
Sorry if I haven't posted my initial script, but unfortunately I almost haven't created any so far for this video, as I was trying to understand how to clean it up.

Thanks a lot for the sample script.

[LMotlow]

IMO Avidemux gave you a screwy mpeg and every filter has different scanning errors with it. Mpeg samples with field and frame accuracy are usually made with DGIndex. Here's an old post showing how to make an m2v sample video: https://forum.videohelp.com/threads/359295-Record-without-interlacing?p=2272359&viewfu...=1#post2272359. If you want to post audio with it, post the AC3 audio file that's made by DGindex when it creates the project. What you're getting so far is barely workable. DGIndex is almost always used to decode mpeg/vob for processing. If you're going to do any cleanup you need a frame-accurate video to work with.

[Marcoh_]

All right, thanks for your tips and sorry for the delay
I've tested your script starting from a proper sample mpeg sample made with DGIndex.

And while we're discussing on this thread, I don't want to open a new one for a simple question: I was doing some tests on a Deep Purple b/w live video, and noticed some annoying grain/mosquito noise I would try to remove. Any help please?

PROPER sample attached below.
Thanks again!

[LMotlow]

What grain and mosquito noise? I'm having trouble seeing anything under all the RF horizontal hum bars, which is far worse here than in your previous sample. You can clean up almost any grain and mosquito noise with any of several temporal smoothers, but fixed hum bars aren't going away anytime soon. I don't think you have enough grain to worry about anyway, or this video will look plastic. Grainy b/w is intentional here and is common. It's been de rigueur for decades on MTV.

[jagabo]

By the way, the link to fsn.zip in post #10 is wrong. The fsn that's used in the script is at: http://acobw.narod.ru/file/fsn.zip

[LMotlow]

By the way, the link to fsn.zip in post #10 is wrong. The fsn that's used in the script is at: http://acobw.narod.ru/file/fsn.zip

Thank you. This must have changed recently at infognition. I'll update my post and credit yours. Thanx again.